coder-eval 0.11.4__tar.gz → 0.11.6__tar.gz

coder_eval-0.11.6/.claude-plugin/marketplace.json

@@ -0,0 +1,35 @@
+{
+  "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
+  "name": "coder-eval",
+  "owner": {
+    "name": "UiPath",
+    "email": "coder-eval@uipath.com",
+    "url": "https://github.com/UiPath/coder_eval"
+  },
+  "description": "Test whether your Claude Code skills actually trigger, and benchmark AI coding agents against your own tasks.",
+  "plugins": [
+    {
+      "name": "coder-eval",
+      "displayName": "Coder Eval",
+      "source": "./plugins/coder-eval",
+      "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
+      "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
+      "homepage": "https://coder-eval.com",
+      "repository": "https://github.com/UiPath/coder_eval",
+      "license": "Apache-2.0",
+      "category": "testing",
+      "keywords": [
+        "claude-code-skills",
+        "skill-testing",
+        "skill-activation",
+        "evaluation",
+        "benchmark",
+        "agent-testing",
+        "github-actions",
+        "sandbox",
+        "llm-judge",
+        "ci"
+      ]
+    }
+  ]
+}

{coder_eval-0.11.4 → coder_eval-0.11.6}/.github/workflows/pr-checks.yml

@@ -997,22 +997,37 @@ jobs:
         id: dogfood
         uses: ./
         with:
+          # Exercises what tests/test_action_inputs.py cannot: it asserts the argv
+          # both step scripts build, but only a real runner proves that
+          # `working-directory:` works on a composite step and that a plugin
+          # installed via `--with` is discovered at runtime. Hence a relative
+          # run-dir (landing under tasks/), the bare task filename and the `../`
+          # plugin path, all resolved from `working-directory`.
           version: local
-          tasks: tasks/hello_date.yaml
-          model: claude-haiku-4-5-20251001
+          working-directory: tasks
           run-dir: runs/ci-action-dogfood
-          junit-path: runs/ci-action-dogfood/junit.xml
-          # Credentials go through the generic env passthrough (the only channel);
+          extra-packages: ../tests/fixtures/byoa_demo_plugin
+          # The bracketed `-D` override is the case one-argv-entry-per-line exists
+          # for: `[...]` is a bash character class, so a whitespace-split input
+          # would collapse the list whenever a file in the cwd matched. It adds
+          # Glob to hello_date.yaml's three tools, so the assertions below can tell
+          # "arrived" from "ignored".
+          args: |
+            hello_date.yaml
+            --model
+            claude-haiku-4-5-20251001
+            -D
+            agent.allowed_tools=[Read,Write,Bash,Glob]
           # ANTHROPIC_API_KEY reaching the run is proven by the API-backed task
-          # succeeding. A floor of 0.0 passes for any produced score (exercises
-          # the gate path green in CI without flakiness); the second line
-          # exercises multi-line env parsing.
-          minimum-task-score: "0.0"
+          # succeeding; the second line exercises multi-line env parsing.
           env: |
             ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
             CE_DOGFOOD_MARKER=1
 
+      # In `tasks/` because that is the assertion: `run-dir` is reported exactly as
+      # passed, so a relative one is relative to `working-directory`.
       - name: Verify outputs and JUnit file
+        working-directory: tasks
         env:
           JUNIT: ${{ steps.dogfood.outputs.junit-path }}
           RUNDIR: ${{ steps.dogfood.outputs.run-dir }}
@@ -1023,11 +1038,66 @@ jobs:
           # our writer emits no DTDs/entities) — stdlib ET is fine here.
           python3 -c "import sys, xml.etree.ElementTree as ET; ET.parse(sys.argv[1])" "$JUNIT"
           test -f "$RUNDIR/run.json" || { echo "run.json missing"; exit 1; }
+          # An absolute path would mean working-directory was ignored.
+          case "$RUNDIR" in /*) echo "run-dir output was rewritten to an absolute path: $RUNDIR"; exit 1 ;; esac
+
+      # The action does not touch $GITHUB_STEP_SUMMARY: this is both the assertion
+      # that `run-md-path` is real and the recipe the docs hand consumers.
+      - name: Append the run report to the job summary
+        if: always()
+        working-directory: tasks
+        env:
+          RUN_MD: ${{ steps.dogfood.outputs.run-md-path }}
+        run: |
+          set -euo pipefail
+          test -f "$RUN_MD" || { echo "run-md-path output does not exist: $RUN_MD"; exit 1; }
+          cat "$RUN_MD" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Verify the plugin was discovered and the bracketed override arrived
+        working-directory: tasks
+        env:
+          RUNDIR: ${{ steps.dogfood.outputs.run-dir }}
+        run: |
+          set -euo pipefail
+
+          # `coder-eval` on PATH is the action's uv tool shim, so this interrogates
+          # its environment: a task naming the fixture's agent kind validates only
+          # if the entry point was discovered there, else plan exits 1 with
+          # "No agent registered for type 'byoa-demo'".
+          cat > byoa-probe.yaml <<'YAML'
+          task_id: "action_extra_packages_probe"
+          description: "Validates only when the byoa-demo plugin is discoverable."
+          initial_prompt: "not executed - plan validates without running an agent"
+          agent:
+            type: "byoa-demo"
+          success_criteria:
+            - type: "file_exists"
+              path: "app.py"
+              description: "not executed"
+          YAML
+          coder-eval plan byoa-probe.yaml
+          rm -f byoa-probe.yaml
+
+          # And the `-D` value survived as a 4-element list, not word-split or
+          # glob-rewritten.
+          RUN_JSON="$RUNDIR/run.json" python3 <<'PY'
+          import json, os, sys
+
+          data = json.load(open(os.environ["RUN_JSON"], encoding="utf-8"))
+          rows = data.get("task_results") or []
+          if not rows:
+              sys.exit("run.json has no task_results to check the -D override against")
+          tools = (rows[0].get("agent_config") or {}).get("allowed_tools")
+          expected = ["Read", "Write", "Bash", "Glob"]
+          if tools != expected:
+              sys.exit(f"-D override did not arrive intact: allowed_tools={tools!r}, expected {expected!r}")
+          print(f"bracketed -D override resolved to {tools!r}")
+          PY
 
       - name: Upload dogfood run on failure
         if: failure()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a  # v7.0.1
         with:
           name: action-dogfood-runs
-          path: runs/ci-action-dogfood/
+          path: tasks/runs/ci-action-dogfood/
           retention-days: 7

{coder_eval-0.11.4 → coder_eval-0.11.6}/.github/workflows/verify-published-action.yml

@@ -359,13 +359,11 @@ jobs:
           echo "--- task YAML:"; cat tasks/published_smoke.yaml
 
       # continue-on-error, because this step's exit code is NOT the gate. The action
-      # exits with coder-eval's own code (action.yml combines them), and coder-eval
-      # exits 1 on any failed task -- so a model flake failing `file_exists` would
-      # redden this workflow even with minimum-task-score at 0.0, which does not
-      # neutralize that path. This check must answer "does the published action still
-      # work", not "is the model still good": the verification step below gates on
-      # ARTIFACTS instead. A genuine model/credential outage still surfaces there, via
-      # the zero-token assertion.
+      # exits with coder-eval's own code, and coder-eval exits 1 on any failed task,
+      # so a model flake failing `file_exists` would redden this workflow. This check
+      # must answer "does the published action still work", not "is the model still
+      # good": the verification step below gates on ARTIFACTS instead. A genuine
+      # model/credential outage still surfaces there, via the zero-token assertion.
       - name: Run the published action
         id: run
         continue-on-error: true
@@ -373,11 +371,12 @@ jobs:
         with:
           # `version:` intentionally omitted -- the whole point is to exercise the
           # default pin baked into action.yml at the v0 tag.
-          tasks: tasks/published_smoke.yaml
-          model: claude-haiku-4-5-20251001
           run-dir: runs/verify-published
-          junit-path: runs/verify-published/junit.xml
-          minimum-task-score: "0.0"
+          # Task path and flags both go through `args` — the action promotes no CLI flag.
+          args: |
+            tasks/published_smoke.yaml
+            --model
+            claude-haiku-4-5-20251001
           env: |
             ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
 

{coder_eval-0.11.4 → coder_eval-0.11.6}/CHANGELOG.md

@@ -2,6 +2,205 @@
 
 <!-- version list -->
 
+## v0.11.6 (2026-09-01)
+
+### Bug Fixes
+
+- **action**: Drop the literal expression syntax from a run-step comment
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **action**: Stop the env passthrough mutating the action's own shell
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **docs**: Drop the last references to inputs that no longer exist
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **evalboard**: Even out the variant tiles and keep a task's arms adjacent
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+
+- **evalboard**: Resolve a variant-less task link to the task's first arm
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+
+### Documentation
+
+- **action**: Trim the comment bulk on the eight-input surface
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **evalboard**: Trim variant commentary and redundant tests
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+
+### Features
+
+- **action**: Add working-directory, extras, extra-packages, prerelease and args inputs
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **action**: Refactor github action input surface around args; harden env passthrough
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **evalboard**: Register an unlisted gha source for ad-hoc dispatch runs
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **evalboard**: Render multi-variant runs, one row per (task, arm)
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+
+### Refactoring
+
+- **action**: Drop every forwarding input; flags go through args
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **evalboard**: Drop the variant colour palette and the spread helper
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+
+- **evalboard**: Report pass rate per arm instead of alongside a blended one
+  ([#145](https://github.com/UiPath/coder_eval/pull/145),
+  [`6780227`](https://github.com/UiPath/coder_eval/commit/6780227e5e431b4c1099f2f88839be37540dcd71))
+
+### Testing
+
+- Drop the skip-guard unit test ([#146](https://github.com/UiPath/coder_eval/pull/146),
+  [`d679ff2`](https://github.com/UiPath/coder_eval/commit/d679ff2667f2c1e76a5d6384059107be4538c750))
+
+- Skip the enforcement live tests when the agent declines to try
+  ([#146](https://github.com/UiPath/coder_eval/pull/146),
+  [`d679ff2`](https://github.com/UiPath/coder_eval/commit/d679ff2667f2c1e76a5d6384059107be4538c750))
+
+- **action**: Record stub argv from bash so Windows argv is byte-exact
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+- **action**: Stop Git Bash rewriting the absolute path in the install-spec tests
+  ([#147](https://github.com/UiPath/coder_eval/pull/147),
+  [`7b81456`](https://github.com/UiPath/coder_eval/commit/7b81456e4f4ef8d26eaa9fdf8afe499376164d6b))
+
+
+## v0.11.5 (2026-08-28)
+
+### Bug Fixes
+
+- **opencode**: Bound the post-EOF reap, reap the whole process group, test every failure path
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Close the remaining review findings on the harness
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Gate on captured tokens, canonicalize tool args, pin max_turns
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Inject plugin skills so skill suites measure the skills
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Keep the smoke task out of the CI smoke-pass bucket
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Map apply_patch to Write so GPT-family edits are seen by criteria
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Reap the CLI on every turn exit, test the sandbox env contract
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Spread super() in get_environment_info; guard with CE046
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Typecheck on Windows, satisfy both CodeQL findings
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **plugin**: Close the review's verified gaps — regex blind spot, parallel surfaces, runtime guard
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **plugin**: Correct the skill-reachability path — every generated activation suite reports recall
+  0.0 ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **plugin**: Correct the skill-reachability path, and reuse PR #109's measured descriptions
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **routing**: Decouple simulator route from checker_context.api_route
+  ([#144](https://github.com/UiPath/coder_eval/pull/144),
+  [`88ff0f0`](https://github.com/UiPath/coder_eval/commit/88ff0f0ff910f4c3d9c6b81c5b2b88356240a3c4))
+
+- **routing**: Reinstate litellm+agent_judge rejection guard
+  ([#144](https://github.com/UiPath/coder_eval/pull/144),
+  [`88ff0f0`](https://github.com/UiPath/coder_eval/commit/88ff0f0ff910f4c3d9c6b81c5b2b88356240a3c4))
+
+- **routing**: Restore LiteLLM->Claude pin on simulator_route
+  ([#144](https://github.com/UiPath/coder_eval/pull/144),
+  [`88ff0f0`](https://github.com/UiPath/coder_eval/commit/88ff0f0ff910f4c3d9c6b81c5b2b88356240a3c4))
+
+- **test**: Add CE045, document the plugin-path divergence, unpin a test from ordering
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **utils**: Use explicit concatenation in the plugin-root warning
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+### Chores
+
+- **opencode**: Standardize on deepseek-v4-pro, drop the flash-0731 rate entry
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **plugin**: Answer "skills only?", fold tags into keywords, add CE044
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Give both author objects the same contact address
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Give the marketplace owner a contact address
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Lead both manifests with skill evaluation, add discovery metadata
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Pin both manifests to their published JSON schemas
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+### Documentation
+
+- **opencode**: Note _TERM_GRACE_SECONDS's second role as the post-EOF exit grace
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+### Features
+
+- **agents**: Add OpenCode harness with opt-in [opencode] extra
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Add require_token_telemetry, an escape hatch for the zero-token guard
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+
 ## v0.11.4 (2026-08-27)
 
 ### Chores

{coder_eval-0.11.4 → coder_eval-0.11.6}/CLAUDE.md

@@ -144,7 +144,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half.
-- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex and Antigravity (both run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), and `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it). Full table + rationale: docs/agents/HARNESS_PARITY.md.
+- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex and Antigravity (both run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `<root>/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
 - **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged.

{coder_eval-0.11.4 → coder_eval-0.11.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.11.4
+Version: 0.11.6
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -60,6 +60,7 @@ Requires-Dist: pytest>=9.0.2; extra == 'dev'
 Requires-Dist: ruff>=0.15.7; extra == 'dev'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.95.0; extra == 'litellm'
+Provides-Extra: opencode
 Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
@@ -77,14 +78,14 @@ Description-Content-Type: text/markdown
 **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
 **evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
-Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
+Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Not an "agentic coding" benchmark: it measures how
 effective your CLI and skills are when used by coding agents.
 
 Reach for it when you want to **test whether a Claude Code skill triggers**,
-**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
-prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,
+prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
 SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
 tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
 `skill_triggered` activation check, an A/B experiment layer, and per-tool cost
@@ -99,14 +100,14 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent abstraction** — Claude Code, Codex, and Antigravity (Gemini) today, extensible via a plugin SPI
+- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -183,8 +184,8 @@ That adds six slash commands: `/coder-eval:init`, `/coder-eval:check-skill`,
 A composite action — on the Marketplace as
 [**coder_eval**](https://github.com/marketplace/actions/coder_eval) — runs
 `coder-eval` as a CI gate. It installs the pinned CLI, runs your tasks, writes a
-JUnit XML report, appends `run.md` to the job summary, and fails the step on any
-task/gate failure:
+JUnit XML report, reports where its artifacts landed, and fails the step on any
+task failure:
 
 ```yaml
 - uses: actions/setup-node@v4      # the claude-code agent needs the Claude CLI…
@@ -192,35 +193,45 @@ task/gate failure:
 - run: npm install -g @anthropic-ai/claude-code
 
 - uses: UiPath/coder_eval@v0       # …then run the gate (@v1 once 1.0.0 ships; @vX.Y.Z pins exactly)
+  id: eval
   with:
-    tasks: tests/tasks/*.yaml tests/tasks/*/*.yaml
-    model: claude-sonnet-5
+    args: |
+      tests/tasks/**/*.yaml
+      --model
+      claude-sonnet-5
     env: |
       ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
 ```
 
+Eight inputs, and **none of them is a `coder-eval run` flag**. The CLI has 21;
+GitHub silently ignores an input the referenced tag does not define, so a
+forwarding input that is mistyped or newer than your pin yields a run that
+measured something else and still exits 0. A wrong CLI flag is a hard error. So
+flags and task globs all go through `args`, and an input exists only where the
+action does something with the value besides pass it along.
+
 | Input | Default | Purpose |
 | --- | --- | --- |
-| `tasks` | *(all `tasks/`)* | Task YAML path(s)/glob |
-| `tags` | — | `--tags` filter |
-| `model` | — | `--model` override |
-| `extra-args` | — | Verbatim extra args (`--experiment`, `-D …`, …) |
+| `args` | — | Task paths/globs and every flag for `coder-eval run`, one argument per line, verbatim |
 | `version` | pinned release | PyPI version, or `local` to install from the checkout |
-| `run-dir` | `runs/ci` | Run directory |
-| `junit-path` | `coder-eval-junit.xml` | Where to write the JUnit report |
-| `step-summary` | `true` | Append `run.md` to the job summary |
+| `extras` | — | coder-eval extras, composed into the install requirement (`codex`, `antigravity,litellm`) |
+| `extra-packages` | — | Extra requirements installed into coder-eval's environment (`--with`), one per line |
+| `install-flags` | — | Flags for `uv tool install`, one per line (`--prerelease=allow`, `--extra-index-url …`) |
 | `env` | — | Credentials/backend passthrough: newline-separated `NAME=VALUE` pairs, exported for the run step only |
-| `minimum-task-score` | *(off)* | Strict floor (0.0–1.0): fail the step if any task's `weighted_score` is below it |
+| `working-directory` | `.` | Directory every step of the action runs in |
+| `run-dir` | `runs/ci` | Run directory; also where the reports are written |
 
-Outputs: `run-dir` and `junit-path`. Feed the JUnit file to your platform's
-test-report renderer — e.g. on GitHub Actions with
-[`mikepenz/action-junit-report`](https://github.com/mikepenz/action-junit-report):
+Outputs: `run-dir`, `junit-path` (`<run-dir>/junit.xml`) and `run-md-path`
+(`<run-dir>/run.md`). The action writes nothing to the job summary — a consumer
+that has to redact the report first cannot undo a write that already happened:
 
 ```yaml
+- if: always()
+  run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
 - uses: mikepenz/action-junit-report@v5
   if: always()
   with:
-    report_paths: coder-eval-junit.xml
+    report_paths: ${{ steps.eval.outputs.junit-path }}
 ```
 
 **Credentials and backend config** are the sole responsibility of `env` — a
@@ -230,17 +241,13 @@ it can't leak into later steps). Set whatever the run needs, Anthropic or not:
 ```yaml
 - uses: UiPath/coder_eval@v0
   with:
-    tasks: tests/tasks/*.yaml tests/tasks/*/*.yaml
-    minimum-task-score: "0.8"   # fail the build if any task scores below 0.8
+    args: tests/tasks/**/*.yaml
     env: |
       API_BACKEND=bedrock
       AWS_BEARER_TOKEN_BEDROCK=${{ secrets.BEDROCK_TOKEN }}
 ```
 
-`minimum-task-score` is a strict floor **on top of** coder-eval's own exit
-code: the step fails if *either* coder-eval exits non-zero *or* any task's
-`weighted_score` falls below the floor. Leave it unset to gate on the exit code
-alone.
+The step's exit code is coder-eval's own: non-zero on any failed task.
 
 > **Agent runtime is the caller's responsibility.** The action is agent-agnostic —
 > it installs `coder-eval` but no coding-agent runtime, which is why the example
@@ -275,12 +282,13 @@ alone.
 | [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
+| [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |
 | [Dialog Mode](docs/DIALOG_MODE.md) | Evaluate agents in multi-turn conversation via a simulated user |
 | [Docker Isolation](docs/DOCKER_ISOLATION.md) | The container sandbox driver, with custom images |
-| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, score floor |
+| [CI Gate & GitHub Action](docs/CI_GATE.md) | Run Coder Eval as a CI gate — the Marketplace Action, JUnit output, run reports |
 | [Claude Code Plugin](docs/PLUGIN.md) | Install the Claude Code plugin — author, run, and analyze suites from inside the agent |
 | [Extending Coder Eval](docs/EXTENDING.md) | Author a custom agent, criterion, or model pricing via the plugin SPI |
 | [Report Schema](docs/REPORT_SCHEMA.md) | Field-level reference for run.json / variant.json / task.json |