codeer-cli 0.1.6__tar.gz → 0.1.8__tar.gz

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/API_REFERENCE.md

@@ -164,9 +164,11 @@ for the apply → test → publish workflow. Pass the draft `AgentHistory.id` fr
 | `GET /eval/evaluators?wid=<ws>` | List evaluators |
 | `PUT /eval/evaluators/{id}` | Update |
 | `DELETE /eval/evaluators/{id}` | Delete |
-| `POST /eval/trigger` | Run a set of cases with evaluators, optionally pinned to `agent_history_id` |
+| `POST /eval/case-evaluator-infos:batch` | Read assigned evaluators/rubrics for cases |
+| `PUT /eval/cases/{case_id}/case-evaluator-infos` | Replace assigned evaluators/rubrics for one case |
+| `POST /eval/trigger` | Run explicit assigned `case_evaluator_pairs` pinned to `agent_history_id` |
 | `POST /eval/stop` | Cancel running case+evaluator combo |
-| `POST /eval/rubric` | Set/override the rubric for one (case, evaluator) — write-only |
+| `POST /eval/rubric` | Set/override the rubric for one (case, evaluator); also creates assignment |
 | `POST /eval/rubrics/batch` | **Read** rubrics for a batch of (case, evaluator) pairs |
 
 Eval case labels are workspace-scoped reusable objects. The case create/update
@@ -220,17 +222,18 @@ normalized `tool_calls`, `tool_calls_summary`, and `tool_total_duration_ms`;
 in `eval_table_full.json`. Per-tool time is computed from `start_at/end_at`.
 
 **`evaluator_id` is singular — one call returns results for one evaluator
-only.** To see the full picture for a case, you must call this endpoint
-once per evaluator. A case might score 1.0 on Style/Tone but 0.3 on
-Content Compliance — checking only one evaluator hides the other failure.
-Always iterate all evaluators in the workspace (or at least all evaluators
-the agent's cases are judged by). `codeer eval run` and `codeer eval rubrics`
-handle this automatically; if calling the API directly, loop over
-`eval_mod.list_evaluators(workspace_id)` and call `get_results()` for each.
-
-Regression workflow (apply prompt change → re-run all cases → spot side
-effects): `codeer eval run --agent <agent_id>` runs the latest AgentHistory
-by default. Review the full result set against the previous run summary.
+only.** To see the full picture for a case, call results once per evaluator
+you care about. `codeer eval run` and `codeer eval rubrics` handle this
+automatically; when no evaluator is supplied, the public CLI uses external
+rubric batches to find case/evaluator pairs with configured rubrics.
+
+Regression workflow (apply prompt change → re-run all assigned pairs → spot
+side effects): `codeer eval run --agent <agent_id>` runs the latest
+AgentHistory by default. For the common "many cases, one tester" flow, use
+`codeer eval run --agent <agent_id> --cases <ids> --evaluator <evaluator_id>`.
+When triggering runs, call `POST /external/eval/runs` once per evaluator with
+that evaluator's case IDs. Do not call legacy internal trigger endpoints from
+the public CLI.
 
 ## Stage 7 — Publish
 
@@ -305,10 +308,12 @@ analytics/column name, `question` is the user-facing prompt.
 ### 3. The `Standard` shown in Test Suite is a per-(case, evaluator) rubric
 
 `POST /eval/cases` has a `rubric` field — this is NOT what the Test Suite's
-`Standard` column reads. That column is populated by `POST /eval/rubric` keyed
-on `(evaluation_case_id, evaluator_id)`. Set it explicitly for each
-(case, evaluator) pair after creating the case, or use the eval helpers in
-`codeer-cli/src/codeer_cli/eval_.py` which do both in one call.
+`Standard` column reads. That column is populated by `CaseEvaluatorInfo` keyed
+on `(evaluation_case_id, evaluator_id)`. The same row is also the assignment
+that makes the pair eligible to run. Set it explicitly for each
+(case, evaluator) pair, or use the eval helpers in
+`codeer-cli/src/codeer_cli/eval_.py` which create assignments and rubrics
+together.
 
 To **read** rubrics back, use `POST /eval/rubrics/batch` with
 `{case_ids: [...], evaluator_id}` — it returns the raw rubric strings out of
@@ -477,22 +482,22 @@ If the user's source tree has deeper nesting, flatten it first with the
 `products／a.md` using `／` U+FF0F as separator) so `list_kb_files` regex
 can still recover structure. That skill's docs explain the full flow.
 
-### 13. Eval results and rubrics are **per-evaluator** — always iterate all evaluators
+### 13. Eval results and rubrics are **per assigned case/evaluator pair**
 
 Both `POST /eval/results/batch` and `POST /eval/rubrics/batch` take a
 **singular** `evaluator_id`, not a list. Each call returns data for one
-evaluator only. A common mistake is to check results for one evaluator
-(e.g. Content Compliance), see a perfect score, and conclude the case is
-passing — while the other evaluator (e.g. Style/Tone) scored it 0.3.
+evaluator only. A case only runs with evaluators it is assigned to through
+`CaseEvaluatorInfo`.
 
 When pulling eval data manually, always:
-1. List all evaluators: `eval_mod.list_evaluators(workspace_id)`.
-2. Call `get_results()` or `get_rubrics_batch()` once **per evaluator**.
-3. Join the results by `case_id` to get the full (case × evaluator) matrix.
+1. Read assignments with `eval_mod.get_case_evaluator_infos(case_ids=[...])`.
+2. Call `get_results()` or `get_rubrics_batch()` once per assigned evaluator.
+3. Join the results by `(case_id, evaluator_id)`.
 
 The CLI commands (`codeer eval run`, `codeer eval rubrics`,
 `codeer eval rubrics-apply`) and the `get_case_rubrics()` helper all handle
-this iteration automatically. Prefer them for normal operations.
+this iteration automatically. Use `codeer eval rubrics --all-pairs` only when
+you intentionally want the old full matrix scan.
 
 ---
 

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: codeer-cli
-Version: 0.1.6
+Version: 0.1.8
 Summary: Command line tools for managing Codeer agents over the Codeer API.
 Project-URL: Homepage, https://www.codeer.ai
 Author: Codeer.AI
@@ -143,7 +143,7 @@ Use this pattern during agent lifecycle work:
 ```bash
 codeer agent list
 codeer history list --agent <agent-id> --limit 50
-codeer eval run --agent <agent-id> --evaluators <evaluator-id> --out .codeer/eval_run.json
+codeer eval run --agent <agent-id> --cases <case-ids> --evaluator <evaluator-id> --out .codeer/eval_run.json
 ```
 
 Flags:

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/README.md

@@ -125,7 +125,7 @@ Use this pattern during agent lifecycle work:
 ```bash
 codeer agent list
 codeer history list --agent <agent-id> --limit 50
-codeer eval run --agent <agent-id> --evaluators <evaluator-id> --out .codeer/eval_run.json
+codeer eval run --agent <agent-id> --cases <case-ids> --evaluator <evaluator-id> --out .codeer/eval_run.json
 ```
 
 Flags:

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "codeer-cli"
-version = "0.1.6"
+version = "0.1.8"
 description = "Command line tools for managing Codeer agents over the Codeer API."
 readme = "README.md"
 requires-python = ">=3.11"

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/src/codeer_cli/commands/_util.py

@@ -62,5 +62,7 @@ def print_json(value: Any) -> None:
 def write_json(path: str | None, value: Any) -> None:
     if not path:
         return
-    Path(path).write_text(json.dumps(value, ensure_ascii=False, indent=2, default=str) + "\n")
+    out = Path(path)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(json.dumps(value, ensure_ascii=False, indent=2, default=str) + "\n")
     log(f"wrote full detail to {path}")

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/src/codeer_cli/commands/eval_cmd.py

@@ -154,7 +154,9 @@ def register(subparsers):
     g.add_argument("--latest", action="store_true",
                    help="Auto-select the newest AgentHistory (default)")
     p.add_argument("--cases", default=None, help="Comma-separated case UUIDs (default: all)")
-    p.add_argument("--evaluators", required=True, help="Comma-separated evaluator UUIDs")
+    g = p.add_mutually_exclusive_group()
+    g.add_argument("--evaluator", default=None, help="Evaluator UUID; common path for running many cases with one tester")
+    g.add_argument("--evaluators", default=None, help="Comma-separated evaluator UUIDs")
     p.add_argument("--poll-timeout", type=int, default=POLL_TIMEOUT)
     p.add_argument("--full", action="store_true",
                    help="Use longer previews in stdout. Raw outputs/tool calls still require --out.")
@@ -195,10 +197,12 @@ def register(subparsers):
     p.set_defaults(func=run_cases_apply)
 
     # codeer eval rubrics
-    p = sub.add_parser("rubrics", help="Read per-(case, evaluator) rubrics")
+    p = sub.add_parser("rubrics", help="Read assigned per-(case, evaluator) rubrics")
     p.add_argument("--agent", required=True)
     p.add_argument("--evaluators", default=None, help="Comma-separated evaluator UUIDs")
     p.add_argument("--cases", default=None, help="Comma-separated case UUIDs")
+    p.add_argument("--all-pairs", action="store_true",
+                   help="With omitted --evaluators, scan every workspace evaluator instead of assigned pairs only.")
     p.add_argument("--full", action="store_true",
                    help="Print complete rubric text. Default prints matrix summaries/previews.")
     p.add_argument("--out", default=None,
@@ -599,6 +603,114 @@ def run_evaluator_update(args, client) -> int:
 # eval run
 # ---------------------------------------------------------------------------
 
+def _assigned_evaluators_by_case(info_rows: list[dict]) -> dict[str, dict[str, dict]]:
+    out: dict[str, dict[str, dict]] = {}
+    for row in info_rows:
+        case_id = row.get("case_id")
+        if not case_id:
+            continue
+        out[str(case_id)] = {
+            str(info.get("evaluator_id")): info
+            for info in (row.get("evaluators") or [])
+            if info.get("evaluator_id")
+        }
+    return out
+
+
+def _planned_eval_pairs(
+    *,
+    case_ids: list[str],
+    assigned_by_case: dict[str, dict[str, dict]],
+    requested_evaluator_ids: list[str] | None,
+) -> tuple[list[dict[str, str]], list[dict[str, str]]]:
+    pairs: list[dict[str, str]] = []
+    skipped: list[dict[str, str]] = []
+
+    if requested_evaluator_ids:
+        for case_id in case_ids:
+            assigned = assigned_by_case.get(case_id, {})
+            for evaluator_id in requested_evaluator_ids:
+                if evaluator_id in assigned:
+                    pairs.append({"case_id": case_id, "evaluator_id": evaluator_id})
+                else:
+                    skipped.append({
+                        "case_id": case_id,
+                        "evaluator_id": evaluator_id,
+                        "reason": "not_assigned",
+                    })
+        return pairs, skipped
+
+    for case_id in case_ids:
+        for evaluator_id in assigned_by_case.get(case_id, {}):
+            pairs.append({"case_id": case_id, "evaluator_id": evaluator_id})
+    return pairs, skipped
+
+
+def _group_case_ids_by_evaluator(pairs: list[dict[str, str]]) -> dict[str, list[str]]:
+    grouped: dict[str, list[str]] = defaultdict(list)
+    for pair in pairs:
+        grouped[pair["evaluator_id"]].append(pair["case_id"])
+    return dict(grouped)
+
+
+def _pairs_from_rubric_batches(
+    client,
+    *,
+    case_ids: list[str],
+    evaluator_ids: list[str],
+) -> list[dict[str, str]]:
+    pairs: list[dict[str, str]] = []
+    for evaluator_id in evaluator_ids:
+        for row in eval_mod.get_rubrics_batch(client, case_ids=case_ids, evaluator_id=evaluator_id):
+            if row.get("rubric"):
+                case_id = row.get("case_id") or row.get("evaluation_case_id")
+                if case_id:
+                    pairs.append({"case_id": str(case_id), "evaluator_id": evaluator_id})
+    return pairs
+
+
+def _pair_key(pair: dict[str, str]) -> tuple[str, str]:
+    return pair["case_id"], pair["evaluator_id"]
+
+
+def _skipped_pairs_from_trigger_response(response: Any) -> list[dict[str, str]]:
+    if not isinstance(response, dict):
+        return []
+    payload = response.get("data") if isinstance(response.get("data"), dict) else response
+    skipped = payload.get("skipped_pairs") if isinstance(payload, dict) else None
+    if not isinstance(skipped, list):
+        return []
+
+    out: list[dict[str, str]] = []
+    for row in skipped:
+        if not isinstance(row, dict):
+            continue
+        case_id = row.get("case_id")
+        evaluator_id = row.get("evaluator_id")
+        if not case_id or not evaluator_id:
+            continue
+        out.append({
+            "case_id": str(case_id),
+            "evaluator_id": str(evaluator_id),
+            "reason": str(row.get("reason") or "skipped"),
+        })
+    return out
+
+
+def _remove_non_runnable_skipped_pairs(
+    pairs: list[dict[str, str]],
+    skipped_pairs: list[dict[str, str]],
+) -> list[dict[str, str]]:
+    non_runnable = {
+        _pair_key(pair)
+        for pair in skipped_pairs
+        if pair.get("reason") == "not_assigned"
+    }
+    if not non_runnable:
+        return pairs
+    return [pair for pair in pairs if _pair_key(pair) not in non_runnable]
+
+
 def run_run(args, client) -> int:
     workspace_id, _ = client.resolve_scope()
     if args.latest or not args.history:
@@ -625,41 +737,113 @@ def run_run(args, client) -> int:
         log("error: no cases to run")
         return 2
 
-    evaluator_ids = _ids(args.evaluators) or []
-    if not evaluator_ids:
-        log("error: --evaluators is required")
+    evaluator_ids = [args.evaluator] if args.evaluator else (_ids(args.evaluators) or [])
+    requested_evaluator_ids = evaluator_ids or None
+
+    skipped_unassigned: list[dict[str, str]] = []
+    if requested_evaluator_ids:
+        pairs = [
+            {"case_id": case_id, "evaluator_id": evaluator_id}
+            for evaluator_id in requested_evaluator_ids
+            for case_id in case_ids
+        ]
+    else:
+        evaluator_ids = [e["id"] for e in eval_mod.list_evaluators(client, workspace_id)]
+        pairs = _pairs_from_rubric_batches(
+            client,
+            case_ids=case_ids,
+            evaluator_ids=evaluator_ids,
+        )
+    if not pairs:
+        log("error: no case/evaluator pairs to run")
+        print_json({
+            "agent_id": args.agent,
+            "history_id": args.history,
+            "requested_case_count": len(case_ids),
+            "requested_evaluator_count": len(evaluator_ids),
+            "triggered_pair_count": 0,
+            "skipped_unassigned": skipped_unassigned,
+        })
         return 2
+
+    evaluator_ids = _dedupe_preserve_order([pair["evaluator_id"] for pair in pairs])
     evaluators = [eval_mod.get_evaluator(client, eid) for eid in evaluator_ids]
 
     case_label_by_id = {c["id"]: truncate(c.get("input") or "", 60) for c in case_objs}
     evaluator_name_by_id = {e["id"]: e.get("name", e["id"]) for e in evaluators}
 
-    log(f"triggering: {len(case_ids)} cases x {len(evaluator_ids)} evaluators on history {args.history}")
-    eval_mod.trigger(client, case_ids=case_ids, evaluator_ids=evaluator_ids,
-                     agent_history_id=args.history)
+    requested_pairs = list(pairs)
+    requested_pair_count = len(requested_pairs)
+    log(f"triggering: {requested_pair_count} case/evaluator pairs on history {args.history}")
+    trigger_response: list[dict[str, Any]] = []
+    skipped_pairs: list[dict[str, str]] = []
+    for ev_id, ev_case_ids in _group_case_ids_by_evaluator(pairs).items():
+        response = eval_mod.trigger(
+            client,
+            case_ids=ev_case_ids,
+            evaluator_ids=[ev_id],
+            agent_history_id=args.history,
+        )
+        response_skipped = _skipped_pairs_from_trigger_response(response)
+        skipped_pairs.extend(response_skipped)
+        trigger_response.append({
+            "evaluator_id": ev_id,
+            "case_ids": ev_case_ids,
+            "response": response,
+            "skipped_pairs": response_skipped,
+        })
+
+    pairs = _remove_non_runnable_skipped_pairs(pairs, skipped_pairs)
+    skipped_unassigned = [pair for pair in skipped_pairs if pair.get("reason") == "not_assigned"]
+    if skipped_unassigned:
+        log(f"skipping {len(skipped_unassigned)} not-assigned pairs from polling")
+    if not pairs:
+        log("error: no runnable case/evaluator pairs after trigger response")
+        print_json({
+            "agent_id": args.agent,
+            "history_id": args.history,
+            "requested_case_count": len(case_ids),
+            "requested_evaluator_count": len(requested_evaluator_ids or evaluator_ids),
+            "requested_pair_count": requested_pair_count,
+            "triggered_pair_count": 0,
+            "skipped_pair_count": len(skipped_pairs),
+            "skipped_unassigned_count": len(skipped_unassigned),
+            "trigger_response": trigger_response,
+            "skipped_pairs": skipped_pairs,
+            "skipped_unassigned": skipped_unassigned,
+        })
+        return 2
 
     deadline = time.time() + args.poll_timeout
     results_by_eval: dict[str, list[dict]] = {}
+    case_ids_by_evaluator = _group_case_ids_by_evaluator(pairs)
+    target_pair_keys = {(pair["case_id"], pair["evaluator_id"]) for pair in pairs}
     while time.time() < deadline:
         results_by_eval = {}
-        done = 0
-        total = len(case_ids) * len(evaluator_ids)
-        for ev_id in evaluator_ids:
+        done_pairs: set[tuple[str, str]] = set()
+        total = len(pairs)
+        for ev_id, ev_case_ids in case_ids_by_evaluator.items():
             rows = eval_mod.get_results(
-                client, case_ids=case_ids, evaluator_id=ev_id,
+                client, case_ids=ev_case_ids, evaluator_id=ev_id,
                 agent_history_id=args.history, workspace_id=workspace_id,
                 include_output=True, include_reasoning_steps=True,
             )
             results_by_eval[ev_id] = rows
-            done += sum(1 for r in rows if r.get("score") is not None)
-        log(f"  progress: {done}/{total}")
-        if done >= total:
+            for row in rows:
+                key = (row.get("case_id") or row.get("evaluation_case_id"), ev_id)
+                if key in target_pair_keys and row.get("score") is not None:
+                    done_pairs.add(key)
+        log(f"  progress: {len(done_pairs)}/{total}")
+        if len(done_pairs) >= total:
             break
         time.sleep(POLL_INTERVAL)
 
     flat: list[dict] = []
     for ev_id, rows in results_by_eval.items():
         for r in rows:
+            row_case_id = r.get("case_id") or r.get("evaluation_case_id")
+            if (row_case_id, ev_id) not in target_pair_keys:
+                continue
             result_summary = parse_eval_result(r)
             tool_calls = parse_eval_tool_calls(r)
             total_tool_duration_ms = sum(
@@ -684,7 +868,15 @@ def run_run(args, client) -> int:
                 "raw_result": r,
             })
 
-    all_perfect = all((r.get("score") or 0.0) >= 1.0 for r in flat) if flat else False
+    scored_pair_keys = {
+        (r.get("case_id"), r.get("evaluator_id"))
+        for r in flat
+        if r.get("score") is not None
+    }
+    all_perfect = (
+        len(scored_pair_keys) == len(target_pair_keys)
+        and all((r.get("score") or 0.0) >= 1.0 for r in flat)
+    )
     log("\n" + "=" * 80)
     log(f"RESULTS  agent={args.agent}  history={args.history}")
     log("=" * 80)
@@ -726,15 +918,30 @@ def run_run(args, client) -> int:
     out = {
         "agent_id": args.agent,
         "history_id": args.history,
+        "requested_case_count": len(case_ids),
+        "requested_evaluator_count": len(requested_evaluator_ids or evaluator_ids),
+        "requested_pair_count": requested_pair_count,
+        "triggered_pair_count": len(pairs),
+        "scored_pair_count": len(scored_pair_keys),
+        "skipped_pair_count": len(skipped_pairs),
+        "skipped_unassigned_count": len(skipped_unassigned),
         "all_perfect": all_perfect,
         "result_count": len(result_summaries),
         "non_perfect_count": len(non_perfect),
         "wrote_full_detail": bool(args.out),
+        "trigger_response": trigger_response,
+        "skipped_pairs": skipped_pairs,
+        "skipped_unassigned": skipped_unassigned,
         "results": result_summaries,
     }
     full_out = {
         "agent_id": args.agent,
         "history_id": args.history,
+        "requested_pairs": requested_pairs,
+        "triggered_pairs": pairs,
+        "trigger_response": trigger_response,
+        "skipped_pairs": skipped_pairs,
+        "skipped_unassigned": skipped_unassigned,
         "all_perfect": all_perfect,
         "results": flat,
     }
@@ -1365,17 +1572,37 @@ def run_rubrics(args, client) -> int:
     else:
         evaluators = eval_mod.list_evaluators(client, workspace_id)
         evaluator_ids = [e["id"] for e in evaluators]
-    evaluator_name = {e["id"]: e.get("name", e["id"]) for e in evaluators}
-    if not evaluator_ids:
-        log("error: no evaluators in workspace")
-        return 2
-
-    log(f"reading {len(case_ids)} cases x {len(evaluator_ids)} evaluators...")
 
     rubrics = eval_mod.get_case_rubrics(
         client, agent_id=args.agent, workspace_id=workspace_id,
         evaluator_ids=evaluator_ids, case_ids=case_ids,
     )
+    assigned_by_case = {
+        cid: {
+            ev_id: {"evaluator_id": ev_id, "rubric": rubric_text}
+            for ev_id, rubric_text in (rubrics.get(cid) or {}).items()
+            if rubric_text
+        }
+        for cid in case_ids
+    }
+
+    if args.evaluators or args.all_pairs:
+        pass
+    else:
+        evaluator_ids = _dedupe_preserve_order([
+            evaluator_id
+            for case_id in case_ids
+            for evaluator_id in assigned_by_case.get(case_id, {})
+        ])
+        evaluator_id_set = set(evaluator_ids)
+        evaluators = [e for e in evaluators if e["id"] in evaluator_id_set]
+    evaluator_name = {e["id"]: e.get("name", e["id"]) for e in evaluators}
+    if not evaluator_ids:
+        log("error: no evaluators with configured rubrics for these cases")
+        return 2
+
+    mode = "all requested pairs" if args.evaluators or args.all_pairs else "pairs with configured rubrics"
+    log(f"reading {mode}: {len(case_ids)} cases, {len(evaluator_ids)} evaluators...")
 
     if args.full:
         for cid in case_ids:
@@ -1383,8 +1610,14 @@ def run_rubrics(args, client) -> int:
             log(f"CASE {cid}")
             log(f"  input: {truncate(case_input.get(cid, ''), 120)}")
             for ev_id in evaluator_ids:
+                is_assigned = ev_id in assigned_by_case.get(cid, {})
+                if not is_assigned and not (args.evaluators or args.all_pairs):
+                    continue
                 ev_name = evaluator_name.get(ev_id, ev_id)
                 rubric_text = (rubrics.get(cid) or {}).get(ev_id, "")
+                if not is_assigned:
+                    log(f"  [{ev_name}] (not assigned)")
+                    continue
                 if not rubric_text:
                     log(f"  [{ev_name}] (rubric not set)")
                 else:
@@ -1396,9 +1629,13 @@ def run_rubrics(args, client) -> int:
     for cid in case_ids:
         rubrics_summary = {}
         for ev_id in evaluator_ids:
+            is_assigned = ev_id in assigned_by_case.get(cid, {})
+            if not is_assigned and not (args.evaluators or args.all_pairs):
+                continue
             rubric_text = (rubrics.get(cid) or {}).get(ev_id, "")
             rubrics_summary[ev_id] = {
                 "evaluator_name": evaluator_name.get(ev_id, ev_id),
+                "is_assigned": is_assigned,
                 "is_set": bool(rubric_text),
                 "chars": len(rubric_text),
                 "preview": truncate(rubric_text, 240),
@@ -1412,6 +1649,7 @@ def run_rubrics(args, client) -> int:
     out = {
         "agent_id": args.agent,
         "workspace_id": workspace_id,
+        "mode": mode,
         "evaluators": [{"id": e["id"], "name": e.get("name")} for e in evaluators],
         "cases": [
             {
@@ -1420,7 +1658,9 @@ def run_rubrics(args, client) -> int:
                 "rubrics_by_evaluator": {
                     ev_id: (rubrics.get(cid) or {}).get(ev_id)
                     for ev_id in evaluator_ids
+                    if ev_id in assigned_by_case.get(cid, {}) or args.evaluators or args.all_pairs
                 },
+                "assigned_evaluator_ids": list(assigned_by_case.get(cid, {})),
             }
             for cid in case_ids
         ],
@@ -1432,6 +1672,7 @@ def run_rubrics(args, client) -> int:
         print_json({
             "agent_id": args.agent,
             "workspace_id": workspace_id,
+            "mode": mode,
             "evaluator_count": len(evaluators),
             "case_count": len(case_ids),
             "wrote_full_detail": bool(args.out),

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/src/codeer_cli/eval_.py

@@ -23,6 +23,7 @@ def create_case(
     rubric: Optional[str] = None,
     attachment_ids: Optional[List[str]] = None,
     label_ids: Optional[List[str]] = None,
+    evaluators: Optional[List[dict[str, Any]]] = None,
     meta: Optional[dict] = None,
     note: Optional[str] = None,
 ) -> dict:
@@ -44,6 +45,8 @@ def create_case(
         body["attachment_ids"] = attachment_ids
     if label_ids is not None:
         body["label_ids"] = label_ids
+    if evaluators is not None:
+        body["evaluators"] = evaluators
     if meta:
         body["meta"] = meta
     if note is not None:
@@ -51,8 +54,26 @@ def create_case(
     return client.post("/external/eval/cases", json=body)
 
 
+def _unwrap_list_response(value: Any, *keys: str) -> list[dict]:
+    """Normalize list endpoints that may return either a bare list or envelope."""
+    if isinstance(value, list):
+        return value
+    if isinstance(value, dict):
+        for key in keys:
+            rows = value.get(key)
+            if isinstance(rows, list):
+                return rows
+    return []
+
+
 def list_cases(client: CodeerClient, agent_id: str) -> list[dict]:
-    return client.get(f"/external/eval/agents/{agent_id}/cases")
+    return _unwrap_list_response(
+        client.get(f"/external/eval/agents/{agent_id}/cases"),
+        "cases",
+        "evaluation_cases",
+        "data",
+        "items",
+    )
 
 
 def get_case(client: CodeerClient, case_id: str) -> dict:
@@ -93,6 +114,31 @@ def delete_case(client: CodeerClient, case_id: str) -> dict:
     return client.delete(f"/external/eval/cases/{case_id}")
 
 
+# --- case/evaluator assignments ------------------------------------------
+
+def get_case_evaluator_infos(client: CodeerClient, *, case_ids: List[str]) -> list[dict]:
+    """Read assigned evaluator metadata for each case.
+
+    Returns rows shaped like ``{"case_id": str, "evaluators": [...]}``, where
+    each evaluator entry is the assigned ``{"evaluator_id", "rubric"}`` pair.
+    """
+    return client.post("/eval/case-evaluator-infos:batch", json={"case_ids": case_ids})
+
+
+def replace_case_evaluator_infos(
+    client: CodeerClient,
+    *,
+    case_id: str,
+    evaluators: list[dict[str, Any]],
+) -> dict:
+    """Replace a case's assigned evaluators.
+
+    This is intentionally separate from rubric upsert: replacing removes
+    evaluator assignments that are not included in ``evaluators``.
+    """
+    return client.put(f"/eval/cases/{case_id}/case-evaluator-infos", json={"evaluators": evaluators})
+
+
 # --- case labels -----------------------------------------------------------
 
 def list_case_labels(client: CodeerClient, *, workspace_id: str) -> list[dict]:
@@ -202,6 +248,22 @@ def trigger(
     return client.post("/external/eval/runs", json=body)
 
 
+def trigger_pairs(
+    client: CodeerClient,
+    *,
+    case_evaluator_pairs: list[dict[str, str]],
+    agent_history_id: str,
+) -> dict:
+    """Kick off evaluation for explicit assigned case/evaluator pairs."""
+    return client.post(
+        "/eval/trigger",
+        json={
+            "case_evaluator_pairs": case_evaluator_pairs,
+            "agent_history_id": agent_history_id,
+        },
+    )
+
+
 def stop(client: CodeerClient, *, case_id: str, evaluator_id: str) -> Any:
     return client.post("/external/eval/runs:stop", json={"case_id": case_id, "evaluator_id": evaluator_id})
 
@@ -451,9 +513,9 @@ def create_case_with_rubrics(
     is filled in for every evaluator it will be judged by.
 
     ``rubrics_by_evaluator`` maps ``evaluator_id → rubric_text``. Each entry
-    becomes a ``POST /eval/rubric`` call after the case is created. Use
-    different rubric wording per evaluator when the evaluators judge different
-    aspects (e.g. Style/Tone vs Content Compliance).
+    becomes a case/evaluator assignment on create. Use different rubric wording
+    per evaluator when the evaluators judge different aspects (e.g. Style/Tone
+    vs Content Compliance).
     """
     case = create_case(
         client,
@@ -462,12 +524,11 @@ def create_case_with_rubrics(
         expected_output=expected_output,
         attachment_ids=attachment_ids,
         label_ids=label_ids,
+        evaluators=[
+            {"evaluator_id": ev_id, "rubric": rubric}
+            for ev_id, rubric in rubrics_by_evaluator.items()
+        ],
         meta=meta,
         note=note,
     )
-    set_rubric_bulk(
-        client,
-        evaluation_case_id=case["id"],
-        rubrics_by_evaluator=rubrics_by_evaluator,
-    )
     return case

codeer_cli-0.1.8/tests/test_eval_pairs.py

@@ -0,0 +1,218 @@
+from __future__ import annotations
+
+import unittest
+
+from codeer_cli import eval_ as eval_mod
+from codeer_cli.commands.eval_cmd import (
+    _assigned_evaluators_by_case,
+    _pairs_from_rubric_batches,
+    _planned_eval_pairs,
+    _remove_non_runnable_skipped_pairs,
+    _skipped_pairs_from_trigger_response,
+)
+
+
+class FakeClient:
+    def __init__(self) -> None:
+        self.calls: list[tuple[str, str, dict]] = []
+
+    def get(self, path: str, **kwargs):
+        self.calls.append(("GET", path, kwargs))
+        if path == "/external/eval/agents/agent-1/cases":
+            return {"cases": [{"id": "case-1"}, {"id": "case-2"}], "total": 2}
+        return []
+
+    def post(self, path: str, **kwargs):
+        self.calls.append(("POST", path, kwargs))
+        if path == "/external/eval/cases":
+            return {"id": "case-1"}
+        if path == "/external/eval/rubrics:batch":
+            return [
+                {"case_id": "case-1", "evaluator_id": "eval-1", "rubric": "Must pass"},
+                {"case_id": "case-2", "evaluator_id": "eval-1", "rubric": ""},
+            ]
+        return {"ok": True}
+
+    def put(self, path: str, **kwargs):
+        self.calls.append(("PUT", path, kwargs))
+        return {"ok": True}
+
+
+class EvalPairClientTests(unittest.TestCase):
+    def test_list_cases_accepts_enveloped_server_response(self) -> None:
+        client = FakeClient()
+
+        cases = eval_mod.list_cases(client, "agent-1")  # type: ignore[arg-type]
+
+        self.assertEqual(cases, [{"id": "case-1"}, {"id": "case-2"}])
+        self.assertEqual(client.calls[0][0:2], ("GET", "/external/eval/agents/agent-1/cases"))
+
+    def test_external_trigger_endpoint_for_agent_history_runs(self) -> None:
+        client = FakeClient()
+
+        eval_mod.trigger(  # type: ignore[arg-type]
+            client,
+            case_ids=["case-1", "case-2"],
+            evaluator_ids=["eval-1"],
+            agent_history_id="hist-1",
+        )
+
+        self.assertEqual(client.calls[0][0:2], ("POST", "/external/eval/runs"))
+        self.assertEqual(
+            client.calls[0][2]["json"],
+            {
+                "case_ids": ["case-1", "case-2"],
+                "evaluator_ids": ["eval-1"],
+                "version_id": "hist-1",
+            },
+        )
+
+    def test_pairs_from_rubric_batches_keeps_configured_rubrics_only(self) -> None:
+        client = FakeClient()
+
+        pairs = _pairs_from_rubric_batches(  # type: ignore[arg-type]
+            client,
+            case_ids=["case-1", "case-2"],
+            evaluator_ids=["eval-1"],
+        )
+
+        self.assertEqual(pairs, [{"case_id": "case-1", "evaluator_id": "eval-1"}])
+        self.assertEqual(client.calls[0][0:2], ("POST", "/external/eval/rubrics:batch"))
+
+    def test_assignment_helper_paths(self) -> None:
+        client = FakeClient()
+
+        eval_mod.get_case_evaluator_infos(client, case_ids=["case-1"])  # type: ignore[arg-type]
+        eval_mod.replace_case_evaluator_infos(  # type: ignore[arg-type]
+            client,
+            case_id="case-1",
+            evaluators=[{"evaluator_id": "eval-1", "rubric": "Must pass"}],
+        )
+        eval_mod.trigger_pairs(  # type: ignore[arg-type]
+            client,
+            case_evaluator_pairs=[{"case_id": "case-1", "evaluator_id": "eval-1"}],
+            agent_history_id="hist-1",
+        )
+
+        self.assertEqual(client.calls[0][0:2], ("POST", "/eval/case-evaluator-infos:batch"))
+        self.assertEqual(client.calls[0][2]["json"], {"case_ids": ["case-1"]})
+        self.assertEqual(client.calls[1][0:2], ("PUT", "/eval/cases/case-1/case-evaluator-infos"))
+        self.assertEqual(
+            client.calls[1][2]["json"],
+            {"evaluators": [{"evaluator_id": "eval-1", "rubric": "Must pass"}]},
+        )
+        self.assertEqual(client.calls[2][0:2], ("POST", "/eval/trigger"))
+        self.assertEqual(
+            client.calls[2][2]["json"],
+            {
+                "case_evaluator_pairs": [{"case_id": "case-1", "evaluator_id": "eval-1"}],
+                "agent_history_id": "hist-1",
+            },
+        )
+
+    def test_create_case_with_rubrics_sends_atomic_evaluator_assignments(self) -> None:
+        client = FakeClient()
+
+        eval_mod.create_case_with_rubrics(  # type: ignore[arg-type]
+            client,
+            agent_id="agent-1",
+            input="Question",
+            rubrics_by_evaluator={"eval-1": "Must pass", "eval-2": "Must cite"},
+        )
+
+        self.assertEqual(len(client.calls), 1)
+        self.assertEqual(client.calls[0][0:2], ("POST", "/external/eval/cases"))
+        self.assertEqual(
+            client.calls[0][2]["json"]["evaluators"],
+            [
+                {"evaluator_id": "eval-1", "rubric": "Must pass"},
+                {"evaluator_id": "eval-2", "rubric": "Must cite"},
+            ],
+        )
+
+
+class EvalPairPlannerTests(unittest.TestCase):
+    def test_plans_only_assigned_pairs_for_specific_evaluator(self) -> None:
+        rows = [
+            {"case_id": "case-1", "evaluators": [{"evaluator_id": "eval-1", "rubric": "A"}]},
+            {"case_id": "case-2", "evaluators": [{"evaluator_id": "eval-2", "rubric": "B"}]},
+        ]
+
+        pairs, skipped = _planned_eval_pairs(
+            case_ids=["case-1", "case-2"],
+            assigned_by_case=_assigned_evaluators_by_case(rows),
+            requested_evaluator_ids=["eval-1"],
+        )
+
+        self.assertEqual(pairs, [{"case_id": "case-1", "evaluator_id": "eval-1"}])
+        self.assertEqual(
+            skipped,
+            [{"case_id": "case-2", "evaluator_id": "eval-1", "reason": "not_assigned"}],
+        )
+
+    def test_omitted_evaluator_runs_all_assigned_pairs(self) -> None:
+        rows = [
+            {
+                "case_id": "case-1",
+                "evaluators": [
+                    {"evaluator_id": "eval-1", "rubric": "A"},
+                    {"evaluator_id": "eval-2", "rubric": "B"},
+                ],
+            },
+            {"case_id": "case-2", "evaluators": []},
+        ]
+
+        pairs, skipped = _planned_eval_pairs(
+            case_ids=["case-1", "case-2"],
+            assigned_by_case=_assigned_evaluators_by_case(rows),
+            requested_evaluator_ids=None,
+        )
+
+        self.assertEqual(
+            pairs,
+            [
+                {"case_id": "case-1", "evaluator_id": "eval-1"},
+                {"case_id": "case-1", "evaluator_id": "eval-2"},
+            ],
+        )
+        self.assertEqual(skipped, [])
+
+    def test_trigger_response_skipped_pairs_are_normalized(self) -> None:
+        skipped = _skipped_pairs_from_trigger_response({
+            "skipped_pairs": [
+                {"case_id": "case-1", "evaluator_id": "eval-1", "reason": "not_assigned"},
+                {"case_id": "case-2", "evaluator_id": "eval-1"},
+                {"case_id": None, "evaluator_id": "eval-1", "reason": "not_assigned"},
+            ]
+        })
+
+        self.assertEqual(
+            skipped,
+            [
+                {"case_id": "case-1", "evaluator_id": "eval-1", "reason": "not_assigned"},
+                {"case_id": "case-2", "evaluator_id": "eval-1", "reason": "skipped"},
+            ],
+        )
+
+    def test_non_runnable_skipped_pairs_are_removed_from_poll_targets(self) -> None:
+        pairs = [
+            {"case_id": "case-1", "evaluator_id": "eval-1"},
+            {"case_id": "case-2", "evaluator_id": "eval-1"},
+            {"case_id": "case-3", "evaluator_id": "eval-1"},
+        ]
+        skipped = [
+            {"case_id": "case-2", "evaluator_id": "eval-1", "reason": "not_assigned"},
+            {"case_id": "case-3", "evaluator_id": "eval-1", "reason": "already_running"},
+        ]
+
+        self.assertEqual(
+            _remove_non_runnable_skipped_pairs(pairs, skipped),
+            [
+                {"case_id": "case-1", "evaluator_id": "eval-1"},
+                {"case_id": "case-3", "evaluator_id": "eval-1"},
+            ],
+        )
+
+
+if __name__ == "__main__":
+    unittest.main()

codeer_cli-0.1.8/tests/test_util.py

@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+from codeer_cli.commands._util import write_json
+
+
+class WriteJsonTests(unittest.TestCase):
+    def test_write_json_creates_parent_directories(self) -> None:
+        with tempfile.TemporaryDirectory() as tmp:
+            out = Path(tmp) / "nested" / "artifact.json"
+
+            write_json(str(out), {"ok": True})
+
+            self.assertEqual(json.loads(out.read_text()), {"ok": True})
+
+
+if __name__ == "__main__":
+    unittest.main()

{codeer_cli-0.1.6 → codeer_cli-0.1.8}/uv.lock

@@ -26,7 +26,7 @@ wheels = [
 
 [[package]]
 name = "codeer-cli"
-version = "0.1.6"
+version = "0.1.8"
 source = { editable = "." }
 dependencies = [
     { name = "httpx" },