@nexus-cortex/server 4.38.0 → 4.40.0

package/.cortex/agents/autoresearch-agent.md

@@ -40,9 +40,21 @@ Your prompt contains:
 ## EXECUTION MODE: mcp  (offload to the hosted harness)
 Use the configured **auto-research MCP** tools instead of the internal CLI: submit the experiment, poll its status, read the verdict. Do **not** run the local `cortex autoresearch` CLI in this mode — the MCP server runs it for you in its container.
 
+## Your candidate is also JUDGED — fix on the merits
+A passing score is **necessary but not sufficient**. After the statistical gate accepts a
+candidate, an independent **LLM judge** reads its `diff` (`cortex autoresearch judge`) and
+can VETO the merge — the hosted campaign runs this judge by default; the loop runs it under
+`--require-judge`. The judge rejects candidates that **game the eval** (hardcode/special-case
+expected outputs, branch on test inputs, edit the verifier/test files) or **smuggle in
+damage** (hallucinated/nonexistent APIs, unsafe shell-outs / network calls / filesystem
+destruction, backdoor- or exfiltration-looking code, unrelated churn). So make the
+**smallest, real fix of the underlying cause** that would generalize to unseen inputs — do
+not chase the metric with tricks or add anything the deficiency did not require. A clean
+minimal diff survives both gates.
+
 ## Discipline (do not break these)
 - **fixed ≠ verified.** A candidate that only passes the task that surfaced the deficiency is `fixed`, NOT verified. Only a HELD-OUT confirmation makes it `verified`.
-- **Do not self-merge.** Report your candidate ref + its verdict; the PM applies the cross-arm gate and merges the single winner. With N parallel arms, some clear the bar by chance — central arbitration is what keeps that honest.
+- **Do not self-merge.** Report your candidate ref + its verdict; the PM applies the cross-arm statistical gate AND the judge, then merges the single winner. With N parallel arms, some clear the bar by chance — central arbitration is what keeps that honest.
 - **Stay isolated.** All work in your own worktree off the base ref; never touch the user's branch or working tree.
 
 ## Report back

package/.cortex/skills/autoresearch/SKILL.md

@@ -46,7 +46,7 @@ Identical agents on identical prompts waste the parallelism — they trace the s
 
 ## 3. DELEGATE (pick the execution path by how you're accessed)
 - **Local cortex harness** (you're driving cortex, or inside it): set `AUTORESEARCH_AGENTS=native` and delegate via the **Task tool** (`subagent_type: autoresearch-agent`), one per strategy, each prompt = the plan + that arm's persona/strategy + `EXECUTION MODE: native`. Or drive the CLI directly: `cortex autoresearch fix` / `experiment` / `loop`.
-- **Hosted at scale** (external agent): a hosted auto-research MCP is planned (ships after the npm release). When one is configured, relay the plan to its tools — `EXECUTION MODE: mcp`.
+- **Hosted at scale** (external agent): the hosted **`nexus-autoresearch` MCP** is LIVE. Relay the plan to its tools (`start_autoresearch_campaign` / `autoresearch_experiment` / `autoresearch_fix`) — `EXECUTION MODE: mcp`. Hosted campaigns run the **LLM judge gate ON by default** (see §4); pass `requireJudge:false` to disable or `judgeRubric` to customize.
 
 The agents EXPLORE; they do not merge. They each return a candidate + its verdict.
 
@@ -55,7 +55,8 @@ Collect every candidate + verdict and keep **only the holdout-verified winner**:
 - **fixed ≠ verified.** A candidate that only passes the task that surfaced the deficiency is `fixed`. It is `verified` ONLY after a **held-out** set it was never tuned against confirms it.
 - **N-aware significance.** With N parallel arms some clear the bar by chance — the gate's family-wise-error (FWER) correction handles this; apply it across *all* arms (including the discarded ones). A single arm "winning" is not enough on its own.
 - **You arbitrate; the arms don't self-merge.** This central single-judge step is what makes aggressive diversity safe.
-- The gate is deterministic code (`cortex autoresearch evaluate` / `AutoResearchGate`) — never an LLM deciding significance.
+- The **statistical** gate is deterministic code (`cortex autoresearch evaluate` / `AutoResearchGate`) — never an LLM deciding significance.
+- **Two gates, not one — add the qualitative judge.** The statistical gate measures whether the *scores* improved; it is blind to *how*. A candidate can pass every statistical check by **gaming the eval** (hardcoding outputs, branching on test inputs, editing the verifier) or by **smuggling damage** (a hallucinated import, an unsafe shell-out, an exfiltration backdoor, unrelated churn) alongside a real fix. The **LLM judge gate** (`cortex autoresearch judge`) reads the candidate's *diff* and vetoes those — so the merge rule is `accept = mergeEligible ∧ judge-approve`. It's opt-in on the CLI (`--require-judge` / `--judge-rubric`) and **ON by default** on hosted MCP campaigns. This is the qualitative complement to the statistical gate — not a replacement: a score gain is necessary but NOT sufficient.
 
 ## 5. Discipline (the overfitting guards — load-bearing)
 - **Human owns the metric.** You (or the operator) define success; the agents optimize against it. An agent that chooses its own metric games the eval.

package/.cortex/skills/cortex-bench/SKILL.md

@@ -279,6 +279,8 @@ cortex autoresearch bench --task-set <file|dir of *.json> --experiment-tag <id>
 ```
 Task = `{id, prompt, verifier, taskType?}`; verifier ∈ `exact|regex|contains|llm-judge`. **Prefer `contains` (partial credit → continuous score) or graded rubrics** over binary exact/regex — the bootstrap/permutation gate separates arms far better on continuous scores. Sample: `.cortex/bench/tasks/sample-tasks.json`. Run it in the base build and the candidate build (different `--harness-ref`/worktree), then `cortex autoresearch evaluate`. Keep holdout task FILES out of any fixing agent's context (overfitting guard).
 
+> **Two different "judges" — don't conflate them.** The `llm-judge` *verifier* above is **task-level**: it scores one task's OUTPUT against a rubric, producing a number that feeds the statistical gate. The autoresearch **judge gate** (`cortex autoresearch judge`, `loop --require-judge`) is **candidate-level**: it reads the whole candidate *diff* and approves/vetoes the MERGE (`accept = mergeEligible ∧ judge-approve`), catching eval-gaming and unsafe code the per-task scores cannot see. Orthogonal roles: one grades outputs, the other gates merges.
+
 **The one-shot runner — `cortex autoresearch experiment` (v4.7.0).** Does the whole single-experiment loop in one call (build+serve both arms → bench train+holdout → gate → `verifyOnHoldout` → teardown), so you don't orchestrate `bench`×2 + `evaluate` by hand:
 ```
 cortex autoresearch experiment --experiment-tag <id> \

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@nexus-cortex/server",
-  "version": "4.38.0",
+  "version": "4.40.0",
   "description": "Thin Express server wrapper for Nexus Cortex core library",
   "type": "module",
   "main": "dist/index.js",
@@ -20,8 +20,8 @@
     "prepack": "node ../../scripts/copy-pkg-cortex-scaffold.mjs"
   },
   "dependencies": {
-    "@nexus-cortex/core": "^4.38.0",
-    "@nexus-cortex/executors": "^4.38.0",
+    "@nexus-cortex/core": "^4.40.0",
+    "@nexus-cortex/executors": "^4.40.0",
     "chalk": "^5.3.0",
     "cors": "^2.8.5",
     "dotenv": "^16.4.5",