@iris-eval/mcp-server 0.8.1 → 0.8.2

package/README.md

@@ -220,7 +220,7 @@ When `IRIS_OTEL_ENDPOINT` is configured, `log_trace` calls also emit a best-effo
 - **`score`** (0..1) is the weighted average across the rules that ran — a quality gradient.
 - **`passed`** is the ship/no-ship verdict: `true` only when the score clears the pass threshold (default **0.7**) **and no critical rule failed**.
 
-Genuine safety violations hard-fail. `no_pii`, `no_injection_patterns`, and `no_blocklist_words` are **critical rules**: if one fails, the eval reports `passed: false` no matter how well the other rules scored, and the response names the culprits in `critical_failures`. A leaked SSN can't be averaged away. Custom rules deployed with `severity: "high"` or `"critical"` hard-fail the same way; `low`/`medium` severities only affect the score. One boundary to know: a critical rule that **skipped** (missing context, or any other cause of a skip) has not judged the output and does not veto — it is listed in `critical_skipped`, and `rule_results` shows every skip and its reason, so a gate that must fail closed on non-verdicts can.
+Genuine safety violations hard-fail. By default `no_pii`, `no_injection_patterns`, and `no_blocklist_words` are **critical rules**: if one fails, the eval reports `passed: false` no matter how well the other rules scored, and the response names the culprits in `critical_failures`. A leaked SSN can't be averaged away. Which built-in rules are critical is a deployment setting (`eval.criticalRules` / `eval.nonCriticalRules`); every rule result carries the effective `critical` flag and `criticalSource`, and `list_rules` reports the roster this server applies. Custom rules deployed with `severity: "high"` or `"critical"` hard-fail the same way; `low`/`medium` severities only affect the score. One boundary to know, stated the same way on every surface: a critical rule that **skipped** (missing context, a broken definition, or a regex killed at the sandbox budget) has not judged the output and does not veto — every such rule is named in `critical_skipped`. **A gate that must fail closed treats a non-empty `critical_skipped` as unknown, not clean**, and may treat any `budgetExceeded` skip in `rule_results` the same way.
 
 For CI gates: if you omit `eval_type`, **every bundle runs** — completeness, relevance, safety, cost and any custom rules — and the response says `eval_type: "all"` with a `note` that the default ran, plus a per-bundle `categories` map. A bundle with nothing to judge (cost without `cost_usd`, relevance without `input`) reports `passed: null` there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the `eval_type` that ran, so your gate can verify coverage; key on `passed` for the verdict and name a bundle only when you want a narrower run.