@iris-eval/mcp-server 0.6.0 → 0.8.0

package/README.md

@@ -13,7 +13,7 @@
 [![PulseMCP](https://img.shields.io/badge/PulseMCP-Listed-blue?style=flat-square)](https://www.pulsemcp.com/servers/iris-eval)
 [![mcp.so](https://img.shields.io/badge/mcp.so-Listed-blue?style=flat-square)](https://mcp.so/server/iris/iris-eval)
 
-**Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account.** Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 13 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.
+**Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account.** Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 15 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.
 
 **Requires Node.js 20 or later.** Check with `node --version`.
 
@@ -185,7 +185,7 @@ docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data ghcr.io/iris-eval/mcp-se
 | | |
 |---|---|
 | **Trace Logging** | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
-| **Output Evaluation** | 13 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (19 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, PEM private-key blocks and seed phrases), prompt injection (37 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass `input` to ground them against the agent's source material). Add custom rules with Zod schemas. |
+| **Output Evaluation** | 15 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (19 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, PEM private-key blocks and seed phrases), prompt injection (37 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass `input` to ground them against the agent's source material), and two trajectory rules that read the agent's `tool_calls`: an unacknowledged failed tool call, and a repeated one. Add custom rules with Zod schemas. |
 | **LLM-as-Judge** | Optional semantic scoring via Anthropic or OpenAI — bring your own API key. Five templates. Hard per-eval cost cap (`IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL`, default $0.25), per-eval pricing disclosed in the result. |
 | **Cost Visibility** | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. |
 | **Web Dashboard** | Real-time dark-mode UI that lands on the failures, worst and newest first — trace visualization, eval results, cost breakdowns, and a command palette (⌘K) that searches your own rules, traces, and evals. |
@@ -193,6 +193,10 @@ docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data ghcr.io/iris-eval/mcp-se
 
 Where this is going next: [the roadmap](https://github.com/iris-eval/mcp-server/blob/main/docs/roadmap.md).
 
+### Measured, not claimed
+
+Every built-in rule has a published precision, recall and F1 with 95% confidence intervals, measured on a labelled corpus that lives in this repository (`proof/corpus/`) and regenerates with one command — `npm run proof` — offline, with no key and no model in the loop. CI re-runs the measurement on every pull request and fails if the committed numbers differ from what the code produces, so a rule cannot change without its numbers changing with it. The numbers are on [iris-eval.com/proof](https://iris-eval.com/proof) and in [`proof/RESULTS.md`](https://github.com/iris-eval/mcp-server/blob/main/proof/RESULTS.md); how the corpus was made, what it is not, and how to read an interval are in [docs/proof.md](https://github.com/iris-eval/mcp-server/blob/main/docs/proof.md). The corpus is synthetic and model-labelled — a human blind label is pending, and the page says so.
+
 ## MCP Tools
 
 Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:
@@ -218,7 +222,7 @@ When `IRIS_OTEL_ENDPOINT` is configured, `log_trace` calls also emit a best-effo
 
 Genuine safety violations hard-fail. `no_pii`, `no_injection_patterns`, and `no_blocklist_words` are **critical rules**: if one fails, the eval reports `passed: false` no matter how well the other rules scored, and the response names the culprits in `critical_failures`. A leaked SSN can't be averaged away. Custom rules deployed with `severity: "high"` or `"critical"` hard-fail the same way; `low`/`medium` severities only affect the score. One boundary to know: a critical rule that **skipped** (missing context, or any other cause of a skip) has not judged the output and does not veto — it is listed in `critical_skipped`, and `rule_results` shows every skip and its reason, so a gate that must fail closed on non-verdicts can.
 
-One gotcha for CI gates: if you omit `eval_type`, the default `completeness` bundle runs — **safety rules don't**. The response echoes `eval_type` (plus a `note` when it was defaulted) so your gate can verify which bundle actually ran. Key on `passed` for the verdict and `eval_type: "safety"` for coverage.
+For CI gates: if you omit `eval_type`, **every bundle runs** — completeness, relevance, safety, cost and any custom rules — and the response says `eval_type: "all"` with a `note` that the default ran, plus a per-bundle `categories` map. A bundle with nothing to judge (cost without `cost_usd`, relevance without `input`) reports `passed: null` there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the `eval_type` that ran, so your gate can verify coverage; key on `passed` for the verdict and name a bundle only when you want a narrower run.
 
 ### Authoring a custom rule
 

package/dist/config/defaults.js

@@ -38,10 +38,26 @@ export const defaultConfig = {
             min_output_length: 50,
             min_sentences: 2,
             keyword_overlap: 0.35,
-            topic_consistency: 0.10,
+            // Share of content-bearing sentences that must connect to the input's
+            // topic (a third — see topic_consistency in src/eval/rules/relevance.ts).
+            topic_consistency: 0.33,
             cost_threshold: 0.10,
             max_token_ratio: 5,
+            // Identical tool calls tolerated before no_tool_loop fires. Three
+            // allows a legitimate retry-with-backoff; the fourth is a loop.
+            max_tool_repeats: 3,
         },
+        /*
+         * Which built-in rules veto is a deployment's call, and the shipped
+         * answer is "the three the rules themselves declare" — no_pii,
+         * no_injection_patterns, no_blocklist_words. Both lists start empty so
+         * the default behaviour is exactly the rules' own declarations; a
+         * config.json that sets either one REPLACES the empty list (arrays are
+         * not deep-merged). See src/eval/criticality.ts for why this is
+         * configurable at all, and /proof for the error rates to decide with.
+         */
+        criticalRules: [],
+        nonCriticalRules: [],
     },
     logging: {
         level: 'info',

package/dist/config/index.js

@@ -1,6 +1,7 @@
 import { readFileSync, mkdirSync, existsSync } from 'node:fs';
 import { join, dirname } from 'node:path';
 import { defaultConfig } from './defaults.js';
+import { assertValidCriticality } from '../eval/criticality.js';
 import { irisHome } from '../utils/iris-home.js';
 /*
  * Owner-only (0700) for the iris home directory, matching the 0600 the data
@@ -172,5 +173,12 @@ export function loadConfig(cliArgs) {
     config = deepMerge(config, envConfig);
     config = deepMerge(config, argsConfig);
     ensureIrisDirectory(dirname(config.storage.path), 'the database directory (IRIS_DB_PATH / --db-path)');
+    /*
+     * Fail at STARTUP on a bad rule name, before a single evaluation runs.
+     * A typo in eval.criticalRules that quietly did nothing would leave an
+     * operator believing their deploy gate exists when it does not — the same
+     * "detection that reports an all-clear" failure the veto exists to stop.
+     */
+    assertValidCriticality(config.eval);
     return config;
 }