@iris-eval/mcp-server 0.8.1 → 0.9.0

package/README.md

@@ -211,6 +211,13 @@ Iris registers nine tools that any MCP-compatible agent can invoke — full rule
 - **`evaluate_with_llm_judge`** — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. **Bring your own API key** (`IRIS_ANTHROPIC_API_KEY` or `IRIS_OPENAI_API_KEY`) — Iris doesn't proxy or relay LLM calls.
 - **`verify_citations`** — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as `evaluate_with_llm_judge`.
 
+**Enable the LLM judge (optional; the deterministic rules never need it)**
+1. Get an API key from Anthropic or OpenAI.
+2. Put it in the environment of the process that runs Iris, not only your shell. Claude Code, Claude Desktop, Cursor and most MCP clients: the "env" block of the iris-eval entry in your MCP config — "iris-eval": { "command": "npx", "args": ["-y", "@iris-eval/mcp-server"], "env": { "IRIS_ANTHROPIC_API_KEY": "sk-ant-..." } } (IRIS_OPENAI_API_KEY for an OpenAI key). Docker: -e IRIS_ANTHROPIC_API_KEY=... on the run command. HTTP or CI: export it before starting iris-mcp.
+3. Restart the MCP session. A running process never sees a variable set after it started.
+4. Confirm from inside your client: read iris://capabilities — judge.enabled must be true there. A key exported in your shell is not passed to the process your client spawns unless its config lists it. On a machine, `npx @iris-eval/mcp-server --self-test` prints the judge line for that shell, and GET /api/v1/health reports judge.enabled on a running dashboard.
+5. Spend guard: each call is capped by IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25 USD) and refused before any spend if the worst case would exceed it. Iris calls the provider directly with your key and never proxies it.
+
 When `IRIS_OTEL_ENDPOINT` is configured, `log_trace` calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See [docs/otel-integration.md](https://github.com/iris-eval/mcp-server/blob/main/docs/otel-integration.md).
 
 ### How `passed` is decided
@@ -220,7 +227,7 @@ When `IRIS_OTEL_ENDPOINT` is configured, `log_trace` calls also emit a best-effo
 - **`score`** (0..1) is the weighted average across the rules that ran — a quality gradient.
 - **`passed`** is the ship/no-ship verdict: `true` only when the score clears the pass threshold (default **0.7**) **and no critical rule failed**.
 
-Genuine safety violations hard-fail. `no_pii`, `no_injection_patterns`, and `no_blocklist_words` are **critical rules**: if one fails, the eval reports `passed: false` no matter how well the other rules scored, and the response names the culprits in `critical_failures`. A leaked SSN can't be averaged away. Custom rules deployed with `severity: "high"` or `"critical"` hard-fail the same way; `low`/`medium` severities only affect the score. One boundary to know: a critical rule that **skipped** (missing context, or any other cause of a skip) has not judged the output and does not veto — it is listed in `critical_skipped`, and `rule_results` shows every skip and its reason, so a gate that must fail closed on non-verdicts can.
+Genuine safety violations hard-fail. By default `no_pii`, `no_injection_patterns`, and `no_blocklist_words` are **critical rules**: if one fails, the eval reports `passed: false` no matter how well the other rules scored, and the response names the culprits in `critical_failures`. A leaked SSN can't be averaged away. Which built-in rules are critical is a deployment setting (`eval.criticalRules` / `eval.nonCriticalRules`); every rule result carries the effective `critical` flag and `criticalSource`, and `list_rules` reports the roster this server applies. Custom rules deployed with `severity: "high"` or `"critical"` hard-fail the same way; `low`/`medium` severities only affect the score. One boundary to know, stated the same way on every surface: a critical rule that **skipped** (missing context, a broken definition, or a regex killed at the sandbox budget) has not judged the output and does not veto — every such rule is named in `critical_skipped`. **A gate that must fail closed treats a non-empty `critical_skipped` as unknown, not clean**, and may treat any `budgetExceeded` skip in `rule_results` the same way.
 
 For CI gates: if you omit `eval_type`, **every bundle runs** — completeness, relevance, safety, cost and any custom rules — and the response says `eval_type: "all"` with a `note` that the default ran, plus a per-bundle `categories` map. A bundle with nothing to judge (cost without `cost_usd`, relevance without `input`) reports `passed: null` there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the `eval_type` that ran, so your gate can verify coverage; key on `passed` for the verdict and name a bundle only when you want a narrower run.
 
@@ -357,7 +364,7 @@ With a key set, API clients — MCP clients, capture SDKs, `POST /api/v1/traces`
 
 ### Your data on disk
 
-Everything Iris stores lives under your Iris home (`~/.iris`, or `IRIS_HOME`). `iris.db` keeps every trace's `input` and `output` **verbatim** — including any text `no_pii` goes on to flag; detection does not redact. At startup, traces and evaluations older than `retention.days` (default `30`, `0` disables, set in `config.json`) are deleted and the write-ahead log is checkpointed. To remove everything now, stop the server and run `--purge`: it deletes every stored trace, span and evaluation, compacts the database and truncates the write-ahead log so the text is gone from disk, and keeps your deployed rules, audit log and preferences.
+Everything Iris stores lives under your Iris home (`~/.iris`, or `IRIS_HOME`). `iris.db` keeps every trace's `input` and `output` **verbatim** — including any text `no_pii` goes on to flag; detection does not redact unless you ask it to: `storage.redact: "critical_spans"` in `config.json` stores each evaluation's output with the spans a critical detector flagged replaced by `[REDACTED:<pattern>]` (off by default; the evidence offsets still index the text the caller saw). At startup, and every `retention.sweepIntervalHours` (default `24`, `0` disables the timer) after that, traces and evaluations older than `retention.days` (default `30`, `0` disables, set in `config.json`) are deleted and the write-ahead log is checkpointed. Deleting a trace — by `delete_trace` or by the sweep — erases the text of every evaluation linked to it (the output, the expected text, the suggestions and the rule messages) and stamps `erased_at`; the verdict, the scores and the evidence offsets stay. To remove everything now, stop the server and run `--purge`: it deletes every stored trace, span and evaluation, compacts the database and truncates the write-ahead log so the text is gone from disk, and keeps your deployed rules, audit log and preferences.
 
 </details>
 

package/dist/capabilities.d.ts

@@ -0,0 +1,64 @@
+import type { IrisConfig } from './types/config.js';
+import type { EvalEngine } from './eval/engine.js';
+import type { CustomRuleStore } from './custom-rule-store.js';
+import { type BuiltInRuleMeta } from './eval/criticality.js';
+import { type EvaluationQuestion } from './eval/questions.js';
+import { type PublishedRuleAccuracy } from './eval/accuracy.js';
+import { type JudgeProvider } from './judge-enablement.js';
+/** The most inline custom rules one evaluate_output call may carry. */
+export declare const MAX_INLINE_CUSTOM_RULES = 10;
+/** The most citations one verify_citations call may verify. */
+export declare const MAX_CITATIONS_PER_CALL = 50;
+export interface RuleProof extends PublishedRuleAccuracy {
+    /** Positive predictive value at four prevalences: what a fire is worth when the failure is rare. */
+    ppvAt: Record<string, number | null>;
+    corpusVersion: string;
+    release: string;
+    labelling: 'same-model' | 'human-verified';
+}
+export interface Capabilities {
+    version: string;
+    transport: 'stdio' | 'http';
+    questions: readonly EvaluationQuestion[];
+    rules: Array<BuiltInRuleMeta & {
+        proof: RuleProof | null;
+    }>;
+    customRules: {
+        count: number;
+        enabled: number;
+        quarantined: unknown[];
+    };
+    judge: {
+        enabled: boolean;
+        provider: JudgeProvider | null;
+        providers: JudgeProvider[];
+        costCapUsd: number;
+        howToEnable: readonly string[];
+    };
+    citations: {
+        fetchAllowed: boolean;
+        domainsRestricted: boolean;
+    };
+    dashboard: {
+        enabled: boolean;
+        url: string | null;
+        mode: 'real' | 'demo';
+    };
+    limits: {
+        customRulesPerCall: number;
+        regexBudgetMs: number;
+        httpRateLimitPerMin: number;
+        citationsPerCall: number;
+    };
+    tools: readonly string[];
+    resources: readonly string[];
+    prompts: readonly string[];
+}
+export interface CapabilitiesContext {
+    config: IrisConfig;
+    evalEngine?: EvalEngine;
+    customRuleStore?: CustomRuleStore;
+    mode?: 'real' | 'demo';
+}
+export declare function ruleProof(name: string): RuleProof | null;
+export declare function buildCapabilities(ctx: CapabilitiesContext): Capabilities;

package/dist/capabilities.js

@@ -0,0 +1,65 @@
+import { builtInRuleRoster } from './eval/criticality.js';
+import { QUESTIONS } from './eval/questions.js';
+import { publishedAccuracyFor, publishedProvenance, ppvAt } from './eval/accuracy.js';
+import { REGEX_MATCH_BUDGET_MS } from './eval/rules/regex-sandbox.js';
+import { JUDGE_ENABLE_STEPS, judgeState } from './judge-enablement.js';
+import { LOCAL_TENANT } from './types/tenant.js';
+import { TOOL_NAMES } from './tools/index.js';
+import { RESOURCE_URIS } from './resources/uris.js';
+import { EVALUATE_MY_AGENT_PROMPT } from './instructions.js';
+/** The most inline custom rules one evaluate_output call may carry. */
+export const MAX_INLINE_CUSTOM_RULES = 10;
+/** The most citations one verify_citations call may verify. */
+export const MAX_CITATIONS_PER_CALL = 50;
+export function ruleProof(name) {
+    const acc = publishedAccuracyFor(name);
+    if (!acc)
+        return null;
+    const prov = publishedProvenance();
+    return {
+        ...acc,
+        ppvAt: ppvAt(name),
+        corpusVersion: prov.corpusVersion,
+        release: prov.release,
+        labelling: prov.labelling,
+    };
+}
+export function buildCapabilities(ctx) {
+    const { config } = ctx;
+    const roster = builtInRuleRoster(ctx.evalEngine ? (rule) => ctx.evalEngine.effectiveCriticality(rule) : undefined);
+    const custom = ctx.customRuleStore?.list(LOCAL_TENANT) ?? [];
+    const judge = judgeState();
+    const dashboardHost = config.dashboard.host === '0.0.0.0' || config.dashboard.host === '::' ? 'localhost' : config.dashboard.host;
+    return {
+        version: config.server.version,
+        transport: config.transport.type,
+        questions: QUESTIONS,
+        rules: roster.map((r) => ({ ...r, proof: ruleProof(r.name) })),
+        customRules: { count: custom.length, enabled: custom.filter((r) => r.enabled).length, quarantined: [] },
+        judge: {
+            enabled: judge.enabled,
+            provider: judge.provider,
+            providers: judge.providers,
+            costCapUsd: judge.costCapUsd,
+            howToEnable: JUDGE_ENABLE_STEPS,
+        },
+        citations: {
+            fetchAllowed: process.env.IRIS_CITATION_ALLOW_FETCH === '1',
+            domainsRestricted: Boolean(process.env.IRIS_CITATION_DOMAINS && process.env.IRIS_CITATION_DOMAINS.trim().length > 0),
+        },
+        dashboard: {
+            enabled: config.dashboard.enabled,
+            url: config.dashboard.enabled ? `http://${dashboardHost}:${config.dashboard.port}` : null,
+            mode: ctx.mode ?? 'real',
+        },
+        limits: {
+            customRulesPerCall: MAX_INLINE_CUSTOM_RULES,
+            regexBudgetMs: REGEX_MATCH_BUDGET_MS,
+            httpRateLimitPerMin: config.security.rateLimit.mcp,
+            citationsPerCall: MAX_CITATIONS_PER_CALL,
+        },
+        tools: TOOL_NAMES,
+        resources: RESOURCE_URIS,
+        prompts: [EVALUATE_MY_AGENT_PROMPT],
+    };
+}

package/dist/config/defaults.js

@@ -16,6 +16,7 @@ export const PKG_VERSION = pkgVersion;
 export const defaultConfig = {
     storage: {
         type: 'sqlite',
+        redact: 'none',
         path: join(irisHome(), 'iris.db'),
     },
     server: {
@@ -64,6 +65,7 @@ export const defaultConfig = {
     },
     retention: {
         days: 30,
+        sweepIntervalHours: 24,
     },
     security: {
         apiKey: undefined,

package/dist/custom-rule-store.d.ts

@@ -9,6 +9,8 @@ export interface CustomRuleStore {
     setEnabled(tenantId: TenantId, id: string, enabled: boolean, user?: string): DeployedCustomRule | undefined;
     /** All ENABLED rules for a tenant in deploy order — what the engine should register. */
     enabledRules(tenantId: TenantId): DeployedCustomRule[];
+    /** Entries in the store this version could not validate: kept on disk, never registered, never deleted by a deploy. */
+    quarantined(tenantId: TenantId): unknown[];
     /** Path on disk for diagnostics. Different per tenant. */
     pathFor(tenantId: TenantId): string;
     auditPath: string;
@@ -21,6 +23,8 @@ export interface DeployRuleInput {
     definition: CustomRuleDefinition;
     sourceMomentId?: string;
     user?: string;
+    /** With replace: true, the ids of the same-named rules retired first — recorded on the rule.deploy audit row. */
+    replaces?: string[];
 }
 export declare function createCustomRuleStore(opts?: {
     /**

package/dist/custom-rule-store.js

@@ -310,6 +310,9 @@ export function createCustomRuleStore(opts) {
         list(tenantId) {
             return [...load(tenantId)];
         },
+        quarantined(tenantId) {
+            return state(tenantId).quarantined;
+        },
         get(tenantId, id) {
             return load(tenantId).find((r) => r.id === id);
         },
@@ -344,9 +347,11 @@ export function createCustomRuleStore(opts) {
                 user: input.user ?? 'local',
                 ruleId: id,
                 ruleName: rule.name,
-                details: input.sourceMomentId
-                    ? { sourceMomentId: input.sourceMomentId, severity: rule.severity }
-                    : { severity: rule.severity },
+                details: {
+                    severity: rule.severity,
+                    ...(input.sourceMomentId ? { sourceMomentId: input.sourceMomentId } : {}),
+                    ...(input.replaces?.length ? { replaces: input.replaces } : {}),
+                },
             });
             return validated;
         },