@iris-eval/mcp-server 0.4.6 → 0.5.0

package/README.md

@@ -1,7 +1,7 @@
-# Iris — The Agent Eval Standard for MCP
+# Iris — stop shipping agents on vibes
 
 [![Glama Score](https://glama.ai/mcp/servers/iris-eval/mcp-server/badges/score.svg)](https://glama.ai/mcp/servers/iris-eval/mcp-server)
-[![Install in Cursor](https://cursor.com/deeplink/mcp-install-dark.svg)](cursor://anysphere.cursor-deeplink/mcp/install?name=server&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBpcmlzLWV2YWwvbWNwLXNlcnZlciJdLCJlbnYiOnsiSVJJU19MT0dfTEVWRUwiOiJpbmZvIn19)
+[![Install in Cursor](https://cursor.com/deeplink/mcp-install-dark.svg)](cursor://anysphere.cursor-deeplink/mcp/install?name=iris-eval&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBpcmlzLWV2YWwvbWNwLXNlcnZlciJdLCJlbnYiOnsiSVJJU19MT0dfTEVWRUwiOiJpbmZvIn19)
 [![npm version](https://img.shields.io/npm/v/@iris-eval/mcp-server)](https://npmjs.com/package/@iris-eval/mcp-server)
 [![npm downloads](https://img.shields.io/npm/dt/@iris-eval/mcp-server)](https://npmjs.com/package/@iris-eval/mcp-server)
 [![GitHub stars](https://img.shields.io/github/stars/iris-eval/mcp-server?style=social)](https://github.com/iris-eval/mcp-server)
@@ -13,70 +13,77 @@
 [![PulseMCP](https://img.shields.io/badge/PulseMCP-Listed-blue?style=flat-square)](https://www.pulsemcp.com/servers/iris-eval)
 [![mcp.so](https://img.shields.io/badge/mcp.so-Listed-blue?style=flat-square)](https://mcp.so/server/iris/iris-eval)
 
-**Know whether your AI agents are actually good enough to ship.** Iris is an open-source MCP server that scores output quality, catches safety failures, and enforces cost budgets across all your agents. Any MCP-compatible agent discovers and uses it automatically — no SDK, no code changes.
+**Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account.** Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 13 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.
+
+**Requires Node.js 20 or later.** Check with `node --version`.
 
 ![Iris Dashboard](https://raw.githubusercontent.com/iris-eval/mcp-server/main/docs/assets/dashboard-overview.png)
 
-## The Problem
+## A failure on screen in 60 seconds
 
-Your agents are running in production. Infrastructure monitoring sees `200 OK` and moves on. It has no idea the agent just:
+No agent wiring, no config — one command:
 
-- Leaked a social security number in its response
-- Hallucinated an answer with zero factual grounding
-- Burned $0.47 on a single query — 4.7x your budget threshold
-- Made 6 tool calls when 2 would have sufficed
+```bash
+npx @iris-eval/mcp-server --demo
+```
 
-Iris evaluates all of it.
+This seeds a demo database — a handful of small agents with a week of runs — and serves the dashboard against it at **http://localhost:6920** (your browser opens automatically on first run). The dashboard lands on **Failures**: what failed, worst and newest first. Worth clicking into — a PII leak caught by the safety rules, a flagged prompt-injection attempt, and a failed LLM-judge score with its rationale.
 
-## What You Get
+Demo data lives in its own database (`demo.db` in your Iris home directory — `~/.iris` on macOS/Linux, `%USERPROFILE%\.iris` on Windows) and never mixes with your real traces. Remove all of it with one command:
 
-| | |
-|---|---|
-| **Trace Logging** | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
-| **Output Evaluation** | 13 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (10 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport), prompt injection (13 patterns), stub-output detection, hallucination markers (17 hedging phrases + fabricated-citation heuristic). Add custom rules with Zod schemas. |
-| **Cost Visibility** | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. |
-| **Web Dashboard** | Real-time dark-mode UI with trace visualization, eval results, and cost breakdowns. |
-
-**Requires Node.js 20 or later.** Check with `node --version`.
+```bash
+npx @iris-eval/mcp-server --demo-clear
+```
 
-## Quickstart
+## Hook up your own agent
 
-Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent.
+Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent. One block, dashboard included:
 
 ```json
 {
   "mcpServers": {
     "iris-eval": {
       "command": "npx",
-      "args": ["@iris-eval/mcp-server"]
+      "args": ["@iris-eval/mcp-server", "--dashboard"]
     }
   }
 }
 ```
 
-That's it. Your agent discovers Iris and starts logging traces automatically.
+Your agent discovers Iris's nine tools on connect, and the dashboard serves at **http://localhost:6920**. Now paste this to your agent:
 
-### Turn on the dashboard
+> Log that last task to Iris and evaluate the output.
 
-Iris ships with a real-time web dashboard showing traces, eval results, cost breakdowns, and rule pass-rates. It's off by default so the MCP server stays lightweight — flip it on with a flag.
+The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop `--dashboard` from the args — you can open the same dashboard any time with `npx @iris-eval/mcp-server --dashboard`.
 
-```json
-{
-  "mcpServers": {
-    "iris-eval": {
-      "command": "npx",
-      "args": ["@iris-eval/mcp-server", "--dashboard"]
-    }
-  }
-}
+**One thing worth knowing up front:** MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, `POST /api/v1/traces` does exactly that — your code sends the trace over plain HTTP, no model in the loop (see [docs/http-ingest.md](docs/http-ingest.md)). The CLI and SDKs on the [roadmap](docs/roadmap.md) will be thin clients over the same endpoint.
+
+### Capture over HTTP (no model in the loop)
+
+With the dashboard running, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request:
+
+```bash
+curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "agent_name": "support-bot",
+    "input": "What is the refund policy?",
+    "output": "Refunds are available within 30 days of purchase.",
+    "evaluate": true,
+    "eval_type": "safety"
+  }'
 ```
 
-Then open **http://localhost:6920** after your agent runs a trace. The same dashboard is available via CLI:
+Returns `201` with the stored `trace_id` and the evaluation result. The endpoint accepts the same body as the `log_trace` tool and sits behind the same loopback-only middleware stack as the rest of the dashboard. Full contract, field reference, and error semantics: [docs/http-ingest.md](docs/http-ingest.md).
+
+### Check the install
 
 ```bash
-npx @iris-eval/mcp-server --dashboard
+npx @iris-eval/mcp-server --self-test
 ```
 
+An offline install diagnostic: storage round-trip, deterministic evals, dashboard + DNS-rebinding guard — all inside an isolated temp home, so your real database is never opened. Exit code 0 = healthy, 1 = a check failed.
+
 <details>
 <summary><strong>Setup by tool</strong></summary>
 
@@ -171,6 +178,19 @@ docker run -p 3000:3000 -v iris-data:/data ghcr.io/iris-eval/mcp-server
 
 > **Tip:** Global install (`npm install -g`) stores traces persistently at `~/.iris/iris.db`. With `npx`, traces persist in the same location, but startup is slower due to package resolution.
 
+## What You Get
+
+| | |
+|---|---|
+| **Trace Logging** | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
+| **Output Evaluation** | 13 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (19 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, PEM private-key blocks and seed phrases), prompt injection (37 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass `input` to ground them against the agent's source material). Add custom rules with Zod schemas. |
+| **LLM-as-Judge** | Optional semantic scoring via Anthropic or OpenAI — bring your own API key. Five templates. Hard per-eval cost cap (`IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL`, default $0.25), per-eval pricing disclosed in the result. |
+| **Cost Visibility** | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. |
+| **Web Dashboard** | Real-time dark-mode UI that lands on the failures, worst and newest first — trace visualization, eval results, cost breakdowns, and a command palette (⌘K) that searches your own rules, traces, and evals. |
+| **Local-first** | Everything lives in SQLite on your disk. No account, no sign-up, no telemetry. Outbound HTTP happens only where you opt in: your own LLM-judge key, citation fetching, or an OTel exporter you configure. |
+
+Where this is going next: [the roadmap](docs/roadmap.md).
+
 ## MCP Tools
 
 Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:
@@ -187,13 +207,26 @@ Iris registers nine tools that any MCP-compatible agent can invoke — full rule
 
 When `IRIS_OTEL_ENDPOINT` is configured, `log_trace` calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See [docs/otel-integration.md](docs/otel-integration.md).
 
+### How `passed` is decided
+
+`evaluate_output` returns both a `score` and a `passed` flag — they answer different questions:
+
+- **`score`** (0..1) is the weighted average across the rules that ran — a quality gradient.
+- **`passed`** is the ship/no-ship verdict: `true` only when the score clears the pass threshold (default **0.7**) **and no critical rule failed**.
+
+Genuine safety violations hard-fail. `no_pii`, `no_injection_patterns`, and `no_blocklist_words` are **critical rules**: if one fails, the eval reports `passed: false` no matter how well the other rules scored, and the response names the culprits in `critical_failures`. A leaked SSN can't be averaged away. Custom rules deployed with `severity: "high"` or `"critical"` hard-fail the same way; `low`/`medium` severities only affect the score. One boundary to know: a critical rule that **skipped** (missing context, or any other cause of a skip) has not judged the output and does not veto — `rule_results` shows every skip and its reason, so a gate that must fail closed on non-verdicts can.
+
+One gotcha for CI gates: if you omit `eval_type`, the default `completeness` bundle runs — **safety rules don't**. The response echoes `eval_type` (plus a `note` when it was defaulted) so your gate can verify which bundle actually ran. Key on `passed` for the verdict and `eval_type: "safety"` for coverage.
+
 Full tool schemas and configuration: [iris-eval.com](https://iris-eval.com)
 
-## Cloud Tier (Coming Soon)
+## Hosted features
+
+Iris runs entirely on your machine today, and everything it does is free and MIT licensed with no limits and no account.
 
-Self-hosted Iris runs on your machine with SQLite. As your team's eval needs grow, the cloud tier adds PostgreSQL, team dashboards, alerting on quality regressions, and managed infrastructure.
+Hosted storage, shared team history and alerting are **under consideration, not under construction**. There is no pricing, and nothing to buy. If shared history would be useful to you, [the waitlist](https://iris-eval.com#waitlist) is how we find out whether it's worth building — it commits you to nothing.
 
-[Join the waitlist](https://iris-eval.com#waitlist) to get early access.
+Two commitments hold regardless: **nothing that is free today will move behind a paywall**, and **no compliance certification will be claimed before it is held**.
 
 ## Examples
 
@@ -208,6 +241,7 @@ Self-hosted Iris runs on your machine with SQLite. As your team's eval needs gro
 - [GitHub Issues](https://github.com/iris-eval/mcp-server/issues) — Bug reports and feature requests
 - [GitHub Discussions](https://github.com/iris-eval/mcp-server/discussions) — Questions and ideas
 - [Contributing Guide](CONTRIBUTING.md) — How to contribute
+- [HTTP Ingest](docs/http-ingest.md) — Deterministic trace capture via `POST /api/v1/traces`
 - [Roadmap](docs/roadmap.md) — What's coming next
 
 <details>
@@ -224,6 +258,10 @@ Self-hosted Iris runs on your machine with SQLite. As your team's eval needs gro
 | `--api-key` | — | API key for HTTP authentication |
 | `--dashboard` | `false` | Enable web dashboard |
 | `--dashboard-port` | `6920` | Dashboard port |
+| `--dashboard-host` | `127.0.0.1` | Dashboard bind address. Loopback by default — the dashboard is unauthenticated unless `--api-key` is set, so binding beyond loopback exposes your full trace history |
+| `--demo` | `false` | Seed a demo database (separate from your real traces) and serve the dashboard against it |
+| `--demo-clear` | `false` | Delete the demo database and exit |
+| `--self-test` | `false` | Run the offline install diagnostic in an isolated temp home, then exit (0 = healthy, 1 = a check failed) |
 
 ### Environment Variables
 
@@ -232,10 +270,12 @@ Self-hosted Iris runs on your machine with SQLite. As your team's eval needs gro
 | `IRIS_TRANSPORT` | Transport type (`stdio` or `http`) |
 | `IRIS_PORT` | HTTP transport port |
 | `IRIS_HOST` | HTTP transport host (default `127.0.0.1`) |
-| `IRIS_DB_PATH` | SQLite database path |
+| `IRIS_HOME` | Directory for all per-user files: `config.json`, `iris.db`, `custom-rules.json`, `audit.log`, `preferences.json` (default `~/.iris`) |
+| `IRIS_DB_PATH` | SQLite database path (overrides `IRIS_HOME` for the DB only) |
 | `IRIS_LOG_LEVEL` | Log level: `debug`, `info`, `warn`, `error` |
-| `IRIS_DASHBOARD` | Enable web dashboard (`true`/`false`) |
+| `IRIS_DASHBOARD` | Enable web dashboard (`true`/`false`; `false` also overrides `dashboard.enabled` in config.json) |
 | `IRIS_DASHBOARD_PORT` | Dashboard port (default `6920`) |
+| `IRIS_DASHBOARD_HOST` | Dashboard bind address (default `127.0.0.1`) |
 | `IRIS_API_KEY` | API key for HTTP authentication |
 | `IRIS_ALLOWED_ORIGINS` | Comma-separated allowed CORS origins |
 
@@ -263,6 +303,14 @@ iris-mcp --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dash
 <details>
 <summary><strong>Troubleshooting</strong></summary>
 
+### First move: run the self-test
+
+```bash
+npx @iris-eval/mcp-server --self-test
+```
+
+It checks storage, the deterministic evals, and the dashboard in an isolated temp home and prints a per-step verdict — the failure output names the broken step. Exit code 0 means the install is healthy.
+
 ### Iris won't start / `ERR_MODULE_NOT_FOUND`
 
 You may have a cached older version. Clear the npx cache and retry:
@@ -283,13 +331,15 @@ MCP tools only load at session start. After adding iris-eval, restart the sessio
 
 ### Version check
 
-Verify which version is running:
+Iris logs its version on the first startup line:
 
 ```bash
-npx @iris-eval/mcp-server --help
-# Shows "Iris — MCP-Native Agent Eval Server vX.Y.Z"
+npx @iris-eval/mcp-server --dashboard
+# First log line: "Starting Iris MCP server vX.Y.Z"
 ```
 
+For a global install, `npm ls -g @iris-eval/mcp-server` shows the installed version.
+
 ### Updating
 
 ```bash

package/dist/audit-log-reader.d.ts

@@ -12,8 +12,6 @@ export interface AuditQueryResult {
     total: number;
     limit: number;
     offset: number;
-    /** Absolute path to the audit log file (for diagnostics). */
-    path: string;
 }
 export declare function readAuditLog(opts?: {
     filePath?: string;

package/dist/audit-log-reader.js

@@ -38,14 +38,14 @@ export function readAuditLog(opts) {
     const limit = Math.min(Math.max(opts?.limit ?? 100, 1), 1000);
     const offset = Math.max(opts?.offset ?? 0, 0);
     if (!existsSync(filePath)) {
-        return { entries: [], total: 0, limit, offset, path: filePath };
+        return { entries: [], total: 0, limit, offset };
     }
     let raw;
     try {
         raw = readFileSync(filePath, 'utf-8');
     }
     catch {
-        return { entries: [], total: 0, limit, offset, path: filePath };
+        return { entries: [], total: 0, limit, offset };
     }
     // Parse line-by-line, drop malformed rows silently. The audit log is
     // append-only so a partially-flushed last line is the only realistic
@@ -83,5 +83,5 @@ export function readAuditLog(opts) {
     filtered.sort((a, b) => b.ts.localeCompare(a.ts));
     const total = filtered.length;
     const entries = filtered.slice(offset, offset + limit);
-    return { entries, total, limit, offset, path: filePath };
+    return { entries, total, limit, offset };
 }

package/dist/config/index.js

@@ -45,6 +45,23 @@ function parsePortEnv(value, name) {
     }
     return n;
 }
+/*
+ * IRIS_DASHBOARD used to be `value === 'true'`, which silently read every
+ * other spelling — 1, yes, on, TRUE — as an explicit DISABLE that then
+ * overrode config.json's dashboard.enabled in the layer merge. The user who
+ * exported IRIS_DASHBOARD=1 got no dashboard plus a pointer log telling
+ * them to set the very variable they believed they had set. Unrecognized
+ * values now throw, same contract as parsePortEnv: loud beats silently
+ * wrong for a startup switch.
+ */
+function parseBooleanEnv(value, name) {
+    const normalized = value.trim().toLowerCase();
+    if (['true', '1', 'yes', 'on'].includes(normalized))
+        return true;
+    if (['false', '0', 'no', 'off'].includes(normalized))
+        return false;
+    throw new Error(`${name}=${JSON.stringify(value)} is not a valid boolean (use true/1/yes/on or false/0/no/off)`);
+}
 function loadEnvVars() {
     const config = {};
     if (process.env.IRIS_TRANSPORT) {
@@ -63,7 +80,7 @@ function loadEnvVars() {
         config.logging = { level: process.env.IRIS_LOG_LEVEL };
     }
     if (process.env.IRIS_DASHBOARD) {
-        config.dashboard = { enabled: process.env.IRIS_DASHBOARD === 'true' };
+        config.dashboard = { enabled: parseBooleanEnv(process.env.IRIS_DASHBOARD, 'IRIS_DASHBOARD') };
     }
     if (process.env.IRIS_DASHBOARD_PORT) {
         config.dashboard = {

package/dist/custom-rule-store.js

@@ -24,13 +24,14 @@
  * leave a half-file.
  */
 import { mkdirSync, readFileSync, existsSync, appendFileSync } from 'node:fs';
-import { writeAtomic } from './utils/write-atomic.js';
+import { writeAtomic, ensureOwnerOnly, OWNER_ONLY_FILE_MODE } from './utils/write-atomic.js';
 import { irisHome } from './utils/iris-home.js';
 import { join, dirname } from 'node:path';
 import { randomBytes } from 'node:crypto';
 import { z } from 'zod';
 import isSafeRegex from 'safe-regex2';
 import { regexBacktrackingBudgetExceeded } from './eval/rules/regex-budget.js';
+import { normalizeRegexSource } from './eval/rules/custom.js';
 import { CUSTOM_RULE_CONFIG_KEYS, readNumericConfig, describeKeys } from './eval/rules/config-keys.js';
 import { LOCAL_TENANT } from './types/tenant.js';
 const SEVERITY_VALUES = ['low', 'medium', 'high', 'critical'];
@@ -99,15 +100,18 @@ const DefinitionSchema = z
                 });
                 break;
             }
-            // Strip a leading inline flag group the way the evaluator does, so a
-            // pattern that WILL run is not rejected here for syntax it tolerates.
-            const stripped = pattern.replace(/^\(\?[imsugy]+\)/, '');
+            // Normalize EXACTLY the way the evaluator does — same helper — so
+            // this layer validates and probes the identical pattern+flags pair
+            // that will actually run. (It used to strip the inline flag group
+            // but not merge its flags: a `(?i)` pattern was probed under
+            // different flags than evaluation used.)
+            const { pattern: stripped, flags: normalizedFlags } = normalizeRegexSource(pattern, typeof config.flags === 'string' ? config.flags : '');
             // Syntax BEFORE safety: safe-regex2 returns false for anything it
             // cannot parse, so checking it first reports a plainly broken pattern
             // like `(` as "catastrophic backtracking" — an error that sends the
             // author looking for a performance problem they do not have.
             try {
-                new RegExp(stripped, typeof config.flags === 'string' ? config.flags : '');
+                new RegExp(stripped, normalizedFlags);
             }
             catch (e) {
                 ctx.addIssue({
@@ -130,7 +134,7 @@ const DefinitionSchema = z
             // safe and takes 156ms on 40 characters. Measure what the static
             // check cannot see.
             {
-                const budgetIssue = regexBacktrackingBudgetExceeded(stripped, typeof config.flags === 'string' ? config.flags : '');
+                const budgetIssue = regexBacktrackingBudgetExceeded(stripped, normalizedFlags);
                 if (budgetIssue) {
                     ctx.addIssue({
                         code: z.ZodIssueCode.custom,
@@ -207,7 +211,13 @@ function generateRuleId() {
 function appendAudit(auditPath, entry) {
     try {
         mkdirSync(dirname(auditPath), { recursive: true });
-        appendFileSync(auditPath, `${JSON.stringify(entry)}\n`, 'utf-8');
+        // mode applies only when appendFileSync creates the file; an existing
+        // audit.log keeps its mode, which is why ensureOwnerOnly() also runs at
+        // store construction to repair files created before this change.
+        appendFileSync(auditPath, `${JSON.stringify(entry)}\n`, {
+            encoding: 'utf-8',
+            mode: OWNER_ONLY_FILE_MODE,
+        });
     }
     catch {
         // Audit best-effort. If filesystem is read-only or full, the deploy
@@ -262,7 +272,11 @@ export function createCustomRuleStore(opts) {
     function state(tenantId) {
         let loaded = tenantState.get(tenantId);
         if (loaded === undefined) {
-            loaded = loadRulesFromDisk(pathFor(tenantId));
+            const path = pathFor(tenantId);
+            loaded = loadRulesFromDisk(path);
+            // Repair permissions on files created before the owner-only change
+            // (and on the audit log, which appendFileSync only modes at creation).
+            ensureOwnerOnly(path, auditPath);
             tenantState.set(tenantId, loaded);
         }
         return loaded;