npm - @iris-eval/mcp-server - Versions diffs - 0.8.0 → 0.8.2 - Mend

@iris-eval/mcp-server 0.8.0 → 0.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (14) hide show

package/README.md +3 -2
package/dist/dashboard/assets/index-CyzO6OC7.js +10 -0
package/dist/dashboard/index.html +1 -1
package/dist/tools/delete-rule.js +1 -1
package/dist/tools/delete-trace.js +3 -3
package/dist/tools/deploy-rule.js +1 -1
package/dist/tools/evaluate-output.js +1 -1
package/dist/tools/evaluate-with-llm-judge.js +4 -4
package/dist/tools/get-traces.js +3 -3
package/dist/tools/list-rules.js +1 -1
package/dist/tools/log-trace.js +3 -3
package/package.json +2 -1
package/server.json +121 -2
package/dist/dashboard/assets/index-DTA8DzF_.js +0 -10

package/dist/dashboard/index.html CHANGED Viewed

@@ -6,7 +6,7 @@
     <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
     <!-- Stop shipping agents on vibes is filled from .claims.json brand.tagline at build time (vite.config.ts) — never restate the tagline here. -->
     <title>Iris — Stop shipping agents on vibes</title>
-    <script type="module" crossorigin src="/assets/index-DTA8DzF_.js"></script>
+    <script type="module" crossorigin src="/assets/index-CyzO6OC7.js"></script>
     <link rel="stylesheet" crossorigin href="/assets/index-D0cFfBqn.css">
   </head>
   <body>

package/dist/tools/delete-rule.js CHANGED Viewed

@@ -40,7 +40,7 @@ export function registerDeleteRuleTool(server, customRuleStore, evalEngine) {
             '',
             'Sibling tools — deploy_rule adds custom rules, list_rules enumerates them (including disabled ones, with `enabled: false`), evaluate_output runs them. delete_trace handles trace deletion (separate concern); log_trace / get_traces handle trace I/O. delete_rule is the DESTRUCTIVE remove path for the custom-rule store and the only MCP path that toggles a rule; it does NOT touch traces, eval_results, or built-in (non-custom) rules.',
             '',
-            'Behavior. Without `enabled`: DESTRUCTIVE — rewrites ~/.iris/custom-rules.json without the deleted row and appends a `rule.delete` entry to the audit log (~/.iris/audit.log). Not idempotent: deleting an already-deleted rule returns `deleted: false` rather than re-emitting the audit row. The rule stops firing immediately on the live process. With `enabled`: NOT destructive — the rule row stays, its `enabled` flag and `updatedAt` change, a `rule.toggle` audit entry is appended (none if the flag was already in that state), and the live engine unregisters (false) or re-registers (true) the rule so the change is immediate; a disabled rule is not loaded at the next boot either. Historical eval_results that reference this rule_id stay in the database — drift analytics + audit trail remain valid. Tenant-scoped in Cloud tier; OSS operates on LOCAL_TENANT. Rate-limited to 20 req/min on HTTP MCP.',
+            'Behavior. Without `enabled`: DESTRUCTIVE — rewrites ~/.iris/custom-rules.json without the deleted row and appends a `rule.delete` entry to the audit log (~/.iris/audit.log). Not idempotent: deleting an already-deleted rule returns `deleted: false` rather than re-emitting the audit row. The rule stops firing immediately on the live process. With `enabled`: NOT destructive — the rule row stays, its `enabled` flag and `updatedAt` change, a `rule.toggle` audit entry is appended (none if the flag was already in that state), and the live engine unregisters (false) or re-registers (true) the rule so the change is immediate; a disabled rule is not loaded at the next boot either. Historical eval_results that reference this rule_id stay in the database — drift analytics + audit trail remain valid. Operates on the local tenant. Rate-limited to 20 req/min on HTTP MCP.',
             '',
             'Output shape. Delete: `{ "deleted": boolean, "rule_id": string }` — `deleted=true` if a row was removed; `deleted=false` if no rule with that id existed. Toggle (enabled given): `{ "deleted": false, "toggled": boolean, "rule_id": string, "enabled"?: boolean, "rule"?: { ...the rule } }` — `toggled=true` with the rule\'s current state when the id exists (also when it was already in the requested state), `toggled=false` and no `rule` when it does not.',
             '',

package/dist/tools/delete-trace.js CHANGED Viewed

@@ -26,13 +26,13 @@ export function registerDeleteTraceTool(server, storage) {
             '',
             'Sibling tools — log_trace creates traces, get_traces queries them, evaluate_output / evaluate_with_llm_judge / verify_citations score them. delete_rule handles custom-rule deletion (separate concern); list_rules / deploy_rule manage the custom-rule lifecycle. delete_trace is the DESTRUCTIVE single-row remove for traces; it does NOT touch eval_results (preserved for audit + drift analytics), spans cascade automatically.',
             '',
-            'Behavior. DESTRUCTIVE — SQL DELETE scoped to the caller\'s tenant_id. Cascades: spans belonging to this trace are deleted (FK ON DELETE CASCADE); eval_results that referenced this trace have their trace_id set to NULL (FK ON DELETE SET NULL) so aggregate dashboards + historical scores remain valid even after the trace is gone. Not idempotent: deleting an already-deleted trace returns `deleted: false`. Does not emit an audit log entry in v0.4 — traces are user-scope data, not policy changes. Rate-limited to 20 req/min on HTTP MCP.',
+            'Behavior. DESTRUCTIVE — SQL DELETE scoped to the caller\'s tenant_id. Cascades: spans belonging to this trace are deleted (FK ON DELETE CASCADE); eval_results that referenced this trace have their trace_id set to NULL (FK ON DELETE SET NULL) so aggregate dashboards + historical scores remain valid even after the trace is gone. Not idempotent: deleting an already-deleted trace returns `deleted: false`. Does not emit an audit log entry — traces are user-scope data, not policy changes. Rate-limited to 20 req/min on HTTP MCP.',
             '',
             'Output shape. Returns JSON: `{ "deleted": boolean, "trace_id": string }`. `deleted=true` if a row was removed; `deleted=false` if no trace with that id existed (or it belonged to a different tenant — cross-tenant deletes silently fail).',
             '',
-            "Use when a trace was captured in error, contains sensitive data that must be removed for compliance (e.g., a customer exercises GDPR right-to-erasure), or when cleaning up test data. Combine with get_traces to find candidates: query with filters → review → delete_trace(id) per target. For bulk time-window deletion, use `deleteTracesOlderThan` via the CLI / retention config — delete_trace is the single-row surgical path.",
+            "Use when a trace was captured in error, contains sensitive data that must be removed for compliance (e.g., a customer exercises GDPR right-to-erasure), or when cleaning up test data. Combine with get_traces to find candidates: query with filters → review → delete_trace(id) per target. For bulk time-window deletion, set `retention.days` in config.json (default 30; the sweep runs when the server starts) — delete_trace is the single-row surgical path.",
             '',
-            "Don't use to clean up OLD data in bulk (use retention config with --retention-days). Don't use to PAUSE a trace — traces are immutable once stored; there's nothing to pause. Don't use to delete eval_results — eval_results survive their trace's deletion intentionally (for audit + drift analysis); they're pruned only by retention.",
+            "Don't use to clean up OLD data in bulk (use the `retention.days` setting in config.json; there is no command-line flag for it). Don't use to PAUSE a trace — traces are immutable once stored; there's nothing to pause. Don't use to delete eval_results — eval_results survive their trace's deletion intentionally (for audit + drift analysis); they're pruned only by retention.",
             '',
             'Parameters. trace_id is the only parameter; must match 32-char lowercase hex (Zod regex). The trace_id you pass is exactly what log_trace returned in its response, or what get_traces returned per row. Format mismatch fails Zod with 400 BEFORE the storage layer is touched. Cross-tenant trace_ids return `deleted: false` silently — they\'re invisible to the caller\'s tenant (prevents enumeration attacks; matches delete_rule\'s tenant-isolation contract).',
             '',

package/dist/tools/deploy-rule.js CHANGED Viewed

@@ -153,7 +153,7 @@ export function registerDeployRuleTool(server, customRuleStore, evalEngine) {
             '',
             'Sibling tools — list_rules enumerates deployed rules, delete_rule removes them (or disables/re-enables them with its `enabled` argument), evaluate_output runs them. log_trace / get_traces / delete_trace handle the trace lifecycle separately; evaluate_with_llm_judge / verify_citations run semantic scoring (not heuristic-rule-driven). deploy_rule is the WRITE path that grows the custom-rule library.',
             '',
-            'Behavior. Writes a row to ~/.iris/custom-rules.json (atomic write via temp file + rename) and appends a `rule.deploy` entry to the audit log (~/.iris/audit.log). The rule activates immediately for the running process and persists across restarts. Each call mints a fresh rule_id. Rule names are unique among deployed rules: deploying a name that is already deployed is REJECTED unless `replace: true`, in which case the existing same-named rule(s) are deleted (audit `rule.delete` rows written, unregistered from the live engine) and the new rule takes their place under a new id — the response lists what was replaced. Tenant-scoped in Cloud tier; OSS rules are owned by LOCAL_TENANT. Rate-limited to 20 req/min on HTTP MCP.',
+            'Behavior. Writes a row to ~/.iris/custom-rules.json (atomic write via temp file + rename) and appends a `rule.deploy` entry to the audit log (~/.iris/audit.log). The rule activates immediately for the running process and persists across restarts. Each call mints a fresh rule_id. Rule names are unique among deployed rules: deploying a name that is already deployed is REJECTED unless `replace: true`, in which case the existing same-named rule(s) are deleted (audit `rule.delete` rows written, unregistered from the live engine) and the new rule takes their place under a new id — the response lists what was replaced. Rules are owned by the local tenant. Rate-limited to 20 req/min on HTTP MCP.',
             '',
             'Output shape. Returns JSON: `{ "rule": { "id": "rule-XXXX", "name", "description", "evalType", "severity", "definition", "enabled": true, "createdAt", "updatedAt", "version": 1, "sourceMomentId?" }, "replaced?": [{ "id", "evalType", "severity" }], "warning?": string }`. The returned rule is the canonical persisted form; save the `id` if you plan to disable or delete later. `replaced` and `warning` appear only when `replace: true` removed an earlier rule of the same name.',
             '',

package/dist/tools/evaluate-output.js CHANGED Viewed

@@ -57,7 +57,7 @@ export function registerEvaluateOutputTool(server, storage, evalEngine) {
             '',
             'Sibling tools — evaluate_with_llm_judge runs semantic LLM-based scoring (slower, costs money; this tool is heuristic, free, deterministic), verify_citations checks citation grounding specifically, log_trace records executions, get_traces queries them, list_rules / deploy_rule / delete_rule manage the custom-rule lifecycle. evaluate_output is the FAST PATH for length / keyword / PII / injection / cost-threshold checks where rules are sufficient.',
             '',
-            'Behavior. Deterministic, in-process scoring — same inputs always produce the same result. Writes one eval_result row to Iris storage (linked to trace_id if provided; unlinked otherwise). No external network calls in heuristic mode (v0.4 adds an llm_as_judge eval_type that DOES call LLM APIs; see the separate evaluate_with_llm_judge tool for that). Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Runs in ~5-50ms for rule-based evaluation.',
+            'Behavior. Deterministic, in-process scoring — same inputs always produce the same result. Writes one eval_result row to Iris storage (linked to trace_id if provided; unlinked otherwise). No external network calls — semantic scoring is a separate tool (evaluate_with_llm_judge) that needs an API key you supply. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Runs in-process; no provider is called.',
             '',
             'Output shape. Returns JSON: `{ "id": "<uuid>", "eval_type": "<bundle that ran>", "score": 0..1, "passed": boolean, "critical_failures?": string[], "critical_skipped?": string[], "rule_results": [{ "ruleName", "ruleId?", "category?", "critical", "criticalSource", "passed", "score", "message", "skipped?", "skipReason?", "budgetExceeded?", "configInvalid?" }], "suggestions": string[], "rules_evaluated": number, "rules_skipped": number, "insufficient_data": boolean, "categories?": { "<bundle>": { "score": number|null, "passed": boolean|null, "rules_evaluated", "rules_skipped", "insufficient_data", "critical_failures?", "critical_skipped?" } }, "note?": string }`. `ruleId` is present on results produced by a deployed rule (rule-XXXX) so two rules sharing a name stay distinguishable. `categories` appears only for eval_type="all" and carries one entry per bundle that had rules, each with the same threshold + critical-veto semantics as a single-bundle run; `category` on each rule result says which bundle it came from. `insufficient_data=true` means no applicable rules fired (e.g., safety eval with only cost data). Inside `categories`, a bundle that evaluated no rule (every rule skipped for missing context — cost without `cost_usd`, relevance without `input`) reports `passed: null` and `score: null` with `insufficient_data: true`: it was not judged, so it is neither passing nor failing, and it does not count toward the overall verdict. The top-level `passed` stays a boolean and is false when NOTHING was evaluated — a gate keyed on it fails closed; read `insufficient_data` to tell "failed" from "not judged". `note` appears only when eval_type was omitted, saying that the default ran every bundle.',
             '',

package/dist/tools/evaluate-with-llm-judge.js CHANGED Viewed

@@ -59,17 +59,17 @@ export function registerEvaluateWithLLMJudgeTool(server, storage) {
     server.registerTool('evaluate_with_llm_judge', {
         title: 'Evaluate With LLM Judge',
         description: [
-            'Score agent output using an LLM as the judge (Anthropic or OpenAI). Returns a calibrated 0..1 score with rationale, per-dimension breakdown, and exact cost.',
+            'Score agent output using an LLM as the judge (Anthropic or OpenAI). Returns a 0..1 score with rationale, per-dimension breakdown, and exact cost. The judge\'s own accuracy is measurable on a key you supply and is not yet published — see https://iris-eval.com/proof.',
             '',
-            'Sibling tools — evaluate_output runs heuristic rules (free, deterministic, ~ms latency, no API key needed); this tool runs LLM-based semantic scoring (paid, 1-10s latency, requires API key). verify_citations is a SPECIALIZED form of LLM judging that focuses on citation grounding only. log_trace / get_traces handle trace I/O; list_rules / deploy_rule / delete_rule manage heuristic-rule lifecycle. evaluate_with_llm_judge is the GENERAL semantic-scoring path.',
+            'Sibling tools — evaluate_output runs heuristic rules (free, deterministic, no API key needed); this tool runs LLM-based semantic scoring (paid, requires an API key). verify_citations is a SPECIALIZED form of LLM judging that focuses on citation grounding only. log_trace / get_traces handle trace I/O; list_rules / deploy_rule / delete_rule manage heuristic-rule lifecycle. evaluate_with_llm_judge is the GENERAL semantic-scoring path.',
             '',
-            'Behavior. Calls an external LLM API (Anthropic or OpenAI) — costs money per call, takes 1-10 seconds, respects an IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL cap. Non-deterministic at temperature > 0; default temperature=0 gives near-deterministic scores. Writes one eval_result row to Iris storage (linked to trace_id if provided) plus captures provider response id + latency + token counts + cost in the rule_results payload. Rate-limited to 20 req/min on HTTP MCP; your LLM provider also enforces its own rate limits (we transparently retry once on 429).',
+            'Behavior. Calls an external LLM API (Anthropic or OpenAI) — costs money per call, takes as long as the provider takes, respects an IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL cap. Non-deterministic at temperature > 0; default temperature=0 gives near-deterministic scores. Writes one eval_result row to Iris storage (linked to trace_id if provided) plus captures provider response id + latency + token counts + cost in the rule_results payload. Rate-limited to 20 req/min on HTTP MCP; your LLM provider also enforces its own rate limits (we transparently retry once on 429).',
             '',
             'Output shape. Returns JSON: `{ "id": "<uuid>", "score": 0..1, "passed": boolean, "rationale": string, "dimensions": {...}, "model": string, "provider": "anthropic"|"openai", "template": string, "input_tokens": number, "output_tokens": number, "cost_usd": number, "latency_ms": number }`. `dimensions` has per-dimension sub-scores (e.g., accuracy template returns `{factual_claims, citations, internal_consistency}`).',
             '',
             'Use when heuristic rules (via evaluate_output) are too coarse for the quality signal you need — semantic correctness, factual accuracy vs a reference, RAG faithfulness to sources, nuanced safety/helpfulness. Pick the template that matches: `accuracy` (hallucination detection), `helpfulness` (does it address the ask), `safety` (harm potential beyond regex PII), `correctness` (vs reference answer — pass `expected`), `faithfulness` (RAG grounding — pass `source_material`).',
             '',
-            "Don't use for simple regex/length/keyword checks (use evaluate_output with heuristic rules — they're free, deterministic, 1000x faster). Don't use without an API key set (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY). Don't use on very large outputs (>8K tokens) without raising max_cost_usd — the pre-check will refuse the call.",
+            "Don't use for simple regex/length/keyword checks (use evaluate_output with heuristic rules — they're free, deterministic and in-process). Don't use without an API key set (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY). Don't use on very large outputs (>8K tokens) without raising max_cost_usd — the pre-check will refuse the call.",
             '',
             'Parameters. model is required (no default — pick consciously since cost varies 100x across models). provider is auto-detected from the model name; override only for ambiguous IDs. expected is REQUIRED when template="correctness" (the reference answer to compare against); ignored for other templates. source_material is REQUIRED when template="faithfulness" (the RAG sources to ground against); ignored otherwise. input is optional but improves scoring on helpfulness/safety templates (gives the judge the user prompt that produced the output). max_cost_usd defaults to env var IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or $0.25 — the worst-case cost is computed BEFORE the call (input_tokens × prompt_price + max_output_tokens × completion_price, PLUS the same for the one retry that fires if the judge\'s first reply is not valid JSON); call refused upfront if that two-attempt worst case would exceed. When a retry does run, the reported input_tokens / output_tokens / cost_usd / latency_ms are totals across both attempts. max_output_tokens caps the judge response (default 512, max 4096); higher = more rationale detail + more cost. temperature default 0 (deterministic). timeout_ms default 60000. trace_id optional but recommended (links eval to trace in dashboard). Defaults: temperature=0, max_output_tokens=512, max_cost_usd=$0.25, timeout_ms=60000.',
             '',

package/dist/tools/get-traces.js CHANGED Viewed

@@ -52,7 +52,7 @@ export function addTraceRangeIssues(args, ctx) {
     }
 }
 const inputSchema = {
-    agent_name: z.string().optional().describe('Filter by agent name — exact match (no wildcards in v0.4)'),
+    agent_name: z.string().optional().describe('Filter by agent name — exact match (no wildcards)'),
     framework: z.string().optional().describe('Filter by agent framework — exact match (e.g., langchain, autogen)'),
     since: isoTimestamp.optional().describe('ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as "no bound"'),
     until: isoTimestamp.optional().describe('ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since`'),
@@ -83,9 +83,9 @@ export function registerGetTracesTool(server, storage) {
             '',
             'Use when you need historical data: investigating a past failure, computing quality trends, comparing agents, or feeding an analytics job. Set `agent_name` / `framework` / `since` / `until` to narrow the query. Set `min_score` / `max_score` to surface outliers. Set `sort_by: "cost_usd"` + `sort_order: "desc"` to find the most expensive traces. Set `include_summary: true` when you want dashboard-style aggregates in one round-trip.',
             '',
-            'Don\'t use to score a trace (use evaluate_output). Don\'t use to create a trace (use log_trace). Don\'t use as a live event stream — it\'s a query, not a subscription; poll with exponential backoff or use the dashboard\'s SSE endpoint for real-time.',
+            'Don\'t use to score a trace (use evaluate_output). Don\'t use to create a trace (use log_trace). Don\'t use as a live event stream — it\'s a query, not a subscription, and Iris has no event-stream endpoint; poll with exponential backoff.',
             '',
-            'Parameters. limit defaults to 50, max 1000 (anything higher returns 400). offset is zero-based pagination (non-negative integer). since / until must be ISO 8601 timestamps or dates — `since` is inclusive (timestamp >= since), `until` is inclusive (timestamp <= until), and `since` may not be later than `until`. min_score / max_score are 0..1 and filter on the LATEST eval per trace, not all evals (so a trace with one failing + one passing eval may or may not match depending on which landed last); min_score may not exceed max_score. Combining since + sort_by="latency_ms" + sort_order="desc" is the canonical "find slow recent traces" query. include_summary returns dashboard-style aggregates in the SAME response (saves a round-trip; use true for dashboard ingest, false for analytics queries that don\'t need them). agent_name and framework are exact-match (no wildcards in v0.4). Defaults: limit=50, offset=0, sort_by="timestamp", sort_order="desc", include_summary=false.',
+            'Parameters. limit defaults to 50, max 1000 (anything higher returns 400). offset is zero-based pagination (non-negative integer). since / until must be ISO 8601 timestamps or dates — `since` is inclusive (timestamp >= since), `until` is inclusive (timestamp <= until), and `since` may not be later than `until`. min_score / max_score are 0..1 and filter on the LATEST eval per trace, not all evals (so a trace with one failing + one passing eval may or may not match depending on which landed last); min_score may not exceed max_score. Combining since + sort_by="latency_ms" + sort_order="desc" is the canonical "find slow recent traces" query. include_summary returns dashboard-style aggregates in the SAME response (saves a round-trip; use true for dashboard ingest, false for analytics queries that don\'t need them). agent_name and framework are exact-match (no wildcards). Defaults: limit=50, offset=0, sort_by="timestamp", sort_order="desc", include_summary=false.',
             '',
             'Error modes. Returns 400 on invalid sort_by / sort_order (Zod enum). Returns 400 if limit > 1000 or offset < 0. Returns 400 — naming both values — on an empty range: min_score > max_score, since later than until, a score outside 0..1, or a since/until that is not an ISO 8601 timestamp or date (an unparseable bound is refused, never silently ignored). Returns 429 when HTTP rate limit exceeded. Storage failures propagate as 500. Empty result with `total: 0` on no matches (not an error).',
         ].join('\n'),

package/dist/tools/list-rules.js CHANGED Viewed

@@ -32,7 +32,7 @@ export function registerListRulesTool(server, customRuleStore, evalEngine) {
             '',
             'Sibling tools — deploy_rule adds custom rules, delete_rule removes them, evaluate_output runs them against agent output. log_trace / get_traces / delete_trace handle the trace lifecycle separately. list_rules is the READ path for the custom-rule store; nothing else exposes the inventory.',
             '',
-            'Behavior. Pure read of ~/.iris/custom-rules.json (in-memory cached; no disk read per call after server boot). No mutation, no external network. Tenant-scoped in Cloud tier; OSS returns all rules for the single local tenant. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Returns in <5ms.',
+            'Behavior. Pure read of ~/.iris/custom-rules.json (in-memory cached; no disk read per call after server boot). No mutation, no external network. Returns the rules of the local tenant. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio.',
             '',
             'Output shape. Returns JSON: `{ "rules": [{ "id": "rule-XXXX", "name", "description", "evalType", "severity", "definition": { name, type, config, weight? }, "enabled": boolean, "createdAt": ISO timestamp, "updatedAt": ISO timestamp, "version": number, "sourceMomentId?": string }], "total": number, "enabled_count": number, "built_in": [{ "name", "category", "weight", "critical": boolean, "criticalSource": "default" | "config" }] }`. Empty `rules` array + total=0 when no custom rules are deployed. A deployed rule fires only on evaluate_output calls whose eval_type equals its evalType (or eval_type="all", which runs every bundle). `built_in` is the shipped rule set, always present and NOT narrowed by the filters; `total` and `enabled_count` count custom rules only.',
             '',

package/dist/tools/log-trace.js CHANGED Viewed

@@ -74,17 +74,17 @@ export function registerLogTraceTool(server, storage) {
             '',
             'Sibling tools — evaluate_output runs heuristic scoring on the trace; evaluate_with_llm_judge runs semantic LLM-based scoring; verify_citations checks citation grounding; get_traces queries stored traces; delete_trace removes a single trace; list_rules / deploy_rule / delete_rule manage custom evaluation rules. log_trace is the WRITE path that records executions; everything else reads, scores, or manages around it.',
             '',
-            'Behavior. Writes one row to Iris storage (SQLite by default; Postgres in Cloud tier). When IRIS_OTEL_ENDPOINT is set, ALSO fires a best-effort async export to the configured OTLP/HTTP collector (Jaeger, Tempo, Datadog OTLP, OTEL Collector). The OTel export is fire-and-forget — its success does not affect the tool response; failures are logged but the trace is still stored locally. No authentication in stdio mode. HTTP mode requires a Bearer token ONLY when --api-key / IRIS_API_KEY is set (recommended); with no key configured the auth middleware is a pass-through and writes are unauthenticated — a default HTTP server is protected by its loopback bind (127.0.0.1) and Origin validation, not by a credential. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Not idempotent: each call mints a fresh trace_id, so resubmitting the same payload creates a duplicate trace.',
+            'Behavior. Writes one row to Iris storage (SQLite). When IRIS_OTEL_ENDPOINT is set, ALSO fires a best-effort async export to the configured OTLP/HTTP collector (Jaeger, Tempo, Datadog OTLP, OTEL Collector). The OTel export is fire-and-forget — its success does not affect the tool response; failures are logged but the trace is still stored locally. No authentication in stdio mode. HTTP mode requires a Bearer token ONLY when --api-key / IRIS_API_KEY is set (recommended); with no key configured the auth middleware is a pass-through and writes are unauthenticated — a default HTTP server is protected by its loopback bind (127.0.0.1) and Origin validation, not by a credential. Rate-limited to 20 req/min on HTTP MCP, unlimited on stdio. Not idempotent: each call mints a fresh trace_id, so resubmitting the same payload creates a duplicate trace.',
             '',
             'Output shape. Returns a JSON string: `{ "trace_id": "<32-hex>", "status": "stored" }`. The trace_id is the key you pass to evaluate_output or get_traces afterwards.',
             '',
             'Use when you want to record an agent execution for later evaluation, analysis, or audit. Call it AFTER the agent has produced output; call evaluate_output afterwards to score it; call get_traces to query historical traces. Store rich context: spans (span tree), tool_calls (which tools were invoked with latency/errors), token_usage, cost_usd, metadata (arbitrary key-value). All optional except agent_name.',
             '',
-            'Don\'t use when you only need a transient log (use console logging). Don\'t use to update an existing trace — there is no update path in v0.4 (traces are immutable once stored).',
+            'Don\'t use when you only need a transient log (use console logging). Don\'t use to update an existing trace — there is no update path (traces are immutable once stored).',
             '',
             'Parameters. agent_name is required; everything else is optional. token_usage and cost_usd are summary fields — if you ALSO pass spans with per-tool-call costs, the summary fields are treated as authoritative (no auto-aggregation). spans without an explicit start_time fall back to the trace timestamp; spans with an end_time get a duration_ms derived. metadata is opaque key-value (queryable in the dashboard, not via get_traces filters). tool_calls record per-tool latency + errors; missing latency_ms means "not reported," not "zero." Defaults: span.kind="INTERNAL", span.status_code="UNSET", timestamp=now() if omitted.',
             '',
-            'Error modes. Throws on missing agent_name. Throws on malformed span or tool_call objects (Zod rejects). Returns 500 on storage failure (disk full, DB locked). Never blocks on the agent — returns within ~50ms for typical payloads.',
+            'Error modes. Throws on missing agent_name. Throws on malformed span or tool_call objects (Zod rejects). Returns 500 on storage failure (disk full, DB locked). Never blocks on the agent — the local write is synchronous and the OTel export is asynchronous.',
         ].join('\n'),
         // Strict at the MCP boundary (unknown args rejected, not stripped).
         // The dashboard's HTTP ingest builds its own — equally strict —

package/package.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "name": "@iris-eval/mcp-server",
-  "version": "0.8.0",
+  "version": "0.8.2",
   "description": "Stop shipping agents on vibes. Score every agent output for quality, safety, and cost.",
   "mcpName": "io.github.iris-eval/mcp-server",
   "type": "module",
@@ -33,6 +33,7 @@
     "claims:check-hardcoded": "node scripts/claims/check-no-hardcoded.mjs",
     "proof": "tsx proof/run.ts",
     "proof:typecheck": "tsc -p proof/tsconfig.json",
+    "proof:review": "tsx proof/human-review.ts",
     "llms:render": "node scripts/claims/render-llms.mjs",
     "llms:check": "node scripts/claims/render-llms.mjs --check",
     "proof:judge": "tsx proof/judge/run.ts",

package/server.json CHANGED Viewed

@@ -6,12 +6,12 @@
     "url": "https://github.com/iris-eval/mcp-server",
     "source": "github"
   },
-  "version": "0.8.0",
+  "version": "0.8.2",
   "packages": [
     {
       "registryType": "npm",
       "identifier": "@iris-eval/mcp-server",
-      "version": "0.8.0",
+      "version": "0.8.2",
       "transport": {
         "type": "stdio"
       },
@@ -36,6 +36,125 @@
           "format": "string",
           "isSecret": false,
           "name": "IRIS_LOG_LEVEL"
+        },
+        {
+          "description": "Directory for the database, custom rules and preferences (default ~/.iris)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_HOME"
+        },
+        {
+          "description": "Enables evaluate_with_llm_judge and verify_citations with an Anthropic model. Optional: the deterministic rules never need a key",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": true,
+          "name": "IRIS_ANTHROPIC_API_KEY"
+        },
+        {
+          "description": "Enables evaluate_with_llm_judge and verify_citations with an OpenAI model. Optional: the deterministic rules never need a key",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": true,
+          "name": "IRIS_OPENAI_API_KEY"
+        },
+        {
+          "description": "Hard cap on the worst-case cost of one judge or citation call (default 0.25)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL"
+        },
+        {
+          "description": "Let verify_citations fetch URLs (off by default)",
+          "isRequired": false,
+          "format": "boolean",
+          "isSecret": false,
+          "name": "IRIS_CITATION_ALLOW_FETCH"
+        },
+        {
+          "description": "Start the dashboard alongside the server (same as --dashboard)",
+          "isRequired": false,
+          "format": "boolean",
+          "isSecret": false,
+          "name": "IRIS_DASHBOARD"
+        },
+        {
+          "description": "Dashboard port (default 6920); IRIS_PORT is the MCP HTTP transport port (default 3000)",
+          "isRequired": false,
+          "format": "number",
+          "isSecret": false,
+          "name": "IRIS_DASHBOARD_PORT"
+        },
+        {
+          "description": "MCP HTTP transport port (default 3000); only used with --transport http",
+          "isRequired": false,
+          "format": "number",
+          "isSecret": false,
+          "name": "IRIS_PORT"
+        },
+        {
+          "description": "OTLP/HTTP collector URL; when set, every stored trace is also exported as OpenTelemetry spans (best-effort, asynchronous)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_OTEL_ENDPOINT"
+        },
+        {
+          "description": "Comma-separated domain allowlist for verify_citations fetches (only consulted when IRIS_CITATION_ALLOW_FETCH is on)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_CITATION_DOMAINS"
+        },
+        {
+          "description": "Transport: stdio (default) or http; same as --transport",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_TRANSPORT"
+        },
+        {
+          "description": "Bind host for the HTTP transport (default 127.0.0.1); only used with --transport http",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_HOST"
+        },
+        {
+          "description": "Comma-separated Origin allowlist for the HTTP transport and dashboard CORS (default http://localhost:*)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_ALLOWED_ORIGINS"
+        },
+        {
+          "description": "Bind host for the dashboard (default 127.0.0.1); same as --dashboard-host",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_DASHBOARD_HOST"
+        },
+        {
+          "description": "Comma-separated key=value headers sent with every OTLP export (for example an auth token for your collector)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_OTEL_HEADERS"
+        },
+        {
+          "description": "service.name resource attribute on exported spans (default iris-mcp)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false,
+          "name": "IRIS_OTEL_SERVICE_NAME"
+        },
+        {
+          "description": "Timeout in milliseconds for each OTLP export request",
+          "isRequired": false,
+          "format": "number",
+          "isSecret": false,
+          "name": "IRIS_OTEL_TIMEOUT_MS"
         }
       ]
     }