failproofai 1.0.1-beta.0 → 1.0.1

package/.next/standalone/docs-old/agenteye/evaluation-suite.mdx

@@ -0,0 +1,402 @@
+---
+title: "Evaluation Suite"
+description: "Failproof AI Observability can automatically score every finished agent run for quality: you supply a small scoring service, and Observability handles the rest."
+---
+
+
+Failproof AI Observability can automatically score every finished agent run for quality: you supply a small scoring service, and Observability handles the rest. Use it to track the dimensions you care about (helpfulness, tool efficiency, factuality, safety; you choose), catch regressions early, and compare agents or environments at a glance. Scoring is opt-in: the pipeline does nothing until you set `EVALUATOR_ENDPOINT` on the server.
+
+> **Note:** You define the score dimensions. Your evaluator can return any numeric keys it likes; Observability stores, trends, and displays whatever you send back.
+
+## At a glance
+
+1. **Write a scorer.** Stand up a small HTTP service that reads a session transcript and returns scores. Observability ships a working reference you can copy. See [Writing an evaluator with the SDK](#writing-an-evaluator-with-the-sdk).
+2. **Point Observability at it.** Set `EVALUATOR_ENDPOINT` (and a shared `EVALUATOR_TOKEN`) on the server process.
+3. **Watch the scores land.** Every completed session is scored automatically; results show up on the session detail page, the sessions grid, and saved dashboards.
+
+![A session detail view with the evaluation summary, per-dimension score bars, and reasoning text in the right rail](/agenteye/images/session-detail.png)
+
+*Once an evaluator is configured, each completed run is scored and the results appear in the session's right rail: the summary on top, then per-dimension score bars with reasoning.*
+
+---
+
+## How it works
+
+```mermaid
+flowchart LR
+    ING["ingest /events<br/>agent_end"] --> SRV["Observability server"]
+    SRV -->|"POST /evaluate"| EV["Evaluator service"]
+    EV -->|"done or pending"| SRV
+    SRV -->|"poll GET /evaluate/{job_id}"| EV
+    EV -->|"done"| SRV
+    SRV --> RES["evaluations<br/>terminal results"]
+```
+
+When the Observability SDK emits an `agent_end` event for a session, the server
+schedules an evaluation. It then POSTs the full event transcript to your
+evaluator service, which can either:
+
+- **Return the result inline** with `{"status":"done", "scores":{...}, "reasoning":{...}, "summary":"..."}`. The
+  result is appended to the session's evaluation timeline. `reasoning` and
+  `summary` are optional.
+- **Defer** with `{"status":"pending", "job_id":"abc-123"}`. Observability then
+  calls `GET {EVALUATOR_ENDPOINT}/evaluate/abc-123` until your evaluator
+  returns `{"status":"done", ...}` or `{"status":"error", "error":"..."}`.
+
+  The polling cadence is per-job: a `pending` response may include
+  `next_poll_secs` to override; otherwise Observability uses the
+  `default_poll_interval_secs` value from `GET /config`; otherwise the server
+  falls back to `EVALUATOR_POLLING_INTERVAL_SECS` (default 10s). All values
+  are clamped to [1s, 1h].
+
+Sessions that never emit `agent_end` (for example, a crashed agent process)
+can also be picked up: the evaluator's `GET /config` may return
+`{"inactivity_timeout_secs": 1800}`, and Observability will evaluate any session
+that has gone idle for that long. Set the field to `null` or omit it to
+disable this fallback.
+
+The pipeline is fully no-op when `EVALUATOR_ENDPOINT` is unset.
+
+A session can accumulate **multiple terminal evaluations over time**: each
+`agent_end` event (and each manual re-eval from the dashboard) appends a
+fresh evaluation row. This is the supported way to evaluate a resumed
+conversation: a user ends an agent, comes back later, sends more events,
+ends the agent again, and a second evaluation runs against the full updated
+transcript. The dashboard renders the most-recent evaluation as the
+headline and the prior evaluations as a collapsible timeline. While one
+evaluation is running for a session, additional `agent_end` events for that
+session are ignored; the next one after the running evaluation completes
+will enqueue a fresh evaluation as usual.
+
+The inactivity fallback re-engages on resumed sessions too: if new events
+arrive after a previous terminal evaluation and the session then goes idle
+past `inactivity_timeout_secs`, a fresh evaluation is enqueued.
+
+Transient failures (5xx, 429, timeouts, network errors) are retried with
+exponential backoff up to `EVALUATOR_MAX_ATTEMPTS`; 4xx responses are
+terminal. Observability is safe to run with multiple horizontally-scaled server
+instances; work is partitioned so the same session is never dispatched
+twice concurrently.
+
+---
+
+## HTTP contract
+
+Every authenticated route uses **bearer token auth**. The same value must be
+configured on both sides:
+
+- Observability server: env var `EVALUATOR_TOKEN`
+- Evaluator service: configured the same way (the `agenteye-evaluator` SDK
+  reads `EVALUATOR_TOKEN` by convention)
+
+If `EVALUATOR_TOKEN` is unset, the server sends no `Authorization` header; the
+evaluator may then accept anonymous requests, which is fine for an
+internal-only network but discouraged on the public internet.
+
+### Routes the evaluator must serve
+
+| Route | Body / params | Response |
+|---|---|---|
+| `GET /health` | none | `{"status":"ok"}` (open, no auth) |
+| `GET /config` | none | `{"inactivity_timeout_secs": <int> \| null, "default_poll_interval_secs": <int> \| omitted}` |
+| `POST /evaluate` | `EvalRequest` JSON | `{"status":"done", ...}` or `{"status":"pending", "job_id":"..."}` |
+| `GET /evaluate/{id}` | none | same response shape as `/evaluate` |
+
+### `EvalRequest` body sent by the server
+
+```json
+{
+  "schema_version": "1",
+  "session_id":     "session-abc123",
+  "agent_id":       "planner",
+  "environment":    "production",
+  "started_at":     "2026-05-10T12:00:00Z",
+  "ended_at":       "2026-05-10T12:05:00Z",
+  "events": [
+    { "id": 1234, "ts": "...", "event_type": "agent_start", "payload": { ... } },
+    ...
+  ]
+}
+```
+
+### Response shapes
+
+**Sync (done):**
+
+```json
+{
+  "status": "done",
+  "scores": { "helpfulness": 0.85, "tool_efficiency": 0.6 },
+  "reasoning": {
+    "helpfulness": "answered the question directly with citations",
+    "tool_efficiency": "called list_files three times when one would have done"
+  },
+  "summary": "strong answer quality, weak tool selection"
+}
+```
+
+`reasoning` (a per-score justification map) and `summary` (an overall
+one-paragraph narrative) are both optional. Keys in `reasoning` should
+mirror keys in `scores`; the dashboard renders each entry inline under
+its score bar. Older evaluators that return only `scores` continue to
+work unchanged; `reasoning` and `summary` simply read as null and
+the corresponding UI affordances are omitted.
+
+**Async (deferred):**
+
+```json
+{ "status": "pending", "job_id": "abc-123", "next_poll_secs": 30 }
+```
+
+`next_poll_secs` is optional; if omitted the server falls back to the
+evaluator's `default_poll_interval_secs` from `/config`, then to its own
+`EVALUATOR_POLLING_INTERVAL_SECS` env var.
+
+**Terminal evaluator-side error:**
+
+```json
+{ "status": "error", "error": "model service unavailable" }
+```
+
+The server treats any other 2xx body as a protocol error and records a
+terminal `error` for the session.
+
+---
+
+## Writing an evaluator with the SDK
+
+You don't have to implement the HTTP contract by hand. The `agenteye-evaluator`
+Python package gives you a typed FastAPI wrapper that handles auth, routing, and
+the request/response shapes for you.
+
+Failproof AI Observability also ships a **working reference evaluator** that
+scores `helpfulness`, `tool_efficiency`, and `factuality` from the shape of the
+transcript. Copy it as a starting point and swap in your own logic: an LLM
+judge, a rule engine, whatever fits your quality bar.
+
+Minimum viable evaluator:
+
+```python
+import os
+from agenteye_evaluator import Evaluator, EvalRequest, EvalResponse
+
+app = Evaluator(token=os.environ["EVALUATOR_TOKEN"])
+
+@app.evaluator
+def run(req: EvalRequest) -> EvalResponse:
+    # Inspect req.events (the full session transcript) and return scores.
+    tool_calls = sum(1 for e in req.events if e.event_type == "tool_use")
+    return EvalResponse(
+        scores={"tool_calls": float(tool_calls)},
+        reasoning={"tool_calls": f"{tool_calls} tool invocations in the transcript"},
+        summary="tight tool loop" if tool_calls < 5 else "agent looped on tools",
+    )
+```
+
+The `app` instance runs under any ASGI server, so `uvicorn module:app` starts it.
+
+For evaluators that need to defer expensive work, return `JobPending`
+instead and register a `@app.job_lookup` handler; the Observability server
+polls `GET /evaluate/{job_id}` until you return a terminal status or the
+`EVALUATOR_MAX_POLL_DURATION_SECS` cap (default 1 h) elapses.
+
+The full API reference, async pattern, and event schema are documented in the
+`agenteye-evaluator` SDK's README.
+
+---
+
+## Running your evaluator
+
+The evaluator is **your service** — Failproof AI Observability does not ship a
+default evaluator, so you build and run it wherever you run your own services.
+It runs under any ASGI server (for example `uvicorn my_evaluator:app`); serve
+the `/health`, `/config`, and `/evaluate` routes from the
+[HTTP contract](#http-contract), then point the server at it (see
+[Configuring the server](#configuring-the-server)).
+
+Once the evaluator is reachable, `GET /health` returns `{"status":"ok"}`. After
+an agent runs end-to-end, `GET /evaluations` on the server returns a row with
+`status: "done"` and the scores your evaluator produced.
+
+---
+
+## Configuring the server
+
+Set on the server process:
+
+| Env var | Meaning |
+|---|---|
+| `EVALUATOR_ENDPOINT` | Base URL of your evaluator (`http://evaluator:9000`). Unset = pipeline disabled. |
+| `EVALUATOR_TOKEN` | Bearer token. Must equal the value the evaluator service is configured with. |
+| `EVALUATOR_WORKERS` | Worker tasks per server instance (default 2). |
+| `EVALUATOR_CLAIM_BATCH` | Rows claimed per worker tick (default 4). Batches are processed **concurrently**; effective concurrency on your evaluator endpoint is `EVALUATOR_WORKERS × EVALUATOR_CLAIM_BATCH`. |
+| `EVALUATOR_POLL_IDLE_SECS` | How long a worker sleeps between dispatch attempts when no evaluation is due (default 2s). |
+| `EVALUATOR_POLLING_INTERVAL_SECS` | Final fallback for `GET /evaluate/{id}` cadence when neither the per-response `next_poll_secs` nor the evaluator's `default_poll_interval_secs` is set (default 10s). |
+| `EVALUATOR_REQUEST_TIMEOUT_MS` | Per-request timeout (default 30000). |
+| `EVALUATOR_MAX_ATTEMPTS` | After this many transient failures the result is recorded as terminal `error` (default 5). |
+| `EVALUATOR_CONFIG_REFRESH_SECS` | `GET /config` cadence (default 300). |
+| `EVALUATOR_MAX_POLL_DURATION_SECS` | Maximum wallclock time a session may remain in the polling queue before it's terminated as `timeout` (default 3600s). Guards against an evaluator that keeps returning `pending` forever. |
+
+To turn on automatic scoring, set both `EVALUATOR_ENDPOINT` and
+`EVALUATOR_TOKEN` on the server, then restart it to pick up the change. With
+`EVALUATOR_ENDPOINT` unset the pipeline stays a no-op.
+
+The tuning knobs above are optional; set the corresponding environment
+variables on the server only if you need to override the defaults.
+
+---
+
+## API reference
+
+| Method | Path | Required permission | Purpose |
+|---|---|---|---|
+| `GET` | `/evaluations` | `evaluations:read` | Query terminal results. Supports `session_id`, `agent_id`, `environment`, `status` (`done`/`error`/`timeout`), `ts_from`, `ts_to`, `cursor`, `limit`, `score_filters`, `latest_per_session`. `limit` defaults to 50 and is capped at 200 (note this differs from `/events`, which caps at 1000). `environment` accepts a comma-separated list (e.g. `environment=prod,staging`); single values still work. With `latest_per_session=true` the response contains at most one row per `session_id` (the most recent by `completed_at`) used by the sessions-list page to collapse a session's evaluation timeline to its current headline. Defaults to false (returns the full history). |
+| `GET` | `/evaluations/aggregate` | `evaluations:read` | Rolled-up eval health for a filtered slice: total count, a done/error/timeout breakdown, per-score-key stats (count/avg/min/max/p50 over the arbitrary `scores` keys), and a time-bucketed timeline. Accepts the **same filter params as `/evaluations`** plus `featured_keys` (CSV of score keys to trend) and `latest_per_session`. Powers the Dashboards feature; metrics are exact over the whole matching set, not sampled. |
+| `GET` | `/evaluations/environments` | `evaluations:read` | Distinct environment values from the `evaluations` table. Used to populate filter dropdowns scoped to evaluation-readable data. |
+| `GET` | `/evaluation-jobs` | `evaluations:read` | Visibility into in-flight evaluations. Filter by `status` (`pending`/`polling`). |
+| `GET` | `/events` | `events:read` | Stream a session's raw events. Supports `session_id`, `agent_id`, `event_type` (CSV), `environment` (CSV), `ts_from`, `ts_to`, `cursor`, `limit`, and `order`. `order` is `desc` (newest-first, the default) or `asc` (oldest-first); an unrecognized value falls back to `desc`. Cursor-paginate via the response's `next_cursor` (an event id): pass it back as `cursor` to get the next page; with `asc` the next page is the events after that id, with `desc` the events before it. `limit` defaults to 50 and is capped at 1000. |
+| `GET` | `/sessions/:session_id/export` | `events:read` | Returns the exact JSON body the evaluator would receive for this session, served as a downloadable attachment named `session-<id>.json`. Useful for replaying production sessions through `agenteye-evaluator` for offline testing. The bytes are byte-identical to what the evaluator pipeline sends. |
+| `POST` | `/sessions/:session_id/re-evaluate` | `evaluations:trigger` | Enqueue a fresh evaluation for a session; runs whether or not a prior evaluation exists. The new result is **appended** to the session's evaluation timeline rather than overwriting the previous one, so prior scores remain visible as history. Returns `202` on enqueue, `404` for an unknown session, `409` if an evaluation is already in flight. Use this after deploying a new evaluator, or for sessions that never emitted `agent_end`. |
+
+### Filtering by score range: `score_filters`
+
+`GET /evaluations` accepts an optional `score_filters` parameter that
+narrows results by numeric values inside the `scores` object. The
+parameter is a comma-separated list of `key:min..max` entries; either
+bound may be omitted. Multiple entries combine with logical AND. Rows
+where the named key is absent or non-numeric are excluded. A request may
+carry at most 20 filter entries; exceeding that returns HTTP 400.
+
+Examples:
+```text
+# helpfulness in [0.5, 0.8]
+GET /evaluations?score_filters=helpfulness:0.5..0.8
+
+# tool_efficiency at most 0.3 (no lower bound)
+GET /evaluations?score_filters=tool_efficiency:..0.3
+
+# helpfulness >= 0.5 AND factuality >= 0.9
+GET /evaluations?score_filters=helpfulness:0.5..,factuality:0.9..
+```
+
+Each `/evaluations` response object has these fields:
+
+| Field | Type | Notes |
+|---|---|---|
+| `evaluation_id` | string (UUID) | The canonical identifier for this terminal evaluation. Each terminal evaluation gets a new UUID; a single session can hold multiple. |
+| `id` | string (UUID) | Backwards-compatibility alias carrying the same value as `evaluation_id`. |
+| `session_id` | string | The session this evaluation ran against. A session can have multiple evaluations in the timeline. |
+| `agent_id` | string | Identifies the agent that produced the session. |
+| `environment` | string | Environment label copied from the session. |
+| `status` | enum | One of `"done"`, `"error"`, `"timeout"`. |
+| `scores` | object \| null | Scores returned by your evaluator. |
+| `reasoning` | object \| null | Optional per-score justification map returned by your evaluator. Keys typically mirror those in `scores`. The dashboard renders each entry under its score bar. |
+| `summary` | string \| null | Optional one-paragraph overall narrative returned by your evaluator. The dashboard renders this above the per-score breakdown as the evaluation's headline. |
+| `error` | string \| null | Populated on `"error"` / `"timeout"` only. |
+| `attempt_count` | integer | Number of dispatch attempts (≥ 1). |
+| `duration_ms` | integer \| null | Duration of the final attempt. |
+| `completed_at` | string (ISO 8601 UTC) | When the terminal result was recorded. Results are ordered by `completed_at` (newest first). |
+| `created_at` | string (ISO 8601 UTC) | Carries the same timestamp as `completed_at` (write-once semantics). |
+
+---
+
+## Permissions
+
+| Permission | Grants |
+|---|---|
+| `evaluations:read` | List evaluation results, view scores in the dashboard, and load dashboard health metrics. |
+| `evaluations:trigger` | Manually enqueue an evaluation for a session via `POST /sessions/:session_id/re-evaluate` or the dashboard's re-evaluate button. |
+| `dashboards:read` | View saved dashboards (also needs `evaluations:read` to load their metrics). |
+| `dashboards:write` | Create and edit dashboards. |
+| `dashboards:delete` | Delete dashboards. |
+
+The bootstrap admin (`ADMIN_KEY`, `ADMIN_EMAIL`) automatically receives these.
+
+---
+
+## Viewing results
+
+- **`/sessions/<id>`**: events timeline + a right rail showing the session's
+  scores and any error from the dispatch attempt. If your key has
+  `evaluations:trigger`, a **re-evaluate** button appears next to the export
+  button, useful for sessions that never emitted `agent_end`, or for
+  refreshing scores after deploying a new evaluator. The dashboard polls for
+  the new result and updates the right rail when it lands.
+- **`/sessions`**: filterable session grid; the score column shows each
+  session's evaluation status and scores at a glance.
+- **`/dashboards`**: saved eval-health views (see [Dashboards](#dashboards) below).
+
+![The Sessions grid with per-session evaluation status pills and colour-coded score badges (helpfulness, factuality, tool_efficiency, safety, coherence)](/agenteye/images/sessions-list.png)
+
+*The sessions grid shows each run's evaluation status and scores at a glance; red/amber/green badges make low scores jump out.*
+
+---
+
+## Dashboards
+
+The **Dashboards** page (`/dashboards`) lets you save a combination of evaluation
+filters as a named, reusable view and watch how that slice of evaluations is
+doing at a glance. Dashboards are **shared across your whole organization**;
+everyone with `dashboards:read` sees the same set.
+
+Each dashboard pins:
+
+- **Filters**: the same controls as the sessions page: environment, status,
+  agent, a rolling time window, and score-range filters (`key:min..max`).
+- **A display configuration**: which score keys to feature, the green/amber/red
+  health thresholds, which panels to show, and whether to collapse to the latest
+  evaluation per session.
+
+Each card shows the number of matching sessions, a done/error/timeout breakdown,
+the average of each featured score, and a small trend sparkline. Opening a
+dashboard shows the full-size panels; **"open in sessions"** drops you into the
+sessions page pre-filtered to exactly that slice. Metrics are computed
+server-side over the whole matching set (via `GET /evaluations/aggregate`), so
+the numbers are exact rather than sampled.
+
+![An eval-health dashboard with average-score bars per evaluator dimension, a tool ok-vs-error breakdown, top tools, and an events-per-hour trend](/agenteye/images/dashboard-quality.png)
+
+**Permissions:** viewing needs both `dashboards:read` and `evaluations:read`;
+creating and editing needs `dashboards:write`; deleting needs `dashboards:delete`.
+The bootstrap admin receives all of these automatically.
+
+---
+
+## Troubleshooting
+
+**Sessions exist but no evaluations are created.** Confirm `EVALUATOR_ENDPOINT`
+is set on the server process, that the server and evaluator share the same
+`EVALUATOR_TOKEN` value, and that the evaluator's `/health` endpoint is
+reachable from the server. With `EVALUATOR_ENDPOINT` unset the pipeline is a
+no-op.
+
+**In-flight evaluations pile up.** Query `GET /evaluation-jobs` to see the
+in-flight queue. Inspect `attempt_count`, `next_attempt_at`, and `last_error`
+on each row. Common causes: evaluator service unreachable or returning 5xx
+(retried with backoff), wrong `EVALUATOR_TOKEN` (401 is terminal), or an
+async evaluator that returns `pending` indefinitely (see below).
+
+**Sessions completed but no terminal evaluation.** Query
+`GET /evaluation-jobs?status=polling`; the result may still be in flight.
+If a job is stuck in `pending`, the server is having trouble reaching the
+evaluator; check that the evaluator is up and that `EVALUATOR_TOKEN` matches.
+
+**`HTTP 401 from evaluator: invalid bearer token`.** The `EVALUATOR_TOKEN`
+on the server does not match the value the evaluator service is configured
+with. They must be identical.
+
+**Async evaluator returns `pending` forever.** The server polls
+`GET /evaluate/{job_id}` until the evaluator returns `done` or `error`, or
+until `EVALUATOR_MAX_POLL_DURATION_SECS` (default 1 h) elapses. After the cap
+the evaluation is recorded as `timeout` and removed from the in-flight queue.
+Raise `EVALUATOR_MAX_POLL_DURATION_SECS` if your evaluator legitimately needs
+longer than the default.
+
+---
+
+## Next steps
+
+- [Evaluator agent skill](/agenteye/evaluator-skill): have a coding agent design your dimensions against real sessions and build this service for you.
+- [Python SDK](/agenteye/python-sdk): emit the `agent_end` events that trigger scoring.
+- [API keys](/agenteye/api-keys): the `evaluations:read` and `evaluations:trigger` permissions.
+- [Audits](/agenteye/audits): Observability's other automated quality feature, for policy-based review.

package/.next/standalone/docs-old/agenteye/evaluations.mdx

@@ -0,0 +1,51 @@
+---
+title: "Evaluations"
+description: "Quality problems find you now, instead of you hearing about them in a user complaint."
+---
+
+
+Quality problems find you now, instead of you hearing about them in a user complaint. Connect your own scoring service once and Failproof AI Observability grades every finished run automatically, so a drop in helpfulness or a spike in hallucinations shows up on its own, before a customer feels it.
+
+![The Sessions grid with a score column: each run carries an evaluation status pill and colour-coded helpfulness, factuality, and tool-efficiency badges](/agenteye/images/sessions-list.png)
+
+*Every run on the sessions grid carries its scores; red, amber, and green badges make the weak runs jump out without you opening a single transcript.*
+
+## Stop sampling runs by hand
+
+You used to spot-check a handful of runs and hope the rest were fine. Now every completed session is scored the moment it finishes, on the dimensions you care about: helpfulness, tool efficiency, factuality, safety, whatever your quality bar is. You define the score keys; Failproof AI Observability stores, trends, and displays whatever your evaluator sends back. No run slips through unscored, and you stop learning about a regression from a support ticket.
+
+The scores ride along on the sessions grid at **`/<org-slug>/sessions`** (sidebar → *observe* → *sessions*), one badge cluster per row. Want just the runs that fell short? Filter the grid by score range, say helpfulness below 0.5, and pull up exactly the runs worth reading. Viewing scores needs the `evaluations:read` permission.
+
+## See why a run scored low
+
+A number tells you a run was weak; the session page tells you why. Open any run and the right rail leads with the headline summary, then shows a bar per dimension with your evaluator's own reasoning under each one, so you go from "this scored 0.4 on factuality" to the exact claim it got wrong in seconds.
+
+![A session's right rail: the evaluation summary on top, then per-dimension score bars each with a line of reasoning, beside the full event timeline](/agenteye/images/session-detail.png)
+
+*The session detail view: summary, per-dimension score bars, and the reasoning behind each score, right next to the run's event timeline.*
+
+Shipped a sharper evaluator, or looking at a run that crashed before it could be scored? A **re-evaluate** button (gated by `evaluations:trigger`) re-scores the session in place and appends the fresh result to its timeline, so earlier scores stay visible as history. You will find it at **`/<org-slug>/sessions/<session-id>`**.
+
+## Watch quality trend across the fleet
+
+One run scoring low is noise; a whole cohort sliding is a signal. Saved dashboards turn your scores into a trend you can watch at a glance: average helpfulness this week against last, per agent, per environment.
+
+![A quality dashboard: average-score bars per evaluator dimension alongside a trend over time](/agenteye/images/dashboard-quality.png)
+
+*A saved quality dashboard trends the score keys you feature, so a slow drift is obvious long before it becomes an incident.*
+
+Dashboards live at **`/<org-slug>/dashboards`** (sidebar → *analyze* → *dashboards*), are shared across your whole organization, and each card rolls up the matching sessions: how many, the average of each featured score, and a trend sparkline. "Open in sessions" drops you straight into the pre-filtered runs behind any number. Viewing needs `dashboards:read` plus `evaluations:read`.
+
+## Connect an evaluator once
+
+Scoring is opt-in and stays completely off until you point Failproof AI Observability at a scorer. You stand up one small HTTP service (Observability ships a working reference you can copy), set two values on your server, and every run from then on is scored for you. The full walkthrough, the scoring contract, and the SDK live in the deep guide.
+
+Not sure which dimensions are worth scoring in the first place? The [evaluator agent skill](/agenteye/evaluator-skill) has your coding agent work that out against your own sessions, then build and deploy the service.
+
+## Related
+
+- [Evaluation suite](/agenteye/evaluation-suite): connect your evaluator, the scoring contract, and the SDK.
+- [Evaluator agent skill](/agenteye/evaluator-skill): let a coding agent pick your score dimensions and build the evaluator.
+- [Sessions](/agenteye/sessions): the run-by-run grid where scores appear.
+- [Dashboards](/agenteye/dashboards): save and share quality trends across your org.
+- [Audits](/agenteye/audits): Observability's other automatic quality feature, for cross-session investigations.

package/.next/standalone/docs-old/agenteye/evaluator-skill.mdx

@@ -0,0 +1,167 @@
+---
+title: "Failproof AI Observability Evaluator Agent Skill"
+description: "Go from \"I think our agent is sometimes bad\" to a deployed scoring service, with your coding agent doing both the deciding and the building."
+---
+
+
+Go from *"I think our agent is sometimes bad"* to a deployed scoring service, with your coding agent doing both the deciding and the building. The **Failproof AI Observability evaluator skill** (`agenteye-evaluator`) is an *Agent Skill*: a small folder of instructions that a coding agent such as Claude Code or Codex loads on demand. It teaches the agent to work out which quality dimensions are worth tracking for *your* agent, then write, test, and deploy the [evaluator service](/agenteye/evaluation-suite) that scores them.
+
+It is **not** a hosted scorer, a registry you upload to, or a plugin system. Your evaluator stays your own HTTP service on your own infrastructure, exactly as described in the [Evaluation suite](/agenteye/evaluation-suite) guide. The skill only teaches your agent to build it well, so everything it does, you could do yourself by writing the same code.
+
+---
+
+## The hard part is deciding what to score
+
+The SDK surface is small — a decorator and two models — and an agent can write that from the [contract](/agenteye/evaluation-suite#http-contract) alone. That's not where evaluators fail. They fail because they score the wrong thing, and an evaluator that scores the wrong thing is worse than none: it produces a dashboard everyone learns to ignore.
+
+So most of the skill is the part before any code exists. It has the agent interview you (*"describe a run that went well; now one that went badly"*), then pull your real sessions through the [`agenteye` CLI](/agenteye/cli) and read them end to end. Those two halves usually disagree, and the gap is the point: what you intend to measure versus what your transcripts can actually support. A dimension only survives if it is **computable** from the events and **discriminating** — if it scores 0.9 on both your good run and your bad one, it teaches nothing and gets cut.
+
+What comes back is a proposal of 2-4 dimensions with the reasoning attached, for you to sign off on before a line is written.
+
+```mermaid
+flowchart TD
+    YOU["you: 'I want evals for my support bot'"] --> AGENT["coding agent (Claude Code / Codex)<br/>loads the agenteye-evaluator skill"]
+    AGENT -->|"interview: what does good vs bad look like?"| YOU
+    AGENT -->|"agenteye --json sessions / events"| DATA["your real sessions<br/>what actually happens"]
+    DATA --> DIMS["2-4 dimensions, you sign off"]
+    DIMS --> SVC["your evaluator service<br/>agenteye-evaluator SDK"]
+    SVC --> SCORES["scores land in the dashboard<br/>and agenteye evals"]
+```
+
+---
+
+## How it relates to the other evaluation pieces
+
+Four docs cover scoring, and they hand off to each other in order:
+
+| Page | What it is | Reach for it when |
+|---|---|---|
+| **[Evaluations](/agenteye/evaluations)** | The feature: scores on the sessions grid, dashboards, re-evaluate | You want to know what automatic scoring gets you |
+| **[Evaluation suite](/agenteye/evaluation-suite)** | The HTTP contract, the SDK, the server env vars | You're implementing or debugging the evaluator yourself |
+| **Evaluator skill** (this doc) | A natural-language front door on designing *and* building the scorer | You want to go from "I want evals" to a running service |
+| **[CLI skill](/agenteye/cli-skill)** | A natural-language front door on the `agenteye` CLI | You want to *read* the scores you already have |
+| **[Python SDK skill](/agenteye/python-sdk-skill)** | A natural-language front door on instrumenting your agent | Your agent isn't emitting sessions yet — there is nothing to score |
+
+### vs. the CLI skill: build versus read
+
+The two skills are deliberately non-overlapping, and installing both is the normal setup — the agent picks between them based on what you ask:
+
+- **`agenteye-evaluator`** (this doc) builds the thing that *produces* scores. Its job ends when scores land for the first time.
+- **[`agenteye-cli`](/agenteye/cli-skill)** reads scores that already exist (`agenteye evals`). *"Did quality drop this week?"* is its question, not this skill's.
+
+---
+
+## Prerequisites
+
+1. The **`agenteye` CLI installed and logged in** (`pipx install agenteye`, then `agenteye login`). The skill leans on it twice: to pull the real sessions it designs against, and to confirm your scores landed at the end. Your login needs `events:read`, plus `evaluations:read` for that final check. As with the CLI skill, it **cannot** complete the emailed one-time-code login for you.
+2. **Somewhere for the evaluator to live.** It gets built into an image and run as a long-running service, so it needs a real repo, not a scratch file. Evaluators often live in their own repo, separate from the agent being scored — the skill looks for an existing one and asks before scaffolding a new one.
+3. **The `agenteye-evaluator` SDK wheel** — read the next section before your agent starts typing `pip` commands.
+
+---
+
+## Where to get it
+
+The skill is published in Failproof AI's public skills collection:
+
+**[github.com/FailproofAI/skills](https://github.com/FailproofAI/skills)** → [`skills/agenteye-evaluator/`](https://github.com/FailproofAI/skills/tree/main/skills/agenteye-evaluator)
+
+The repository is public and the skill needs no credential of its own — it only drives the `agenteye` CLI with the session *you* logged in with, and writes code in *your* repo. Note it ships as its own folder and is **not** inside the `pipx install agenteye` package, so don't look for it there.
+
+## Installing the skill
+
+The quickest path is the [`skills`](https://skills.sh) CLI, which fetches the folder and drops it where your agent looks:
+
+```bash
+# Claude Code, this project only
+npx skills add FailproofAI/skills --skill agenteye-evaluator -a claude-code
+
+# every project (installs to ~/.claude/skills/)
+npx skills add FailproofAI/skills --skill agenteye-evaluator -a claude-code -g --copy
+
+# Codex instead
+npx skills add FailproofAI/skills --skill agenteye-evaluator -a codex
+```
+
+Then manage it like any other skill:
+
+```bash
+npx skills list -a claude-code           # what's installed
+npx skills update agenteye-evaluator     # pull the latest version
+npx skills remove agenteye-evaluator     # remove it
+```
+
+Prefer to install by hand? An Agent Skill is just a folder containing a `SKILL.md` (plus optional references), so copying it works too:
+
+- **Claude Code**: put the `agenteye-evaluator/` folder in `~/.claude/skills/` (every project) or `<your-repo>/.claude/skills/` (that repo only). Claude Code auto-discovers it — verify with the `/skills` list, or just ask for evals.
+- **Codex (OpenAI)**: Codex reads the same `SKILL.md`. The bundled `agents/openai.yaml` sets `allow_implicit_invocation: true`, so Codex auto-selects the skill when a task matches; otherwise invoke it explicitly as `$agenteye-evaluator`.
+
+---
+
+## The SDK is not on public PyPI
+
+> **Warning:** Read this before letting an agent install the SDK.
+
+The skill is public; the SDK it drives is not. `agenteye-evaluator` ships only as a private release artifact, and unlike `agenteye`, the name is **unclaimed on public PyPI** — so a bare `pip install agenteye-evaluator` could pull a stranger's package into the service that reads your production transcripts. That's a supply-chain problem, not a typo.
+
+The skill knows this and works down an install ladder instead, stopping at the first rung that applies: the monorepo source if you're inside the AgentEye repo, otherwise the private release wheel from GitHub Releases (needs access), and if neither is reachable it **stops and tells you to ask your Failproof AI contact for the wheel** rather than improvising.
+
+So if your agent proposes a bare `pip install agenteye-evaluator` from public PyPI, that is the tell that the skill never loaded. Stop there and check it's installed.
+
+---
+
+## What you can ask it
+
+A real round-trip starts with a vague ask and ends with a signed-off design, not with code:
+
+```text
+you   ▸ I want evals for our support bot. I think it's sometimes bad.
+
+agent ▸ Before I write anything: describe a run that went well,
+        and one that went badly.
+
+you   ▸ Good: it answers from the docs. Bad: it makes stuff up,
+        or gives up and says "contact support".
+
+agent ▸ Running: agenteye --json sessions --status error --since 7d
+        Reading 5 sessions, including two you'd call bad.
+
+        Your words say factuality and deflection. The transcripts
+        agree on deflection (2 of 5 end with a support hand-off,
+        and it's plain in model_response.content), but factuality
+        needs a ground truth your events don't carry.
+
+        So I'd score three things:
+          - deflection      rules, cheap, separates your two runs
+          - tool_efficiency rules, catches the retry loop in run-004
+          - groundedness    LLM judge: is every claim in the docs it retrieved?
+
+        "factuality" would need ground truth we don't have; groundedness
+        is the closest thing your transcripts can actually support.
+        OK with those three names? Renaming later splits the history.
+
+you   ▸ yes
+
+agent ▸ No evaluator in this repo. Should I scaffold one here, or do
+        you have one elsewhere?
+```
+
+From there it writes the rules-based dimensions first (free, instant, deterministic), tests them against a real captured session including the empty and never-finished ones that crash naive evaluators, and only reaches for an LLM judge on the subjective dimension. It knows the [dispatcher's limits](/agenteye/evaluation-suite#configuring-the-server) — a 30s request timeout and 8 concurrent calls deployment-wide — so if the judge won't reliably fit, it goes async with `JobPending` rather than letting your judge get cancelled and retried five times at five times the cost.
+
+Then it deploys, sets the two server env vars, and confirms with `agenteye --json evals --session-id <id>` that scores actually landed. Scores landing is the only proof.
+
+---
+
+## What to watch for
+
+- **Dimension names are close to permanent.** Score keys are arbitrary strings and the platform trends whatever you send, which means nothing downstream corrects a bad choice. Rename later and the history splits: old sessions keep the old key and the trend breaks. This is why the skill gets explicit sign-off before writing code — take that prompt seriously.
+- **Fixtures are real production transcripts.** Designing against real sessions means pulling them to disk, and they can contain customer data. The skill asks before committing them to git; if in doubt, keep `fixtures/` out of the repo and have each developer pull their own.
+- **The agent writes and deploys a service that reads every transcript.** It acts as you, bounded by your CLI login's permissions, but review the evaluator like any other code that touches production data.
+
+---
+
+## Next steps
+
+- **[Evaluation suite](/agenteye/evaluation-suite)**: the HTTP contract, the SDK, and the server env vars the skill configures.
+- **[Evaluations](/agenteye/evaluations)**: where the scores show up once they land.
+- **[CLI skill](/agenteye/cli-skill)**: the sibling skill, for reading results rather than building the scorer.
+- **[CLI](/agenteye/cli)**: the command reference behind the session data the skill designs against.