failproofai 1.0.1-beta.1 → 1.0.1

package/.next/standalone/docs-old/agenteye/cli-recipes.mdx

@@ -0,0 +1,179 @@
+---
+title: "CLI recipes for agents"
+description: "Copy-paste query patterns and jq recipes that turn session, event, and evaluation data into something a script or coding agent can automate."
+---
+
+
+Pull session, event, and evaluation data (and trigger re-evaluations) straight from a script or coding agent, with clean JSON on stdout that pipes directly into `jq`. These recipes turn Failproof AI Observability's data into something a terminal user or an AI coding agent (Claude Code, Cursor) can query and automate, without clicking through the dashboard.
+
+The patterns below are copy-paste ready for the Failproof AI Observability CLI (`agenteye`). For installation, authentication, and the full option list see [CLI](/agenteye/cli); run `agenteye -h` or `agenteye <command> -h` for the built-in help.
+
+## Golden rules
+
+1. **Global options go *before* the command.** `agenteye --json sessions` is correct; `agenteye sessions --json` is not. The globals are `--json`, `--base-url`, `--org`, `--token`, `--insecure`/`--secure`, `--timeout`, `--quiet`, `--no-color`.
+2. **Pass `--json` whenever you parse output.** Data goes to **stdout** as JSON; human status and errors go to **stderr**, so stdout stays clean to pipe into `jq`.
+3. **Branch on the exit code**, not on stderr text: `0` ok · `1` unexpected error · `2` bad arguments · `3` cannot reach the dashboard · `4` not logged in or expired · `5` missing permission · `6` resource not found.
+4. **Discover with `-h`.** Every command documents its filters, value formats, and JSON shape.
+
+## One-time setup
+
+```bash
+export AGENTEYE_DASHBOARD_URL=https://agenteye.example.com   # so you don't repeat --base-url
+agenteye login --email you@example.com                       # paste the emailed code; valid ~24h
+```
+
+## Confirm auth before doing work
+
+`whoami` never errors on a missing or expired session; it reports `logged_in:false` instead, so an agent can probe auth state safely. (It can still exit non-zero if no base URL is set or the dashboard is unreachable.)
+
+```bash
+if [ "$(agenteye --json whoami | jq -r .logged_in)" != "true" ]; then
+  echo "Not authenticated. Run: agenteye login" >&2; exit 1
+fi
+```
+
+## Find failing or low-scoring sessions
+
+```bash
+# sessions in the last 24h whose evaluation errored
+agenteye --json sessions --since 24h --status error | jq -r '.sessions[].session_id'
+
+# evaluations scoring <= 0.5 on helpfulness, for one agent
+agenteye --json evals --agent-id checkout-bot --score helpfulness:..0.5 \
+  | jq '.evaluations[] | {session_id, scores}'
+```
+
+Score filtering lives on **`evals`**, not `sessions`. `--score KEY:MIN..MAX` is repeatable and AND-combined; either bound is optional (`..0.5` means ≤ 0.5, `0.9..` means ≥ 0.9). You can pass up to 20 score filters per request; more returns HTTP 400. `sessions` shares the `--env`, `--status`, `--agent-id`, `--session-id`, and time-range filters with `evals`, but has no `--score`.
+
+## Read one session end-to-end
+
+There is no single `session show` command. Combine the event trail with the session's evaluation:
+
+```bash
+# the session's latest evaluation (status + scores)
+agenteye --json evals --session-id run-001 | jq '.evaluations[0] | {status, scores}'
+
+# every event in the run (raise --limit for a full sweep)
+agenteye --json events --session-id run-001 --all --limit 1000 | jq '.events[] | {ts, event_type}'
+
+# just the tool calls in a session (--full is required to get the raw payload)
+agenteye --json events --full --session-id run-001 --event-type tool_use,tool_result --all \
+  | jq '.events[].payload'
+```
+
+> **Note:** By default, `events` reads a fast, payload-free feed. Each event carries a server-computed one-line `summary` plus flags like `is_error` and token counts, but `payload` comes back as `{}`. To pull the raw payload, add `--full` (or `--fields payload`). The full feed is slower at scale, so keep it bounded: pair `--full` with a single `--session-id`.
+
+## Fetch everything (pagination)
+
+Results are newest-first and cursor-paginated.
+
+```bash
+# one shot: fetch up to 500 rows in 200-row pages
+agenteye --json events --session-id run-001 --limit 500 --all > events.json
+
+# manual paging: feed next_cursor back in
+page=$(agenteye --json events --limit 100)
+cursor=$(echo "$page" | jq -r '.next_cursor // empty')
+[ -n "$cursor" ] && agenteye --json events --limit 100 --cursor "$cursor"
+```
+
+## Slim the output with --fields
+
+Restrict the keys (in both the table and `--json`) to reduce what an agent must read.
+
+```bash
+agenteye --json sessions --since 7d --fields session_id,status,scores | jq -c '.sessions[]'
+agenteye --json events --session-id run-001 --fields ts,event_type --all
+```
+
+Unknown field names are rejected (exit `2`) with the valid list, a cheap way to discover field names.
+
+## Discover valid filter values
+
+```bash
+agenteye --json list envs | jq -r '.values[]'           # values for --env
+agenteye --json list tools | jq -r '.values[]'          # tool names; also agents, models, event_types, …
+agenteye --json list score_filters | jq -r '.values[]'  # valid KEY for --score KEY:MIN..MAX
+```
+
+## Pick your org (multi-tenant)
+
+If you belong to more than one org, choose the active tenant at login (it's saved):
+
+```bash
+agenteye login --org acme --email you@corp.com   # set the tenant in the same step as login
+agenteye --json orgs list | jq -r '.orgs[].org_slug'
+agenteye --org globex --json sessions --since 24h   # override for one command
+```
+
+A multi-org login without `--org` exits non-zero and prints the orgs to choose from.
+
+## Provision an API key for the SDK/collector
+
+```bash
+# the secret is printed ONCE, with --json it's the .key field
+key=$(agenteye --json keys create ci-bot --add events:read.add | jq -r '.key')
+agenteye keys regenerate ci-bot --yes    # rotate; agenteye keys disable ci-bot --yes to revoke
+```
+
+## Run a saved or ad-hoc query
+
+```bash
+agenteye --json query run --sql "select count(*) from analytics.events" | jq '.rows'
+agenteye --json query run errs --arg prod | jq '.rows'   # a saved query + a positional $1
+```
+
+## Triage an incident non-interactively
+
+```bash
+id=$(agenteye --json incidents list --state firing | jq -r '.incidents[0].id')
+agenteye incidents ack "$id"
+agenteye incidents assign "$id" --assignee you@corp.com
+agenteye incidents resolve "$id" --yes
+```
+
+> **Note:** Mutations auto-skip their confirmation prompt under `--json` or when stdin isn't a TTY, so agents never hang; pass `--yes`/`-y` to skip it explicitly elsewhere.
+
+## Exit-code handling in a script
+
+```bash
+out=$(agenteye --json sessions --since 1h) || code=$?
+case "${code:-0}" in
+  0) echo "$out" | jq '.sessions | length' ;;
+  4) echo "Session expired - run 'agenteye login'." >&2 ;;
+  5) echo "Missing permission (ask an admin for evaluations:read)." >&2 ;;
+  3) echo "Dashboard unreachable - check the URL." >&2 ;;
+  *) echo "Unexpected error (exit ${code})." >&2 ;;
+esac
+```
+
+## JSON output shapes
+
+| Command | stdout JSON (with `--json`) |
+|---|---|
+| `whoami` | `{"logged_in": true, "id", "email", "is_instance_admin", "active_org", "permissions": [...], "memberships": [...]}` or `{"logged_in": false}` |
+| `orgs list` | `{"active_org", "orgs": [{"org_slug","org_name","permission_set","permissions"}]}` |
+| `events` | `{"events": [...], "next_cursor": <cursor or null>}` |
+| `evals` | `{"evaluations": [...], "next_cursor": <cursor or null>}` |
+| `sessions` | `{"sessions": [...], "next_cursor": <cursor or null>}` |
+| `errors` | `{"errors": [...], "next_cursor": <cursor or null>}` |
+| `list <kind>` | `{"kind", "values": [...]}` |
+| `keys list` / `keys create` | `{"keys": [...]}` / `{id, name, permissions, created_at, key}` (`key` shown once) |
+| `query run` | `{columns: [{name,type}], rows: [[...]], truncated, elapsed_ms}` |
+| `users list` / `settings list` | `{"users": [...]}` / `{"settings": [...]}` |
+| `alerts list` / `incidents list` | `{"alerts": [...]}` / `{"incidents": [...]}` |
+| create/update/delete (any) | the resource object, or `{"deleted": true, "id"}` for deletes |
+| failure (any, with `--json`) | `{"error": "...", "exit_code": <n>, "status"?: <http>, "hint"?: "..."}` on stdout |
+
+- Each **event** item (`events`): `id, session_id, agent_id, event_type, ts, payload, environment, summary, is_error, error_type, output_tokens, context_window, context_fill`. Note that `payload` is `{}` unless you request the full feed with `--full` (or `--fields payload`).
+- Each **evaluation** item (`evals`): `id, session_id, agent_id, environment, status, scores, reasoning, summary, error, attempt_count, duration_ms, completed_at, created_at`.
+- Each **session** item (`sessions`): `session_id, agent_id, environment, status, scores, event_count, started_at, last_event_at, first_event_id, last_event_id, latest_evaluation`.
+
+Each command's `--fields` accepts exactly its own item's field names. The set differs between `sessions` and `evals`, so a name valid for one may be rejected by the other.
+
+## Next steps
+
+- [CLI](/agenteye/cli): installation, authentication, and the full option reference for every command.
+- [CLI agent skill](/agenteye/cli-skill): package these recipes as a skill your coding agent can load.
+- [API keys](/agenteye/api-keys): create and scope the keys the CLI, SDK, and collector authenticate with.
+- [Python SDK](/agenteye/python-sdk): send events into Failproof AI Observability so there is data for these recipes to query.

package/.next/standalone/docs-old/agenteye/cli-skill.mdx

@@ -0,0 +1,159 @@
+---
+title: "Failproof AI Observability CLI Agent Skill"
+description: "Ask your coding agent \"is anything broken today?\" and let it answer from your live Failproof AI Observability data, with no commands to memorize."
+---
+
+
+Ask your coding agent *"is anything broken today?"* and let it answer from your live Failproof AI Observability data, with no commands to memorize. The **Failproof AI Observability CLI skill** (`agenteye-cli`) is an *Agent Skill*: a small folder of instructions that a coding agent such as Claude Code or Codex loads on demand. It teaches the agent to operate your Observability deployment through the [`agenteye` CLI](/agenteye/cli) from plain-English requests like *"give CI a key that can only push events"* or *"ack the firing incident and assign it to me."*
+
+It is **not** a service or a separate binary; there is nothing to deploy. It rides on top of the CLI you have already installed: the agent shells out to `agenteye --json …`, parses the clean JSON, and answers you in prose. Everything it can do, you could do yourself by typing the same commands.
+
+---
+
+## How it relates to the other Failproof AI Observability interfaces
+
+Failproof AI Observability gives you four ways to reach the same data and controls. They complement each other:
+
+| Interface | What it is | Where it runs | Reach for it when |
+|---|---|---|---|
+| **[CLI](/agenteye/cli)** | The command/flag reference for `agenteye` | Your terminal | You want to run or script a specific command |
+| **[CLI recipes](/agenteye/cli-recipes)** | Copy-paste `jq`/pipeline patterns | Your terminal / scripts | You're wiring the CLI into automation |
+| **CLI skill** (this doc) | A natural-language front door on the CLI | Your coding agent, on your workstation | You want to *just ask* and let the agent pick the command |
+| **[Evaluator skill](/agenteye/evaluator-skill)** | A sibling skill that designs and builds your scoring service | Your coding agent, on your workstation | You want to *produce* eval scores rather than read them |
+| **[Python SDK skill](/agenteye/python-sdk-skill)** | A sibling skill that instruments your agent so it emits telemetry at all | Your coding agent, on your workstation | You want your agent to *produce* the events this skill reads |
+| **[In-dashboard AI assistant](/agenteye/assistant)** | A chat embedded in the dashboard | Server-side (in the dashboard) | You want in-dashboard Q&A over your data |
+
+The skill itself has no privileges of its own; it just turns your words into CLI calls that run as you:
+
+```mermaid
+flowchart TD
+    YOU["you: 'ack the firing incident'"] --> AGENT["coding agent (Claude Code / Codex)<br/>loads the agenteye-cli skill"]
+    AGENT --> CLI["agenteye --json incidents ack ..."]
+    CLI -->|your authenticated CLI session| API["Observability dashboard API"]
+```
+
+### vs. the in-dashboard AI assistant: an important distinction
+
+These are two different tools with very different blast radii:
+
+- The **in-dashboard AI assistant** ([AI assistant](/agenteye/assistant)) is a chat embedded in the dashboard, backed by the agent service. It is **read-only plus approval-gated authoring**: it can draft saved queries and dashboards, but every write pauses for your explicit click-approval, and it never deletes. It is gated by the `agent:use` permission and only ever sees data for the org you're viewing.
+- The **CLI skill** runs on *your* workstation inside *your* coding agent and drives the `agenteye` CLI as **you**. It can perform the CLI's **full surface, including mutations** (create/rotate/disable API keys, change org settings, resolve incidents, delete saved queries), bounded only by the permissions of your CLI login. Treat it exactly as carefully as you would treat running those commands by hand.
+
+---
+
+## Prerequisites
+
+1. The **`agenteye` CLI installed** and on `PATH` (see the [CLI](/agenteye/cli) reference: `pipx install agenteye`).
+2. Your **dashboard URL** set (`AGENTEYE_DASHBOARD_URL`, or the agent passes `--base-url`).
+3. A **logged-in session**: run `agenteye login` yourself first. The skill **cannot** complete the emailed one-time-code login for you; it will tell you to run `agenteye login` if the session is missing or expired (CLI exit code `4`).
+
+---
+
+## Where to get it
+
+The skill is published in Failproof AI's public skills collection:
+
+**[github.com/FailproofAI/skills](https://github.com/FailproofAI/skills)** → [`skills/agenteye-cli/`](https://github.com/FailproofAI/skills/tree/main/skills/agenteye-cli)
+
+Nothing about it is gated — the repository is public and the skill needs no credential of its own, because it only drives the **public** `agenteye` CLI against *your* dashboard, using the session *you* logged in with. You do not need to ask anyone for it.
+
+Note it ships as its own folder and is **not** inside the `pipx install agenteye` package, so don't look for it there.
+
+## Installing the skill
+
+The quickest path is the [`skills`](https://skills.sh) CLI, which fetches the folder and drops it where your agent looks:
+
+```bash
+# Claude Code, this project only
+npx skills add FailproofAI/skills --skill agenteye-cli -a claude-code
+
+# every project (installs to ~/.claude/skills/)
+npx skills add FailproofAI/skills --skill agenteye-cli -a claude-code -g --copy
+
+# Codex instead
+npx skills add FailproofAI/skills --skill agenteye-cli -a codex
+```
+
+Then manage it like any other skill:
+
+```bash
+npx skills list -a claude-code      # what's installed
+npx skills update agenteye-cli      # pull the latest version
+npx skills remove agenteye-cli      # remove it
+```
+
+Prefer to install by hand? An Agent Skill is just a folder containing a `SKILL.md` (plus optional references), so copying it works too:
+
+- **Claude Code**: put the `agenteye-cli/` folder in `~/.claude/skills/` (every project) or `<your-repo>/.claude/skills/` (that repo only). Claude Code auto-discovers it — verify with the `/skills` list, or simply ask a question that matches its description.
+- **Codex (OpenAI)**: Codex reads the same `SKILL.md`. The bundled `agents/openai.yaml` sets `allow_implicit_invocation: true`, so Codex auto-selects the skill when a task matches; otherwise invoke it explicitly as `$agenteye-cli`.
+
+---
+
+## Safety: mutations do NOT prompt when an agent runs the CLI
+
+> **Warning:** Read this before letting an agent make changes.
+
+The `agenteye` CLI normally asks *"are you sure?"* before a destructive action. It **auto-skips that confirmation whenever it is not attached to a terminal (which is exactly how a coding agent runs it), and `--json` skips it too.** So the safety prompt will **not** fire for the agent.
+
+The skill is written to compensate: it is instructed to state the exact command it will run and get your explicit **OK before any state change**. Keep that discipline. When you drive Failproof AI Observability through an agent, *you* are the confirmation step. The state-changing commands to watch for:
+
+- `keys create` / `update` / `disable` / `regenerate`
+- `users create` / `update` / `disable` / `enable`
+- `settings set`
+- `alerts create` / `update` / `delete` / `test`
+- the writing `incidents` subcommands: `ack` / `assign` / `resolve` / `open` / `comment-add` / `comment-delete` / `subscribe` / `unsubscribe`
+- `query create` / `update` / `delete`
+- `agent rename` / `delete`
+- `orgs switch`
+
+Everything under **Observe** (`events`, `sessions`, `evals`, `errors`, `list`, `whoami`, `orgs list/current/perms`) is read-only and changes nothing.
+
+Because the agent acts as **you**, it can only do what your login is permitted to do; permissions are resolved **per org** (see [API keys](/agenteye/api-keys)). A command you lack permission for returns exit code `5` with the exact permission named, so the agent can tell you precisely what to ask an admin for rather than failing opaquely.
+
+---
+
+## What you can ask it
+
+Here's what a real round-trip looks like, a read, then a change that waits for your OK:
+
+```text
+you   ▸ Is anything broken in the last day?
+
+agent ▸ Running: agenteye --json errors --since 24h --aggregate
+        3 error types in the last 24h. The biggest is TimeoutError
+        (18 hits, all on the checkout-agent), then RateLimited (5).
+        Want the failing sessions for TimeoutError?
+
+you   ▸ Give CI a key that can only push events.
+
+agent ▸ This creates an API key. I'll run:
+          agenteye keys create ci --add events:add
+        OK to proceed?
+
+you   ▸ yes
+
+agent ▸ Done. Key "ci" created with events:add only.
+        The secret is shown only once, so store it now. I can't reprint it.
+```
+
+The skill maps each plain-English intent to the right `agenteye` command, discovering valid values first (`list <kind>`, `whoami`) so it doesn't guess, and stating the exact command before any change. More examples:
+
+- *"Is anything broken / failing in the last 24 hours?"* → `errors --since 24h --aggregate`, then a breakdown.
+- *"Why did session `run-001` fail?"* → `events --session-id run-001 --all` + `evals --session-id run-001`.
+- *"How is quality trending this week?"* → `evals --aggregate --since 7d`, then drill into low-scoring runs.
+- *"Give CI a key that can only push events."* → `keys create ci --add events:add` (it states the command, then creates it and captures the one-time secret).
+- *"Who has access? Make Dana read-only."* → `users list` → `users update dana@… --permission-set read-only` (after confirming with you).
+- *"Ack the firing incident and assign it to me."* → `incidents list --state firing` → `incidents ack <id>` / `incidents assign <id> you@…`.
+
+For the exact commands, flags, and JSON shapes behind these, see the [CLI](/agenteye/cli) reference and [CLI recipes for agents](/agenteye/cli-recipes).
+
+---
+
+## Next steps
+
+- **[CLI](/agenteye/cli)**: full command and flag reference for `agenteye`.
+- **[CLI recipes for agents](/agenteye/cli-recipes)**: copy-paste `jq` patterns and exit-code handling.
+- **[Evaluator agent skill](/agenteye/evaluator-skill)**: the sibling skill, for building the evaluator whose scores `agenteye evals` reads.
+- **[Python SDK agent skill](/agenteye/python-sdk-skill)**: the sibling skill, for instrumenting an agent so it emits the telemetry `agenteye` reads.
+- **[AI assistant](/agenteye/assistant)**: the in-dashboard assistant (not to be confused with this terminal skill).
+- **[API keys](/agenteye/api-keys)**: the per-org permission model that bounds what the skill can do.