google-agents-cli 1.3.1__tar.gz → 1.4.0__tar.gz

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-agents-cli
-Version: 1.3.1
+Version: 1.4.0
 Summary: Agents CLI — Agent Development Lifecycle toolchain for ADK
 Author-email: Google LLC <agents-cli-dev-pypi@google.com>
 Requires-Python: >=3.11
@@ -112,8 +112,7 @@ See the [full tutorial](https://google.github.io/agents-cli/guide/quickstart-tut
 |---------|-------------|
 | `agents-cli setup` | Install CLI + skills to coding agents |
 | `agents-cli scaffold <name>` | Create a new agent project |
-| `agents-cli eval generate` | Run agent on eval dataset, produce traces |
-| `agents-cli eval grade` | Run agent evaluations on the traces |
+| `agents-cli eval run` | Run the agent over the eval dataset and grade the traces |
 | `agents-cli deploy` | Deploy to Google Cloud |
 | `agents-cli publish gemini-enterprise` | Register with Gemini Enterprise |
 
@@ -133,6 +132,7 @@ See the [full tutorial](https://google.github.io/agents-cli/guide/quickstart-tut
 | `agents-cli install` | Install project dependencies |
 | `agents-cli lint` | Run code quality checks (Ruff) |
 | **Evaluate** | |
+| `agents-cli eval run` | Run agent inference and grade the traces |
 | `agents-cli eval generate` | Run agent inference over eval cases |
 | `agents-cli eval grade` | Grade generated traces against metrics |
 | `agents-cli eval dataset synthesize` | Synthesize multi-turn eval scenarios for your local agent |

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/README.md

@@ -77,8 +77,7 @@ See the [full tutorial](https://google.github.io/agents-cli/guide/quickstart-tut
 |---------|-------------|
 | `agents-cli setup` | Install CLI + skills to coding agents |
 | `agents-cli scaffold <name>` | Create a new agent project |
-| `agents-cli eval generate` | Run agent on eval dataset, produce traces |
-| `agents-cli eval grade` | Run agent evaluations on the traces |
+| `agents-cli eval run` | Run the agent over the eval dataset and grade the traces |
 | `agents-cli deploy` | Deploy to Google Cloud |
 | `agents-cli publish gemini-enterprise` | Register with Gemini Enterprise |
 
@@ -98,6 +97,7 @@ See the [full tutorial](https://google.github.io/agents-cli/guide/quickstart-tut
 | `agents-cli install` | Install project dependencies |
 | `agents-cli lint` | Run code quality checks (Ruff) |
 | **Evaluate** | |
+| `agents-cli eval run` | Run agent inference and grade the traces |
 | `agents-cli eval generate` | Run agent inference over eval cases |
 | `agents-cli eval grade` | Grade generated traces against metrics |
 | `agents-cli eval dataset synthesize` | Synthesize multi-turn eval scenarios for your local agent |

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "google-agents-cli"
-version = "1.3.1"
+version = "1.4.0"
 description = "Agents CLI — Agent Development Lifecycle toolchain for ADK"
 readme = "README.md"
 license = "Apache-2.0"

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-adk-code/SKILL.md

@@ -12,7 +12,7 @@ description: >
 metadata:
   author: Google
   license: Apache-2.0
-  version: 1.3.1
+  version: 1.4.0
   requires:
     bins:
       - agents-cli

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-adk-code/references/adk-python.md

@@ -470,11 +470,11 @@ Pass `auth_scheme` + `auth_credential` from the auth helpers above for authentic
 
 ### MCP Tools
 
-Connect to MCP servers to use external tools. Use `StdioConnectionParams` for local dev, `SseConnectionParams` for production.
+Connect to MCP servers to use external tools (needs the `mcp` extra: scaffolded projects ship `google-adk[gcp,otel-gcp]`, so add `mcp` and re-sync). Use `StdioConnectionParams` for local dev, `StreamableHTTPConnectionParams` for remote HTTP servers.
 
 ```python
 from google.adk.tools.mcp_tool import McpToolset
-from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams, SseConnectionParams
+from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams, StreamableHTTPConnectionParams
 from mcp import StdioServerParameters
 
 # Local MCP server via stdio
@@ -494,9 +494,17 @@ agent = Agent(
     ...
 )
 
-# Remote MCP server via SSE (production)
+# Remote MCP server, e.g. Cloud Run with --no-allow-unauthenticated. ID tokens
+# expire in ~1h, so mint per call via `header_provider` (ADK calls it on every
+# tool call); a static `headers` dict goes stale. Audience = root, not /mcp.
+from google.auth.transport.requests import Request
+from google.oauth2.id_token import fetch_id_token
+
 McpToolset(
-    connection_params=SseConnectionParams(url="https://mcp.example.com/sse"),
+    connection_params=StreamableHTTPConnectionParams(url=f"{MCP_SERVER_URL}/mcp"),
+    header_provider=lambda ctx: {
+        "Authorization": f"Bearer {fetch_id_token(Request(), MCP_SERVER_URL)}"
+    },
 )
 ```
 
@@ -648,7 +656,7 @@ app = App(
 
 ### Context Compaction
 
-Prevent context overflow on long sessions by summarizing older events in a sliding window:
+Prevent context overflow on long sessions by compacting older events into summaries. Use **token-based** compaction: it triggers on actual prompt-token volume, so it handles unpredictable inputs (pasted code, large tool results) better than a fixed turn count.
 
 ```python
 from google.adk.apps import App
@@ -660,8 +668,8 @@ app = App(
     name="my_app",
     root_agent=root_agent,
     events_compaction_config=EventsCompactionConfig(
-        compaction_interval=20,  # summarize every 20 events
-        overlap_size=3,          # include last 3 events in next window for continuity
+        token_threshold=32000,   # compact once prompt tokens reach this
+        event_retention_size=5,  # keep the last 5 raw events un-compacted
         # Optional: custom summarizer model
         summarizer=LlmEventSummarizer(llm=Gemini(model="gemini-3.6-flash")),
     ),

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-deploy/SKILL.md

@@ -4,7 +4,7 @@ description: >
   This skill should be used when the user wants to "deploy an agent",
   "deploy my ADK agent", "set up CI/CD", "configure secrets",
   "troubleshoot a deployment", or needs guidance on Agent Runtime,
-  Cloud Run, or GKE deployment targets, or awareness of Agent Gateway.
+  Cloud Run, or GKE deployment targets, or binding an agent to an Agent Gateway.
   Covers deployment workflows, service accounts, rollback, and production infrastructure.
   Part of the Google ADK (Agent Development Kit) skills suite.
   Do NOT use for API code patterns (use google-agents-cli-adk-code), evaluation
@@ -12,7 +12,7 @@ description: >
 metadata:
   author: Google
   license: Apache-2.0
-  version: 1.3.1
+  version: 1.4.0
   requires:
     bins:
       - agents-cli
@@ -74,7 +74,7 @@ All three targets are container-based, so any language works.
 **Task tracking:** Deployment involves multiple sequential steps (infra setup, CI/CD configuration, deploy, verification). Use a task list to track progress through these steps — skipping one often causes failures in later steps that are hard to trace back.
 
 1. If prototype (no deployment target), first enhance: `agents-cli scaffold enhance . --deployment-target <target>`
-2. **Notify the human**: "Eval scores meet thresholds and tests pass. Ready to deploy to dev?"
+2. **Notify the human**: paste the eval scores and test results, then ask "Ready to deploy to dev?"
 3. **Wait for explicit approval**
 4. Once approved: `agents-cli deploy`
 
@@ -110,6 +110,8 @@ agents-cli infra single-project
 | `--dns-peering-domain` | DNS peering domain suffix, e.g. `my-internal.corp.` (requires `--network-attachment`) | Agent Runtime |
 | `--dns-peering-project` | Project ID hosting the Cloud DNS managed zone for DNS peering (requires `--network-attachment`) | Agent Runtime |
 | `--dns-peering-network` | VPC network name in the target project for DNS peering (requires `--network-attachment`) | Agent Runtime |
+| `--agent-gateway-egress` | Bind the agent to an [Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/agent-gateway-overview) governing outbound traffic. Full resource name of a gateway with `governedAccessPath=AGENT_TO_ANYWHERE`. Empty value unbinds; omit to leave unchanged. See [Agent Gateway](#agent-gateway) | Agent Runtime |
+| `--agent-gateway-ingress` | Bind the agent to an Agent Gateway governing inbound traffic. Full resource name of a gateway with `governedAccessPath=CLIENT_TO_AGENT`. Empty value unbinds; omit to leave unchanged | Agent Runtime |
 | `--memory` | Memory limit (default: `4Gi`) | Agent Runtime, Cloud Run |
 | `--cpu` | CPU limit (default: `1`) | Agent Runtime, Cloud Run |
 | `--min-instances` | Minimum number of instances (default: `1`) | Agent Runtime, Cloud Run |
@@ -172,7 +174,7 @@ For event-driven / ambient agent deployment on Cloud Run, see the [`ambient-expe
 
 Agent Runtime is a managed Vertex AI service for deploying Python ADK agents. Uses container-based deployment: `agents-cli deploy` packages your project and Agent Engine builds the image from your project's `Dockerfile` (required) — the same `fast_api_app:app` image that serves Cloud Run and GKE.
 
-> **No `gcloud` CLI exists for Agent Runtime.** Deploy via `agents-cli deploy`. Query via the Python `vertexai.Client` SDK.
+> **No `gcloud` CLI exists for Agent Runtime.** Deploy via `agents-cli deploy`. Query via the Python `agentplatform.Client` SDK.
 
 Deployments can take 5-10 minutes. Use `--no-wait` to start a deployment and return immediately, then check on it later with `--status`:
 
@@ -360,22 +362,49 @@ For registering deployed agents with Gemini Enterprise, see `/google-agents-cli-
 
 ---
 
-## Agent Gateway & Semantic Governance (Gemini Enterprise Agent Platform)
+## Agent Gateway
 
-> **Note:** There are no `agents-cli` commands for these yet. Deploy your agent as usual
-> with `agents-cli deploy`, then configure Agent Gateway and Semantic Governance separately
-> (via Terraform or the Cloud Console).
+> **Note:** There are no `agents-cli` commands for creating or managing Agent Gateways or
+> Semantic Governance policies — set those up separately (via Terraform or the Cloud Console).
+> `agents-cli deploy` is the only gateway-aware command, and its support is a passthrough: it
+> binds the agent to an *existing* gateway as part of the Agent Runtime create/update call.
 
 **Agent Gateway** is the networking + security entry/exit point for all agent interactions
 (user↔agent, agent↔tool, agent↔agent) — it centralizes access control and governed
-connectivity (ingress/egress). It is not a deployment target. Deploy your agent with
-`agents-cli deploy`, then attach it to a gateway one of two ways:
+connectivity (ingress/egress). It is not a deployment target.
 
-- **Add the existing agent to a gateway** via the Console / docs flow — see [Route Agent Runtime traffic through Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/agent-gateway-runtime-deploy) ("For existing agents").
-- **Manage the gateway in Terraform** with [`google_network_services_agent_gateway`](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/network_services_agent_gateway) (currently in the `google-beta` provider) — recommended for production, so `agents-cli deploy` owns the agent version and CI/CD handles updates.
+To set up a gateway, follow [Set up an Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/set-up-agent-gateway)
+and [Route Agent Runtime traffic through Agent Gateway](https://docs.cloud.google.com/gemini-enterprise-agent-platform/scale/runtime/agent-gateway-runtime-deploy). You may also manage the gateway in Terraform with [`google_network_services_agent_gateway`](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/network_services_agent_gateway) (currently in the `google-beta` provider).
+
+Once a gateway exists, `agents-cli deploy` binds an agent to it with `--agent-gateway-egress`
+and/or `--agent-gateway-ingress`, each taking a full resource name
+(`projects/PROJECT/locations/REGION/agentGateways/GATEWAY`). Only Agent Runtime deployment is
+supported, and the agent must have Agent Identity (the `--agent-identity` flag), which can only
+be set when the agent is created.
+
+An egress gateway performs TLS decryption and inspection on outbound agent communications,
+so the image must trust the gateway's root CA. That setup is opt-in at scaffold time. If you're
+creating a new project, pass `--agent-gateway` flag to `agents-cli create`:
+
+```bash
+agents-cli create my-agent -d agent_runtime --agent-gateway
+```
+
+Alternatively, you can pass the same flag to `agents-cli scaffold enhance` to upgrade an
+existing project:
+```bash
+agents-cli scaffold enhance . --agent-gateway
+```
+
+Either writes a Dockerfile that consumes the `AGENT_GATEWAY_ROOT_CERTIFICATES` build arg the
+platform injects, and records `agent_gateway: true` under `create_params` so `scaffold upgrade`
+keeps it. Deploying with `--agent-gateway-egress` against a Dockerfile that lacks the build arg
+fails with a pointer to the command above; `--no-agent-gateway` removes the setup again.
 
 Background: [Agent Gateway overview](https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/gateways/agent-gateway-overview).
 
+## Semantic Governance
+
 **Semantic Governance Policies (SGP)** add a natural-language security/compliance layer
 that keeps an agent's tool invocations aligned with user intent and organizational
 constraints.

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-deploy/references/agent-runtime.md

@@ -18,7 +18,7 @@ reasoning_engine adapter still constructs an `AdkApp` internally to dispatch the
 native `:streamQuery`/`:query` contract).
 
 The container serves the ADK HTTP surface (`/run_sse`, `/apps/...`), the A2A
-routes under `/a2a/{app_name}` (JSON-RPC + agent card), `/feedback`, and the
+routes under `/a2a/{app_name}` (JSON-RPC + agent card), and the
 reasoning_engine adapter routes `/api/reasoning_engine` +
 `/api/stream_reasoning_engine` (the native `:streamQuery`/`:query` contract used
 by the Console Playground and Gemini Enterprise ADK registration). It deploys as
@@ -106,9 +106,9 @@ agents-cli run --url https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJ
 To query Agent Runtime programmatically:
 
 ```python
-import vertexai
+import agentplatform
 
-client = vertexai.Client(location="us-east1")
+client = agentplatform.Client(location="us-east1")
 agent = client.agent_engines.get(name="projects/PROJECT/locations/LOCATION/reasoningEngines/ENGINE_ID")
 
 async for event in agent.async_stream_query(message="Hello!", user_id="test"):

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-deploy/references/cloud-run.md

@@ -16,7 +16,7 @@ Scaffolded projects include a `Dockerfile` using single-stage build with `uv` fo
 
 ## FastAPI Endpoints
 
-Scaffolded apps serve the ADK HTTP surface (`/run_sse`, `/apps/...`), A2A routes under `/a2a/{app_name}` (JSON-RPC + agent card — A2A is built into every ADK agent), and `/feedback`. Exact routes vary by template; check `app/fast_api_app.py`.
+Scaffolded apps serve the ADK HTTP surface (`/run_sse`, `/apps/...`), A2A routes under `/a2a/{app_name}` (JSON-RPC + agent card — A2A is built into every ADK agent). Exact routes vary by template; check `app/fast_api_app.py`.
 
 ## Session Types
 

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-deploy/references/gke.md

@@ -50,7 +50,7 @@ Cloud SQL in GKE uses a **proxy sidecar container** in the pod (unlike Cloud Run
 
 ## FastAPI Endpoints
 
-Scaffolded apps serve the ADK HTTP surface (`/run_sse`, `/apps/...`), A2A routes under `/a2a/{app_name}` (JSON-RPC + agent card — A2A is built into every ADK agent), and `/feedback`. Exact routes vary by template; check `app/fast_api_app.py`.
+Scaffolded apps serve the ADK HTTP surface (`/run_sse`, `/apps/...`), A2A routes under `/a2a/{app_name}` (JSON-RPC + agent card — A2A is built into every ADK agent). Exact routes vary by template; check `app/fast_api_app.py`.
 
 ## Testing Your Deployed Agent
 

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-deploy/references/testing-deployed-agents.md

@@ -35,12 +35,12 @@ Beyond the `run --url` quick test above, you can query the deployment directly.
 **Option 1: Python Script**
 ```python
 import json
-import vertexai
+import agentplatform
 
 with open("deployment_metadata.json") as f:
     engine_id = json.load(f)["remote_agent_runtime_id"]
 
-client = vertexai.Client(location="us-east1")
+client = agentplatform.Client(location="us-east1")
 agent = client.agent_engines.get(name=engine_id)
 
 async for event in agent.async_stream_query(message="Hello!", user_id="test"):

{google_agents_cli-1.3.1 → google_agents_cli-1.4.0}/skills/google-agents-cli-eval/SKILL.md

@@ -11,7 +11,7 @@ description: >
 metadata:
   author: Google
   license: Apache-2.0
-  version: 1.3.1
+  version: 1.4.0
   requires:
     bins:
       - agents-cli
@@ -32,63 +32,63 @@ metadata:
 | `references/metrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config |
 | `references/user-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics |
 | `references/builtin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |
+| `references/advanced-commands.md` | Opt-in commands: `eval analyze`, `eval optimize`, `eval submit` / `eval results` |
 | `references/multimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |
 
 ---
 
 ## The Quality Flywheel
 
-Improving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.
+Improving agent quality is iterative. The 4 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.
 
 ### 1. Prepare Data
 
 **Default:** Use or edit the scaffolded `tests/eval/datasets/basic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.
 
-**Opt-in:** `agents-cli eval dataset synthesize` — user-simulate multi-turn datasets when you lack data; its output includes traces, so skip Stage 2 and grade directly. See *Eval Commands* and `references/user-simulation.md`.
+**Opt-in:** `agents-cli eval dataset synthesize`: user-simulate multi-turn datasets when you lack data; its output already includes traces, so Stage 2 collapses to `agents-cli eval grade` alone. See *Eval Commands* and `references/user-simulation.md`.
 
-### 2. Run Inference
+### 2. Run the Eval (always run)
 
-`agents-cli eval generate` — executes the agent over the dataset and writes traces to `artifacts/traces/`. Run this when you wrote the dataset by hand in Stage 1 (default path). **Skip this stage if you used `eval dataset synthesize`** — that command already produced traces.
+**Default:** `agents-cli eval run` runs the agent over the dataset and grades the traces, writing `results_<ts>.{json,html}` to `artifacts/grade_results/`.
 
-### 3. Grade Traces (always run)
+**Decoupled form:** `eval generate` then `eval grade`, for a custom traces location, re-grading without re-running the agent, or traces from `synthesize` (`eval grade` alone).
 
-`agents-cli eval grade` — scores the traces and writes `results_<ts>.{json,html}` to `artifacts/grade_results/`. No opt-in alternative; this is the core. Always run, regardless of how Stages 1 and 2 produced the traces.
-
-> **Shortcut:** `agents-cli eval run` chains Stages 2 + 3 in one command using the default `artifacts/traces/` directory between them. Use it for the common path; drop back to the two-step form when you need a custom traces location or want to grade an existing traces file.
-
-### 4. Analyze Failures
+### 3. Analyze Failures
 
 **Default:** Open the latest `artifacts/grade_results/results_<ts>.html` (or `.json`) and identify failed metrics — see *What to fix when scores fail* below for the fix table.
 
-**Opt-in:** `agents-cli eval analyze` — LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See *Eval Commands*.
+**Opt-in:** `agents-cli eval analyze`, LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See `references/advanced-commands.md`.
 
-### 5. Optimize & Code Fix
+### 4. Optimize & Code Fix
 
 **Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.
 
-**Opt-in:** `agents-cli eval optimize` — runs ADK GEPA prompt optimization against a target metric. Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.
+**Opt-in:** `agents-cli eval optimize` runs ADK GEPA prompt optimization against a target metric (see `references/advanced-commands.md`). Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.
 
 > **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.
 
 ### Running the loop
 
-Iterate stages 2 → 3 → 4 → 5 → 2 (or 1 → 3 → 4 → 5 → 1 if using `synthesize`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes — this is normal. Only after a case passes should you expand coverage with more eval cases.
+Iterate stages 2 → 3 → 4 → 2 (with `synthesize`, re-run Stage 1 each pass, then `eval grade`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes, which is normal. Only after a case passes should you expand coverage with more eval cases.
 
 When doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.
 
+**Hold cases back.** Keep a slice of cases out of the loop and grade them only when you think you're done — otherwise you can't tell a fix that generalizes from one fitted to the cases you iterated against.
+
 ### Shortcuts That Waste Time
 
 Recognize these rationalizations and push back — they always cost more time than they save:
 
 | Shortcut | Why it fails |
 |----------|-------------|
-| "I'll tune the eval thresholds down to make it pass" | Lowering thresholds hides real failures. If the agent can't meet the bar, fix the agent — don't move the bar. |
+| "I'll lower the bar so it passes" | Lowering the bar hides real failures. If the agent can't meet the bar, fix the agent, don't move the bar. |
 | "This eval case is flaky, I'll skip it" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. |
 | "I just need to fix the eval dataset, not the agent" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |
+| "I'll iterate until every case I have passes" | Nothing is left to detect overfitting to your own cases. See *Hold cases back* above. |
 
 ## Choosing the Right Metrics
 
-Pick built-in metrics by what you want to measure. Multi-turn metrics evaluate the full conversation; single-turn metrics evaluate one prompt-response pair (with intermediate tool calls). When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).
+Pick built-in metrics by what you want to measure. Only `multi_turn_task_success`, `multi_turn_trajectory_quality`, and `multi_turn_tool_use_quality` accept multi-turn traces; every other built-in 400s on one. When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).
 
 | Goal | Recommended built-in metrics |
 |------|------------------------------|
@@ -96,8 +96,10 @@ Pick built-in metrics by what you want to measure. Multi-turn metrics evaluate t
 | **Was the agent's reasoning path logical and efficient?** | `multi_turn_trajectory_quality` |
 | **Quality of tool / function calling across turns** | `multi_turn_tool_use_quality` |
 | **Final response quality** (no ground-truth reference needed) | `final_response_quality` |
-| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination` |
+| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination`, or `grounding` when the case carries a `context` field |
 | **Safety policy compliance** | `safety` |
+| **Match against a golden answer** | `final_response_match` (needs `reference` on the case) |
+| **Different pass/fail criteria per case** | Put them on the case as `rubric_groups` and grade with a managed rubric metric. See `references/dataset_schema.md` (*Per-Case Rubrics*). |
 | **Domain-specific check no built-in covers** | Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See *Evaluation Configuration Schema* below. |
 
 Run `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Agent Platform metric docs](https://cloud.google.com/gemini-enterprise-agent-platform/optimize/evaluation/manage-metrics) and `references/metrics-guide.md`.
@@ -106,7 +108,7 @@ Run `agents-cli eval metric list` to see all available built-ins. For full metri
 
 ## What to fix when scores fail
 
-After `agents-cli eval grade` completes, inspect the latest `artifacts/grade_results/results_<timestamp>.json` (or open the `.html` file) for per-case scores and judge rationales — that's the input to every fix decision below.
+After `agents-cli eval run` completes, inspect the latest `artifacts/grade_results/results_<timestamp>.json` (or open the `.html` file) for per-case scores and judge rationales, the input to every fix decision below.
 
 | Failure | What to change |
 |---------|---------------|
@@ -119,13 +121,26 @@ After `agents-cli eval grade` completes, inspect the latest `artifacts/grade_res
 | Agent calls wrong tools | Fix tool descriptions, agent instructions, or `tool_config` |
 | Agent calls extra tools | Add strict stop instructions, or switch to `multi_turn_tool_use_quality` |
 
-After applying a fix, rerun `agents-cli eval generate && agents-cli eval grade` and use `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the fix improved the target metric without regressing others.
+After applying a fix, rerun `agents-cli eval run` and use `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the fix improved the target metric without regressing others.
 
 ---
 
 ## Eval Commands
 
-All `agents-cli eval` subcommands support `--help` for the authoritative flag list and defaults — run `agents-cli eval <subcommand> --help` (or `agents-cli eval dataset <subcommand> --help`) when in doubt. The examples below show the most common invocations; flags can change between releases.
+`agents-cli eval <subcommand> --help` is the authoritative flag list; the examples below are the common invocations.
+
+### `eval run` (default)
+
+Runs the agent over the dataset and grades the traces in one command.
+
+```bash
+# Basic: dataset from tests/eval/datasets/, results to artifacts/grade_results/,
+# metrics from tests/eval/eval_config.yaml
+agents-cli eval run
+
+# Advanced: pick the dataset, metrics, and output dir
+agents-cli eval run --dataset tests/eval/datasets/custom.json --metrics final_response_quality,safety --output ./out/
+```
 
 ### `eval generate`
 
@@ -146,7 +161,7 @@ agents-cli eval generate --url https://my-agent.run.app --app-name app
 
 ### `eval grade`
 
-Scores generated traces against built-in or custom metrics. Writes timestamped `results_<YYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console.
+Scores traces (from `eval generate`, `eval dataset synthesize`, or hand-authored) against built-in or custom metrics. Writes timestamped `results_<YYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console.
 
 ```bash
 # Basic — defaults: traces from artifacts/traces/, results to artifacts/grade_results/,
@@ -157,10 +172,7 @@ agents-cli eval grade
 # pairing for `eval generate --output custom_traces/`)
 agents-cli eval grade --traces custom_traces/
 
-# Advanced 2 — pick built-in metrics, custom output dir
-agents-cli eval grade --metrics tool_use_quality,safety --output ./out/
-
-# Advanced 3 — load metrics to run from a config file (YAML or JSON) on a specified trace file.
+# Advanced 2: load metrics to run from a config file (YAML or JSON) on a specified trace file.
 agents-cli eval grade --traces ./artifacts/traces/trace_1.json --config tests/eval/eval_config.yaml
 ```
 
@@ -168,79 +180,19 @@ See *Evaluation Configuration Schema* below for the config file format.
 
 ### `eval compare`
 
-Diffs two `results_*.json` files produced by `eval grade`. Run it after a fix to confirm the target metric improved without regressing others.
+Diffs two `results_*.json` files from an eval run. Run it after a fix to confirm the target metric improved without regressing others.
 
 ```bash
 agents-cli eval compare baseline.json candidate.json
 ```
 
-### `eval metric list`
-
-Lists the built-in metric names usable with `eval grade --metrics`.
-
-```bash
-agents-cli eval metric list
-```
-
-### `eval analyze`
-
-Runs LLM-based failure clustering and root-cause analysis over a `results_*.json` produced by `eval grade`. Use when you have 10+ failing cases and want categorized failure modes instead of reading the HTML case-by-case. Supported `--metric` values: `multi_turn_task_success`, `multi_turn_tool_use_quality`.
-
-```bash
-# Basic — analyze a results file with default settings
-agents-cli eval analyze --eval-result artifacts/grade_results/results_<ts>.json
-
-# Advanced — restrict to a specific metric and cap loss clusters
-agents-cli eval analyze \
-  --eval-result artifacts/grade_results/results_<ts>.json \
-  --metric multi_turn_tool_use_quality \
-  --top-k 5 \
-  --output artifacts/analysis_<ts>.json
-```
-
 ### `eval dataset synthesize`
 
-Generates user scenarios server-side from your agent's tools and instructions, then plays each scenario against an LLM-backed user simulator. The output is a graded-ready trace file with full `agent_data.turns` populated — feed it directly to `eval grade` (skip `eval generate`).
+Generates user scenarios from your agent's tools and instructions, plays each against an LLM-backed user simulator, and writes graded-ready traces to `artifacts/traces/` (feed straight to `eval grade`, skip `eval generate`). Invocations, flags, and compatible metrics: `references/user-simulation.md`.
 
-```bash
-# Basic — generate 3 default scenarios (up to 5 turns each) into artifacts/traces/
-# (where eval grade reads from by default, so synthesize → grade works without flags)
-agents-cli eval dataset synthesize
-
-# Advanced — guide scenario generation with optional instruction and environment context
-agents-cli eval dataset synthesize \
-  -n 5 \
-  --instruction "Customer asking about refunds" \
-  --environment-context "E-commerce support" \
-  --max-turns 8 \
-  -o tests/eval/datasets/refund_scenarios.json
-```
+### Advanced commands
 
-For scenario semantics, the full `eval dataset synthesize` flag table, and which simulator internals are not user-configurable, see `references/user-simulation.md`.
-
-### `eval optimize`
-
-Runs ADK GEPA prompt optimization against a target metric. Suitable after `eval grade` identifies prompt-only failures (wording, not tool/orchestration logic). `--dataset` and `--target-metric` override values in `--config` when both are passed. **Long-running and expensive — see Stage 5 of the Quality Flywheel for usage guidance.**
-
-```bash
-# Basic — optimize against a single metric on a dataset
-agents-cli eval optimize --dataset tests/eval/datasets/basic-dataset.json --target-metric final_response_quality
-
-# Advanced — drive multi-metric / multi-dataset optimization from a config file
-agents-cli eval optimize --config tests/eval/optimization_config.json
-```
-
-### `eval submit` / `eval results` (cloud-side)
-
-The managed, asynchronous counterpart to the local path, for large or CI-driven runs: `eval submit` hands the dataset and metrics to the Agent Platform Eval Service, and `eval results` polls and downloads the scores. Pass `--resource-name <agent>` to also run inference server-side (managed `generate` + `grade`); omit it to grade an existing trace (managed `grade`).
-
-```bash
-# Grade an existing trace server-side; returns a run resource name to poll
-agents-cli eval submit --dataset tests/eval/datasets/basic-dataset.json --dest gs://my-bucket
-# Add --resource-name projects/<p>/locations/<l>/reasoningEngines/<id> to run inference too
-
-agents-cli eval results --run-id <run-resource-name>
-```
+`eval analyze` (cluster failure modes), `eval optimize` (GEPA prompt tuning), and `eval submit` / `eval results` (managed cloud-side runs for CI or large datasets) are documented in `references/advanced-commands.md`.
 
 ---
 
@@ -268,13 +220,6 @@ Two shapes are supported.
         "role": "user",
         "parts": [{"text": "Hello, what can you help me with?"}]
       }
-    },
-    {
-      "eval_case_id": "weather_query",
-      "prompt": {
-        "role": "user",
-        "parts": [{"text": "What's the weather like in San Francisco?"}]
-      }
     }
   ]
 }
@@ -290,9 +235,9 @@ A complete trace — agent responses plus `function_call` / `function_response`
 
 ## Evaluation Configuration Schema
 
-`agents-cli eval grade --config <path>` accepts a single configuration file in either **YAML** (`.yaml` / `.yml`) or **JSON** (`.json`). The file declares two parts:
+`agents-cli eval run --config <path>` (and `eval grade --config <path>`) accepts a single configuration file in either **YAML** (`.yaml` / `.yml`) or **JSON** (`.json`). The file declares two parts:
 
-- `metrics_to_run` — the **selection list** of metric names to execute on this run. Names resolve to built-in metrics first, then to entries in `custom_metrics`.
+- `metrics_to_run`: the **selection list** of metric names to execute on this run. A name resolves to a `custom_metrics` entry when one matches, otherwise to the built-in metric of that name.
 - `custom_metrics` — a **definition pool** of custom metrics available to this project. Defining a metric here does **not** run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).
 
 **Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):**
@@ -321,7 +266,7 @@ custom_metrics:
 
 JSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but **always prefer YAML** for human-readable configs.
 
-Each entry in `custom_metrics` is dispatched by field: presence of `custom_function` makes it a `CodeExecutionMetric` (deterministic Python); otherwise it's an `LLMMetric` (LLM-as-judge with `prompt_template`). Run `agents-cli eval metric list` to see available built-ins. For full custom-metric field reference (judge model options, sampling counts), see `references/metrics-guide.md`.
+Dispatch by field: `custom_function` → Python metric; `prompt_template` → `LLMMetric` (LLM-as-judge); neither, on a built-in name → parameterizes that built-in (e.g. `metric_spec_parameters.rubric_group_key`). Field reference: `references/metrics-guide.md`.
 
 **Agent trace field model.** For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:
 
@@ -329,7 +274,7 @@ Each entry in `custom_metrics` is dispatched by field: presence of `custom_funct
 - `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{"role": "model", "parts": [{"text": "..."}]}`.
 - `{agent_data}` — the full structured `turns`/`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.
 
-`{reference}` and `{context}` resolve only when the eval case has `reference` / `context` fields populated (e.g., golden-answer datasets); they are not populated by `eval generate` / `eval dataset synthesize`.
+`reference`, `context`, and `rubric_groups` are yours to author on the case: `eval generate` carries them onto the trace but never invents them, so `{reference}` / `{context}` resolve only where you wrote them. `rubric_groups` is not a placeholder at all: managed rubric metrics read it off the case, and a `custom_function` sees `instance['rubric_groups']`. See `references/dataset_schema.md` (*Per-Case Rubrics*).
 
 Code-based metrics default to **local in-process execution** (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: "remote"` on the metric to run it server-side in Vertex AI's `CodeExecutionMetric` sandbox instead — that path requires a configured GCP project + region.
 
@@ -358,7 +303,7 @@ app = App(root_agent=root_agent, name="flight_booking_assistant")
 
 ### Vertex eval region
 
-`eval grade` and `eval submit` **default to the `global` endpoint** — they don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region <REGION>` (e.g. data residency); the service rejects an unsupported one:
+`eval run`, `eval grade`, and `eval submit` **default to the `global` endpoint**. They don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region <REGION>` (e.g. data residency); the service rejects an unsupported one:
 
 ```
 400 FAILED_PRECONDITION: Unsupported region for Vertex Evaluation Service: <region>
@@ -395,34 +340,18 @@ Models with "thinking" enabled may skip tool calls. Use `tool_config` with `mode
 
 | Symptom | Cause | Fix |
 |---------|-------|-----|
-| Agent mentions data not in tool output | Hallucination | Tighten agent instructions; add `hallucination` metric |
-| "Session not found" error | App name mismatch | Ensure App `name` matches directory name |
 | Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval with multiple samples |
-| `tool_use_quality` score low | Wrong tool selected or invalid arguments passed | Refine tool descriptions, instructions, or parameter documentation |
 | LLM judge ignores image/audio in eval | `get_text_from_content()` skips non-text parts | Use custom metric with vision-capable judge (see `references/multimodal-eval.md`) |
 
 ---
 
-## Debugging Example
-
-User says: "tool_use_quality is low, what's wrong?"
-
-1. Open the latest `artifacts/grade_results/results_<timestamp>.html` (or read the `.json`) and find the rubric verdicts the adaptive metric generated for the failing case.
-2. Verify whether the agent selected the wrong tool, or called it with wrong arguments — the trace lives in `artifacts/traces/`.
-3. Refine the tool's parameters, Python docstring description, or the agent's tool selection instructions to guide the model better.
-4. Rerun `agents-cli eval generate && agents-cli eval grade`.
-5. `agents-cli eval compare <prev>.json <new>.json` to confirm the score improved.
-
----
-
 ## Proving Your Work
 
 Don't assert that eval passes — show the evidence. Concrete output prevents false confidence and catches issues early.
 
 - **After running eval:** Paste the scores table output so the user can see exactly what passed and failed.
 - **After fixing a failure:** Show before/after scores for the specific case you fixed, and confirm no other cases regressed.
-- **Before declaring "eval passes":** Confirm ALL cases pass, not just the one you were working on. Run `agents-cli eval generate` and `agents-cli eval grade` one final time.
-- **Before moving to deploy:** Show the final `agents-cli eval grade` output with all cases above threshold. This is the gate — no exceptions.
+- **Before deploy:** Rerun `agents-cli eval run` and show the scores for every case, not just the one you fixed. `eval run` exits 0 whatever the scores are, so the numbers you paste are the gate, not the exit code.
 
 ---