harness-mcp-v2 3.0.3 → 3.0.4

package/README.md

@@ -1,6 +1,6 @@
 ## Harness MCP Server 2.0
 
-An MCP (Model Context Protocol) server that gives AI agents full access to the Harness.io platform through 11 consolidated tools and 169 resource types.
+An MCP (Model Context Protocol) server that gives AI agents full access to the Harness.io platform through 11 consolidated tools and 187 resource types.
 
 ## Why Use This MCP Server
 
@@ -8,8 +8,8 @@ Most MCP servers map one tool per API endpoint. For a platform as broad as Harne
 
 This server is built differently:
 
-- **11 tools, 169 resource types.** A registry-based dispatch system routes `harness_list`, `harness_get`, `harness_create`, etc. to any Harness resource — pipelines, services, environments, orgs, projects, feature flags, cost data, and more. The LLM picks from 11 tools instead of hundreds.
-- **Full platform coverage.** 31 toolsets spanning CI/CD, GitOps, Feature Flags, Cloud Cost Management, Security Testing, Chaos Engineering, Database DevOps, Internal Developer Portal, Software Supply Chain, Governance, Service Overrides, Visualizations, and more. Not just pipelines — the entire Harness platform.
+- **11 tools, 187 resource types.** A registry-based dispatch system routes `harness_list`, `harness_get`, `harness_create`, etc. to any Harness resource — pipelines, services, environments, orgs, projects, feature flags, cost data, and more. The LLM picks from 11 tools instead of hundreds.
+- **Full platform coverage.** 32 toolsets spanning CI/CD, GitOps, Feature Flags, Cloud Cost Management, Security Testing, Chaos Engineering, Database DevOps, Internal Developer Portal, Software Supply Chain, Governance, Service Overrides, Visualizations, and more. Not just pipelines — the entire Harness platform.
 - **Multi-project workflows out of the box.** Agents discover organizations and projects dynamically — no hardcoded env vars needed. Ask "show failed executions across all projects" and the agent can navigate the full account hierarchy.
 - **31 prompt templates.** Pre-built prompts for common workflows: build & deploy apps end-to-end, debug failed pipelines, review DORA metrics, triage vulnerabilities, optimize cloud costs, audit access control, plan feature flag rollouts, review pull requests, approve pending pipelines, and more.
 - **Works everywhere.** Stdio transport for local clients (Claude Desktop, Cursor, Windsurf), HTTP transport for remote/shared deployments, Docker and Kubernetes ready.
@@ -528,7 +528,7 @@ The server automatically loads environment variables from a `.env` file in the p
 | `HARNESS_MAX_BODY_SIZE_MB`  | No       | `10`                        | Max HTTP request body size in MB for `http` transport                                                                                                                                                                                                 |
 | `HARNESS_RATE_LIMIT_RPS`    | No       | `10`                        | Client-side request throttle (requests per second) to Harness APIs                                                                                                                                                                                    |
 | `LOG_LEVEL`                 | No       | `info`                      | Log verbosity: `debug`, `info`, `warn`, `error`                                                                                                                                                                                                       |
-| `HARNESS_TOOLSETS`          | No       | *(defaults)*                | Comma-separated toolset list. Empty loads default toolsets and excludes opt-in toolsets such as `ai-evals`. Supports `+name` to add opt-in toolsets and `-name` to remove defaults (see [Toolset Filtering](#toolset-filtering))                      |
+| `HARNESS_TOOLSETS`          | No       | *(all)*                     | Comma-separated toolset list. Empty loads all toolsets. Supports `+name` to explicitly include and `-name` to remove toolsets (see [Toolset Filtering](#toolset-filtering))                                                                           |
 | `HARNESS_READ_ONLY`         | No       | `false`                     | Block all mutating operations (create, update, delete, execute). Only list and get are allowed. Useful for shared/demo environments                                                                                                                   |
 | `HARNESS_AUTO_APPROVE_RISK` | No       | `none`                      | Risk-based auto-approve threshold for autonomous workflows. Operations at or below this risk proceed without confirmation. Values: `none`, `low_write`, `medium_write`, `high_write`, `all`. See [Elicitation](#elicitation)                          |
 | `HARNESS_SKIP_ELICITATION`  | No       | `false`                     | **Deprecated** — use `HARNESS_AUTO_APPROVE_RISK=all` instead. Kept for backward compatibility                                                                                                                                                         |
@@ -991,7 +991,7 @@ Harness pipelines can be stored in three ways:
 
 ## Resource Types
 
-169 resource types organized across 31 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions.
+187 resource types organized across 32 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions.
 
 ### Platform
 
@@ -1446,14 +1446,14 @@ Inline PNG chart visualizations rendered from Harness data. These are metadata-o
 
 ## Toolset Filtering
 
-By default, 31 of 32 toolsets are enabled. One toolset (`ai-evals`) is opt-in — excluded by default to avoid polluting the resource list for users who don't need it.
+By default, all 32 toolsets are enabled.
 
-### Enabling opt-in toolsets
+### Adding toolsets with `+` prefix
 
-Use the `+` prefix to add opt-in toolsets to the defaults:
+Use the `+` prefix to explicitly include toolsets alongside all defaults (useful if a toolset becomes opt-in in the future):
 
 ```bash
-# Enable ai-evals alongside all defaults
+# Explicitly include ai-evals alongside all defaults
 HARNESS_TOOLSETS=+ai-evals
 ```
 
@@ -1518,7 +1518,7 @@ Available toolset names:
 | `overrides`             | service_override                                                                                                                                                                                                                                                                                |
 | `settings`              | setting                                                                                                                                                                                                                                                                                         |
 | `visualizations`        | visual_timeline, visual_stage_flow, visual_health_dashboard, visual_pie_chart, visual_bar_chart, visual_timeseries, visual_architecture                                                                                                                                                         |
-| `ai-evals` **(opt-in)** | eval_dataset, eval_dataset_item, evaluation, eval_run, eval_run_item, eval_run_by_eval, eval_metric, eval_metric_set, eval_metric_set_entry, eval_suite, eval_suite_evaluation, eval_suite_run, eval_target, eval_model, eval_annotation, eval_analytics, eval_git_settings, eval_registry_item |
+| `ai-evals`              | eval_dataset, eval_dataset_item, evaluation, eval_run, eval_run_item, eval_run_by_eval, eval_metric, eval_metric_set, eval_metric_set_entry, eval_suite, eval_suite_evaluation, eval_suite_run, eval_target, eval_model, eval_annotation, eval_analytics, eval_git_settings, eval_registry_item |
 
 
 ## Architecture
@@ -1537,7 +1537,7 @@ Available toolset names:
                  +--------v---------+
                 |    Registry       |  <-- Declarative resource definitions
                 |  32 Toolsets      |      (data files, not code)
-                |  169 Resource Types|
+                |  187 Resource Types|
                  +--------+---------+
                           |
                  +--------v---------+

package/build/prompts/create-eval-suite.js

@@ -27,7 +27,7 @@ export function registerCreateEvalSuitePrompt(server) {
 
 1. **List existing evaluations** — \`harness_list(resource_type="evaluation")\` — pick eval IDs to include.
 
-2. **Create the suite** — \`harness_create(resource_type="eval_suite", body={ name: "${suite_name}", pass_strategy: "all_must_pass" | "weighted_threshold", pass_threshold?: number })\`
+2. **Create the suite** — \`harness_create(resource_type="eval_suite", body={ name: "${suite_name}", pass_strategy: "all_must_pass" | "weighted_threshold", pass_threshold?: number, schedule?: { cron: "0 8 * * *", timezone?: "UTC", enabled?: true } })\`
 
 3. **Add members** (choose one pattern)
    - **Add one at a time**: \`harness_create(resource_type="eval_suite_evaluation", params={ suite_id: "<SUITE_ID>" }, body={ evaluation_id: "<EVAL_ID>", is_required: true })\`

package/build/prompts/run-evaluation.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"run-evaluation.d.ts","sourceRoot":"","sources":["../../src/prompts/run-evaluation.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,yCAAyC,CAAC;AAEzE,wBAAgB,2BAA2B,CAAC,MAAM,EAAE,SAAS,GAAG,IAAI,CAmDnE"}
+{"version":3,"file":"run-evaluation.d.ts","sourceRoot":"","sources":["../../src/prompts/run-evaluation.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,yCAAyC,CAAC;AAEzE,wBAAgB,2BAA2B,CAAC,MAAM,EAAE,SAAS,GAAG,IAAI,CAqDnE"}

package/build/prompts/run-evaluation.js

@@ -40,7 +40,9 @@ ${org_id ? `org_id=${org_id}` : "Use default HARNESS_ORG"}; ${project_id ? `proj
 ## Notes
 - All AI Evals paths are under the Harness platform host (same \`HARNESS_BASE_URL\` as the UI); scope is \`/ai-evals/api/v1/orgs/{org}/projects/{project}/...\`.
 - Ensure \`Harness-Account\` and auth are configured on the MCP server.
-- For comparing runs: \`harness_execute(resource_type="eval_run", action="compare", params={ run_ids: "uuid1,uuid2,uuid3" })\` (no resource_id needed for compare).`,
+- Resources support **git-backed storage**: pass \`storage_type: "git"\` + \`git_source: { file_path: "..." }\` on create to reference YAML definitions in a repo instead of managing inline.
+- For comparing runs: \`harness_execute(resource_type="eval_run", action="compare", params={ run_ids: "uuid1,uuid2,uuid3" })\` (no resource_id needed for compare).
+- For git-backed evals, pass \`branch\` in the trigger body to run against a PR branch.`,
                 },
             },
         ],

package/build/prompts/run-evaluation.js.map

@@ -1 +1 @@
-{"version":3,"file":"run-evaluation.js","sourceRoot":"","sources":["../../src/prompts/run-evaluation.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,CAAC,MAAM,QAAQ,CAAC;AAG5B,MAAM,UAAU,2BAA2B,CAAC,MAAiB;IAC3D,MAAM,CAAC,cAAc,CACnB,gBAAgB,EAChB;QACE,WAAW,EACT,oHAAoH;QACtH,UAAU,EAAE;YACV,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,0EAA0E,CAAC;YACrG,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,wBAAwB,CAAC,CAAC,QAAQ,EAAE;YAChE,UAAU,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,4BAA4B,CAAC,CAAC,QAAQ,EAAE;SACzE;KACF,EACD,KAAK,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC,CAAC;QACvC,QAAQ,EAAE;YACR;gBACE,IAAI,EAAE,MAAe;gBACrB,OAAO,EAAE;oBACP,IAAI,EAAE,MAAe;oBACrB,IAAI,EAAE;;;EAGhB,IAAI;;;EAGJ,MAAM,CAAC,CAAC,CAAC,UAAU,MAAM,EAAE,CAAC,CAAC,CAAC,yBAAyB,KAAK,UAAU,CAAC,CAAC,CAAC,cAAc,UAAU,EAAE,CAAC,CAAC,CAAC,6BAA6B;;;;;;;;;;;;;;;;;;;;;oKAqB+B;iBACzJ;aACF;SACF;KACF,CAAC,CACH,CAAC;AACJ,CAAC"}
+{"version":3,"file":"run-evaluation.js","sourceRoot":"","sources":["../../src/prompts/run-evaluation.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,CAAC,MAAM,QAAQ,CAAC;AAG5B,MAAM,UAAU,2BAA2B,CAAC,MAAiB;IAC3D,MAAM,CAAC,cAAc,CACnB,gBAAgB,EAChB;QACE,WAAW,EACT,oHAAoH;QACtH,UAAU,EAAE;YACV,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,0EAA0E,CAAC;YACrG,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,wBAAwB,CAAC,CAAC,QAAQ,EAAE;YAChE,UAAU,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,4BAA4B,CAAC,CAAC,QAAQ,EAAE;SACzE;KACF,EACD,KAAK,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC,CAAC;QACvC,QAAQ,EAAE;YACR;gBACE,IAAI,EAAE,MAAe;gBACrB,OAAO,EAAE;oBACP,IAAI,EAAE,MAAe;oBACrB,IAAI,EAAE;;;EAGhB,IAAI;;;EAGJ,MAAM,CAAC,CAAC,CAAC,UAAU,MAAM,EAAE,CAAC,CAAC,CAAC,yBAAyB,KAAK,UAAU,CAAC,CAAC,CAAC,cAAc,UAAU,EAAE,CAAC,CAAC,CAAC,6BAA6B;;;;;;;;;;;;;;;;;;;;;;;wFAuB7C;iBAC7E;aACF;SACF;KACF,CAAC,CACH,CAAC;AACJ,CAAC"}

package/build/registry/toolsets/ai-evals.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"ai-evals.d.ts","sourceRoot":"","sources":["../../../src/registry/toolsets/ai-evals.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,OAAO,KAAK,EAAiC,iBAAiB,EAAE,MAAM,aAAa,CAAC;AA8gBpF,eAAO,MAAM,cAAc,EAAE,iBAmqC5B,CAAC"}
+{"version":3,"file":"ai-evals.d.ts","sourceRoot":"","sources":["../../../src/registry/toolsets/ai-evals.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,OAAO,KAAK,EAAiC,iBAAiB,EAAE,MAAM,aAAa,CAAC;AA4jBpF,eAAO,MAAM,cAAc,EAAE,iBAitC5B,CAAC"}

package/build/registry/toolsets/ai-evals.js

@@ -24,10 +24,12 @@ const createDatasetSchema = {
             name: "items",
             type: "array",
             required: false,
-            description: "Inline dataset items (CreateDatasetItemRequest[])",
+            description: "Inline dataset items (CreateDatasetItemRequest[]); must be empty when storage_type='git'",
             itemType: "object",
         },
         { name: "metadata", type: "object", required: false, description: "Arbitrary metadata" },
+        { name: "storage_type", type: "string", required: false, description: "managed (default) | git" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const updateDatasetSchema = {
@@ -40,22 +42,25 @@ const updateDatasetSchema = {
             name: "items",
             type: "array",
             required: false,
-            description: "Replace items (CreateDatasetItemRequest[])",
+            description: "Replace items (CreateDatasetItemRequest[]); must be empty when storage_type='git'",
             itemType: "object",
         },
         { name: "metadata", type: "object", required: false, description: "Metadata" },
+        { name: "storage_type", type: "string", required: false, description: "managed | git (switches storage mode)" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when switching to storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const createDatasetItemSchema = {
     description: "Create dataset item",
     fields: [
-        { name: "id", type: "string", required: false, description: "Business id for the row" },
+        { name: "id", type: "string", required: false, description: "Business id for the row (required for CLI/backend run correlation)" },
         { name: "input", type: "object", required: true, description: "Model input (JSON)" },
         { name: "expected_output", type: "object", required: false, description: "Expected output" },
-        { name: "precomputed_output", type: "object", required: false, description: "Precomputed target output" },
-        { name: "context", type: "array", required: false, description: "Context strings", itemType: "string" },
-        { name: "expected_tools", type: "array", required: false, description: "Expected tool names", itemType: "string" },
-        { name: "comments", type: "string", required: false, description: "Comments" },
+        { name: "precomputed_output", type: "object", required: false, description: "Precomputed target output for offline/metrics-only scoring" },
+        { name: "context", type: "array", required: false, description: "Retrieved chunks for RAG evaluation", itemType: "string" },
+        { name: "expected_tools", type: "array", required: false, description: "Expected tool names for agent evaluation", itemType: "string" },
+        { name: "expected_tool_calls", type: "array", required: false, description: "Structured tool call expectations (name + arguments) for ToolArgumentMatchMetric", itemType: "object" },
+        { name: "comments", type: "string", required: false, description: "Internal notes, not used in evaluation" },
         { name: "metadata", type: "object", required: false, description: "Extra metadata" },
         { name: "sort_order", type: "number", required: false, description: "Sort position" },
     ],
@@ -65,10 +70,11 @@ const updateDatasetItemSchema = {
     fields: [
         { name: "input", type: "object", required: false, description: "Input" },
         { name: "expected_output", type: "object", required: false, description: "Expected output" },
-        { name: "precomputed_output", type: "object", required: false, description: "Precomputed target output" },
-        { name: "context", type: "array", required: false, description: "Context", itemType: "string" },
-        { name: "expected_tools", type: "array", required: false, description: "Expected tool names", itemType: "string" },
-        { name: "comments", type: "string", required: false, description: "Comments" },
+        { name: "precomputed_output", type: "object", required: false, description: "Precomputed target output for offline/metrics-only scoring" },
+        { name: "context", type: "array", required: false, description: "Retrieved chunks for RAG evaluation", itemType: "string" },
+        { name: "expected_tools", type: "array", required: false, description: "Expected tool names for agent evaluation", itemType: "string" },
+        { name: "expected_tool_calls", type: "array", required: false, description: "Structured tool call expectations (name + arguments) for ToolArgumentMatchMetric", itemType: "object" },
+        { name: "comments", type: "string", required: false, description: "Internal notes, not used in evaluation" },
         { name: "metadata", type: "object", required: false, description: "Metadata" },
         { name: "sort_order", type: "number", required: false, description: "Sort position" },
     ],
@@ -87,6 +93,8 @@ const createEvalSchema = {
         { name: "concurrency", type: "number", required: false, description: "Parallelism (default 5, min 1)" },
         { name: "cost_limit_usd", type: "number", required: false, description: "Max cost in USD" },
         { name: "timeout_per_item_ms", type: "number", required: false, description: "Per-item timeout ms (default 30000, min 1000)" },
+        { name: "storage_type", type: "string", required: false, description: "managed (default) | git" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const updateEvalSchema = {
@@ -104,6 +112,8 @@ const updateEvalSchema = {
         { name: "concurrency", type: "number", required: false, description: "Parallelism (min 1)" },
         { name: "cost_limit_usd", type: "number", required: false, description: "Max cost in USD" },
         { name: "timeout_per_item_ms", type: "number", required: false, description: "Per-item timeout ms (min 1000)" },
+        { name: "storage_type", type: "string", required: false, description: "managed | git (switches storage mode)" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when switching to storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const triggerEvalRunSchema = {
@@ -120,19 +130,23 @@ const triggerEvalRunSchema = {
             description: "RunInputs overrides: { model_id?, target_id?, dataset_id?, metric_set_id?, variables? }",
         },
         { name: "input_set_id", type: "string", required: false, description: "Saved input set id" },
+        { name: "branch", type: "string", required: false, description: "Override git branch (e.g. run against a PR branch)" },
     ],
 };
 const createRunSchema = {
     description: "Create run. Provide eval_id XOR suite (not both).",
     fields: [
         { name: "eval_id", type: "string", required: false, description: "Eval UUID (mutually exclusive with suite)" },
-        { name: "suite", type: "object", required: false, description: "Suite config (mutually exclusive with eval_id)" },
+        { name: "suite", type: "object", required: false, description: "Full suite definition (mutually exclusive with eval_id)" },
         { name: "name", type: "string", required: false, description: "Run name" },
         { name: "pass_threshold", type: "number", required: false, description: "Pass threshold" },
         { name: "dataset_id", type: "string", required: false, description: "Dataset UUID" },
-        { name: "dataset_snapshot", type: "object", required: false, description: "Dataset snapshot" },
+        { name: "dataset_snapshot", type: "object", required: false, description: "Dataset snapshot (when dataset_id is omitted)" },
         { name: "variant_id", type: "string", required: false, description: "Variant identifier" },
-        { name: "environment", type: "string", required: false, description: "Environment label" },
+        { name: "suite_run_id", type: "string", required: false, description: "Parent SuiteRun UUID (links child run to suite run)" },
+        { name: "target_id", type: "string", required: false, description: "Target UUID (for single-eval runs)" },
+        { name: "metric_set_id", type: "string", required: false, description: "MetricSet UUID (for single-eval runs)" },
+        { name: "environment", type: "string", required: false, description: "Execution environment (local, ci, prod)" },
         { name: "metadata", type: "object", required: false, description: "Arbitrary metadata" },
         { name: "trigger_type", type: "string", required: false, description: "manual | scheduled | api | ci (default manual)" },
     ],
@@ -147,6 +161,7 @@ const updateRunSchema = {
         { name: "success_count", type: "number", required: false, description: "Successful items" },
         { name: "failed_count", type: "number", required: false, description: "Failed items" },
         { name: "summary_scores", type: "object", required: false, description: "Aggregated scores { metric_name: float }" },
+        { name: "git_commit_sha", type: "string", required: false, description: "Resolved commit SHA (max 64 chars, set once, ignored on subsequent updates)" },
     ],
 };
 const rescoreSchema = {
@@ -165,9 +180,9 @@ const createMetricSchema = {
     description: "Create custom metric",
     fields: [
         { name: "name", type: "string", required: true, description: "Metric name" },
-        { name: "type", type: "string", required: true, description: "Metric type (e.g. exact_match, ai_judge, json_diff)" },
+        { name: "type", type: "string", required: true, description: "heuristic | llm | embedding | code | composite" },
         { name: "description", type: "string", required: false, description: "Description" },
-        { name: "kind", type: "string", required: false, description: "Kind slug (e.g. exact_match)" },
+        { name: "kind", type: "string", required: false, description: "harness-evals metric kind identifier (e.g. exact_match, contains, levenshtein)" },
         { name: "config", type: "object", required: false, description: "Metric config JSON" },
         { name: "default_threshold", type: "number", required: false, description: "Default threshold 0-1 (default 0.8)" },
         { name: "tags", type: "array", required: false, description: "Tags", itemType: "string" },
@@ -191,6 +206,8 @@ const createMetricSetSchema = {
         { name: "name", type: "string", required: true, description: "Name" },
         { name: "description", type: "string", required: false, description: "Description" },
         { name: "tags", type: "array", required: false, description: "Tags", itemType: "string" },
+        { name: "judge_model_id", type: "string", required: false, description: "Default judge model UUID for LLM metrics in this set" },
+        { name: "entries", type: "array", required: false, description: "Initial metric entries (AddMetricSetEntryRequest[])", itemType: "object" },
     ],
 };
 const updateMetricSetSchema = {
@@ -199,6 +216,7 @@ const updateMetricSetSchema = {
         { name: "name", type: "string", required: false, description: "Name" },
         { name: "description", type: "string", required: false, description: "Description" },
         { name: "tags", type: "array", required: false, description: "Tags", itemType: "string" },
+        { name: "judge_model_id", type: "string", required: false, description: "Default judge model UUID for LLM metrics" },
     ],
 };
 const addMetricSetEntrySchema = {
@@ -208,6 +226,7 @@ const addMetricSetEntrySchema = {
         { name: "threshold", type: "number", required: true, description: "Pass threshold 0-1" },
         { name: "weight", type: "number", required: false, description: "Weight" },
         { name: "position", type: "number", required: false, description: "Order" },
+        { name: "config", type: "object", required: false, description: "Per-use-site config override (merged over metric's base config at eval time)" },
     ],
 };
 const updateMetricSetEntrySchema = {
@@ -216,6 +235,7 @@ const updateMetricSetEntrySchema = {
         { name: "threshold", type: "number", required: false, description: "Threshold" },
         { name: "weight", type: "number", required: false, description: "Weight" },
         { name: "position", type: "number", required: false, description: "Position" },
+        { name: "config", type: "object", required: false, description: "Per-use-site config override (merged over metric's base config at eval time)" },
     ],
 };
 const calibrateSchema = {
@@ -230,11 +250,14 @@ const createSuiteSchema = {
     fields: [
         { name: "name", type: "string", required: true, description: "Suite name" },
         { name: "description", type: "string", required: false, description: "Description" },
-        { name: "purpose", type: "string", required: false, description: "Purpose (default custom)" },
+        { name: "purpose", type: "string", required: false, description: "pr_gate | cd_gate | release_gate | custom (default custom)" },
         { name: "pass_strategy", type: "string", required: false, description: "all_must_pass | weighted_threshold (default all_must_pass)" },
         { name: "pass_threshold", type: "number", required: false, description: "0-1, used when strategy is weighted_threshold" },
         { name: "is_blocking", type: "boolean", required: false, description: "Blocking suite (default true)" },
         { name: "triggered_by", type: "string", required: false, description: "Who created the suite" },
+        { name: "schedule", type: "object", required: false, description: "Cron schedule: { cron: string, timezone?: string (default UTC), enabled?: boolean (default true) }" },
+        { name: "storage_type", type: "string", required: false, description: "managed (default) | git" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const updateSuiteSchema = {
@@ -242,10 +265,13 @@ const updateSuiteSchema = {
     fields: [
         { name: "name", type: "string", required: false, description: "Name" },
         { name: "description", type: "string", required: false, description: "Description" },
-        { name: "purpose", type: "string", required: false, description: "Purpose" },
+        { name: "purpose", type: "string", required: false, description: "pr_gate | cd_gate | release_gate | custom" },
         { name: "pass_strategy", type: "string", required: false, description: "all_must_pass | weighted_threshold" },
         { name: "pass_threshold", type: "number", required: false, description: "Pass threshold 0-1" },
         { name: "is_blocking", type: "boolean", required: false, description: "Blocking suite" },
+        { name: "schedule", type: "object", required: false, description: "Cron schedule: { cron, timezone?, enabled? } — set null to remove" },
+        { name: "storage_type", type: "string", required: false, description: "managed | git (switches storage mode)" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when switching to storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const addSuiteEntrySchema = {
@@ -272,8 +298,9 @@ const triggerSuiteRunSchema = {
     description: "Trigger suite run",
     fields: [
         { name: "triggered_by", type: "string", required: false, description: "Who triggered the run" },
-        { name: "trigger_type", type: "string", required: false, description: "manual | ci | ... (default manual)" },
-        { name: "suite_path", type: "string", required: false, description: "Suite YAML path (git-backed)" },
+        { name: "trigger_type", type: "string", required: false, description: "manual | api | ci | scheduled (default manual)" },
+        { name: "suite_path", type: "string", required: false, description: "Suite YAML path (git-backed, overrides suite.source_path)" },
+        { name: "branch", type: "string", required: false, description: "Override git branch (e.g. run against a PR branch)" },
         {
             name: "run_inputs",
             type: "object",
@@ -290,15 +317,18 @@ const triggerSuiteRunSchema = {
     ],
 };
 const createTargetSchema = {
-    description: "Create target",
+    description: "Create target. For managed: type + config are required. For git-backed: omit type/config and provide storage_type='git' + git_source.",
     fields: [
         { name: "name", type: "string", required: true, description: "Name" },
-        { name: "type", type: "string", required: true, description: "prompt | app | static" },
-        { name: "config", type: "object", required: true, description: "Target config (PromptTargetConfig | AppTargetConfig | StaticTargetConfig)" },
+        { name: "type", type: "string", required: false, description: "prompt | agent | precomputed (required when storage_type='managed', omit for git)" },
+        { name: "config", type: "object", required: false, description: "Target config (required when storage_type='managed', omit for git)" },
         { name: "description", type: "string", required: false, description: "Description" },
         { name: "tags", type: "array", required: false, description: "Tags", itemType: "string" },
         { name: "is_active", type: "boolean", required: false, description: "Active (default true)" },
         { name: "env_secrets", type: "object", required: false, description: "Env var to Harness secret ref mapping" },
+        { name: "connector_ref", type: "string", required: false, description: "Harness HTTP connector for endpoint configuration" },
+        { name: "storage_type", type: "string", required: false, description: "managed (default) | git" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const updateTargetSchema = {
@@ -306,15 +336,22 @@ const updateTargetSchema = {
     fields: [
         { name: "name", type: "string", required: false, description: "Name" },
         { name: "description", type: "string", required: false, description: "Description" },
+        { name: "type", type: "string", required: false, description: "prompt | agent | precomputed" },
         { name: "config", type: "object", required: false, description: "Config" },
         { name: "tags", type: "array", required: false, description: "Tags", itemType: "string" },
         { name: "is_active", type: "boolean", required: false, description: "Active" },
         { name: "env_secrets", type: "object", required: false, description: "Env var to Harness secret ref mapping" },
+        { name: "connector_ref", type: "string", required: false, description: "Harness HTTP connector for endpoint configuration" },
+        { name: "storage_type", type: "string", required: false, description: "managed | git (switches storage mode)" },
+        { name: "git_source", type: "object", required: false, description: "Git location (required when switching to storage_type='git'): { connector_ref?, repo?, branch?, file_path }" },
     ],
 };
 const testTargetSchema = {
     description: "Test target invocation",
-    fields: [{ name: "input", type: "string", required: true, description: "Sample input string" }],
+    fields: [
+        { name: "input", type: "string", required: true, description: "Sample input string" },
+        { name: "item_identifier", type: "string", required: false, description: "Dataset item identifier (used by precomputed targets to look up output)" },
+    ],
 };
 const uploadOutputsSchema = {
     description: "Upload static target outputs",
@@ -332,10 +369,11 @@ const createModelSchema = {
     description: "Register AI model",
     fields: [
         { name: "name", type: "string", required: true, description: "Display name" },
-        { name: "provider", type: "string", required: true, description: "openai | anthropic | ..." },
+        { name: "provider", type: "string", required: true, description: "openai | anthropic | google | azure | custom" },
         { name: "model_id", type: "string", required: true, description: "Provider model id" },
         { name: "description", type: "string", required: false, description: "Description" },
         { name: "api_key_secret_ref", type: "string", required: false, description: "Harness secret ref for API key" },
+        { name: "connector_ref", type: "string", required: false, description: "Harness connector identifier for LLM credentials" },
         { name: "default_temperature", type: "number", required: false, description: "Default temperature 0-2" },
         { name: "default_max_tokens", type: "number", required: false, description: "Default max tokens (min 1)" },
         { name: "default_top_p", type: "number", required: false, description: "Default top_p 0-1" },
@@ -349,6 +387,7 @@ const updateModelSchema = {
     fields: [
         { name: "name", type: "string", required: false, description: "Name" },
         { name: "description", type: "string", required: false, description: "Description" },
+        { name: "connector_ref", type: "string", required: false, description: "Harness connector identifier for LLM credentials" },
         { name: "default_temperature", type: "number", required: false, description: "Temperature 0-2" },
         { name: "default_max_tokens", type: "number", required: false, description: "Max tokens (min 1)" },
         { name: "default_top_p", type: "number", required: false, description: "Top_p 0-1" },
@@ -465,6 +504,12 @@ const importSuiteYamlSchema = {
         { name: "dry_run", type: "boolean", required: false, description: "If true, validate the YAML without creating any entities (default false)" },
     ],
 };
+const bulkUpsertDatasetItemsSchema = {
+    description: "Bulk upsert dataset items (insert or update by business ID)",
+    fields: [
+        { name: "items", type: "array", required: true, description: "Dataset items to upsert (CreateDatasetItemRequest[])", itemType: "object" },
+    ],
+};
 /** Merge harness_execute `body` into JSON POST body */
 function bodyFromInput(input) {
     const b = input.body;
@@ -480,19 +525,21 @@ export const aiEvalsToolset = {
     name: "ai-evals",
     displayName: "AI Evals",
     description: "Harness AI Evals control plane: datasets, evaluations, runs, metrics, metric sets, suites, targets, models, annotations, analytics, registry, git settings.",
-    optIn: true,
+    optIn: false,
     resources: [
         // --- Datasets ---
         {
             resourceType: "eval_dataset",
             displayName: "AI Evals Dataset",
-            description: "Evaluation dataset (JSONL-backed rows). CRUD + items sub-resource via eval_dataset_item.",
+            description: "Evaluation dataset (managed JSONL rows or git-backed). CRUD + items sub-resource via eval_dataset_item.",
             toolset: "ai-evals",
             scope: "project",
             scopeOptional: true,
             headerBasedScoping: true,
             identifierFields: ["dataset_id"],
-            listFilterFields: [],
+            listFilterFields: [
+                { name: "search", description: "Search by name, identifier, or description" },
+            ],
             relatedResources: [
                 { resourceType: "eval_dataset_item", relationship: "contains", description: "Dataset rows" },
                 { resourceType: "evaluation", relationship: "uses", description: "Evals reference datasets" },
@@ -503,7 +550,7 @@ export const aiEvalsToolset = {
                     path: "",
                     pathBuilder: (input, config) => `${base(input, config)}/dataset`,
                     operationPolicy: { risk: "read", retryPolicy: "safe" },
-                    queryParams: listQ,
+                    queryParams: { ...listQ, search: "search" },
                     responseExtractor: aiEvalsListExtract,
                     description: "List datasets",
                 },
@@ -645,6 +692,18 @@ export const aiEvalsToolset = {
                     description: "Delete item",
                 },
             },
+            executeActions: {
+                bulk_upsert: {
+                    method: "PATCH",
+                    path: "",
+                    pathBuilder: (input, config) => `${base(input, config)}/dataset/${input.dataset_id}/items/bulk`,
+                    operationPolicy: { risk: "low_write", retryPolicy: "do_not_retry" },
+                    bodyBuilder: bodyFromInput,
+                    bodySchema: bulkUpsertDatasetItemsSchema,
+                    responseExtractor: aiEvalsArrayExtract,
+                    actionDescription: "Bulk upsert dataset items by business ID. Body: { items: CreateDatasetItemRequest[] }",
+                },
+            },
         },
         // --- Evaluations ---
         {
@@ -662,6 +721,9 @@ export const aiEvalsToolset = {
                     description: "Filter by status",
                     enum: ["active", "draft", "archived"],
                 },
+                { name: "target_id", description: "Filter by target UUID(s)" },
+                { name: "metric_set_id", description: "Filter by metric set UUID" },
+                { name: "search", description: "Search by name or description" },
             ],
             executeHint: "Run an eval with harness_execute(resource_type='evaluation', action='run', resource_id=EVAL_ID, body={...}).",
             operations: {
@@ -670,7 +732,7 @@ export const aiEvalsToolset = {
                     path: "",
                     pathBuilder: (input, config) => `${base(input, config)}/evals`,
                     operationPolicy: { risk: "read", retryPolicy: "safe" },
-                    queryParams: { ...listQ, status: "status" },
+                    queryParams: { ...listQ, status: "status", target_id: "target_id", metric_set_id: "metric_set_id", search: "search" },
                     responseExtractor: aiEvalsListExtract,
                     description: "List evals",
                 },
@@ -747,15 +809,14 @@ export const aiEvalsToolset = {
         {
             resourceType: "eval_run",
             displayName: "AI Evals Run",
-            description: "A single evaluation run. Compare runs or rescore via execute actions.",
+            description: "A single evaluation run. Compare runs or rescore via execute actions. To filter by eval_id, use the eval_run_by_eval resource instead.",
             toolset: "ai-evals",
             scope: "project",
             scopeOptional: true,
             headerBasedScoping: true,
             identifierFields: ["run_id"],
             listFilterFields: [
-                { name: "eval_id", description: "Filter runs by evaluation UUID" },
-                { name: "status", description: "Filter by run status" },
+                { name: "target_id", description: "Filter runs by target UUID" },
             ],
             relatedResources: [
                 { resourceType: "eval_run_item", relationship: "contains", description: "Per-item results" },
@@ -766,9 +827,9 @@ export const aiEvalsToolset = {
                     path: "",
                     pathBuilder: (input, config) => `${base(input, config)}/runs`,
                     operationPolicy: { risk: "read", retryPolicy: "safe" },
-                    queryParams: { ...listQ, eval_id: "eval_id", status: "status" },
+                    queryParams: { ...listQ, target_id: "target_id" },
                     responseExtractor: aiEvalsListExtract,
-                    description: "List runs in project (filterable by eval_id, status)",
+                    description: "List runs in project. Use eval_run_by_eval resource to filter by eval_id.",
                 },
                 get: {
                     method: "GET",
@@ -967,19 +1028,22 @@ export const aiEvalsToolset = {
         {
             resourceType: "eval_metric_set",
             displayName: "AI Evals Metric Set",
-            description: "Grouped metrics with thresholds. Manage entries via eval_metric_set_entry.",
+            description: "Grouped metrics with thresholds and optional judge model. Manage entries via eval_metric_set_entry.",
             toolset: "ai-evals",
             scope: "project",
             scopeOptional: true,
             headerBasedScoping: true,
             identifierFields: ["set_id"],
+            listFilterFields: [
+                { name: "search", description: "Search by name or description" },
+            ],
             operations: {
                 list: {
                     method: "GET",
                     path: "",
                     pathBuilder: (input, config) => `${base(input, config)}/metric-sets`,
                     operationPolicy: { risk: "read", retryPolicy: "safe" },
-                    queryParams: listQ,
+                    queryParams: { ...listQ, search: "search" },
                     responseExtractor: aiEvalsListExtract,
                     description: "List metric sets",
                 },
@@ -1104,7 +1168,7 @@ export const aiEvalsToolset = {
         {
             resourceType: "eval_suite",
             displayName: "AI Evals Suite",
-            description: "Multi-eval suite with pass strategy. Members: eval_suite_evaluation.",
+            description: "Multi-eval suite with pass strategy and optional cron schedule. Members: eval_suite_evaluation.",
             toolset: "ai-evals",
             scope: "project",
             scopeOptional: true,
@@ -1272,20 +1336,23 @@ export const aiEvalsToolset = {
         {
             resourceType: "eval_target",
             displayName: "AI Evals Target",
-            description: "Invocation target (prompt, app, or static).",
+            description: "Invocation target (prompt, agent, or precomputed).",
             toolset: "ai-evals",
             scope: "project",
             scopeOptional: true,
             headerBasedScoping: true,
             identifierFields: ["target_id"],
-            listFilterFields: [{ name: "type", description: "prompt | app | static" }],
+            listFilterFields: [
+                { name: "type", description: "prompt | agent | precomputed" },
+                { name: "search", description: "Search by name or description" },
+            ],
             operations: {
                 list: {
                     method: "GET",
                     path: "",
                     pathBuilder: (input, config) => `${base(input, config)}/targets`,
                     operationPolicy: { risk: "read", retryPolicy: "safe" },
-                    queryParams: { ...listQ, type: "type" },
+                    queryParams: { ...listQ, type: "type", search: "search" },
                     responseExtractor: aiEvalsListExtract,
                     description: "List targets",
                 },
@@ -1357,6 +1424,24 @@ export const aiEvalsToolset = {
                     actionDescription: "List uploaded static target outputs (paginated).",
                     bodySchema: { description: "No body", fields: [] },
                 },
+                export_yaml: {
+                    method: "GET",
+                    path: "",
+                    pathBuilder: (input, config) => `${base(input, config)}/targets/${input.target_id}/export-yaml`,
+                    operationPolicy: { risk: "read", retryPolicy: "safe" },
+                    responseExtractor: passthrough,
+                    actionDescription: "Export target config as a standalone YAML document.",
+                    bodySchema: { description: "No body", fields: [] },
+                },
+                overview: {
+                    method: "GET",
+                    path: "",
+                    pathBuilder: (input, config) => `${base(input, config)}/targets/${input.target_id}/overview`,
+                    operationPolicy: { risk: "read", retryPolicy: "safe" },
+                    responseExtractor: passthrough,
+                    actionDescription: "Summary metrics and per-eval health trend (total_evals, total_runs, last_run_at, overall_pass_rate, per-eval pass rates).",
+                    bodySchema: { description: "No body", fields: [] },
+                },
             },
         },
         {
@@ -1368,14 +1453,17 @@ export const aiEvalsToolset = {
             scopeOptional: true,
             headerBasedScoping: true,
             identifierFields: ["model_id"],
-            listFilterFields: [{ name: "active_only", description: "Only active models", type: "boolean" }],
+            listFilterFields: [
+                { name: "active_only", description: "Only active models", type: "boolean" },
+                { name: "search", description: "Search by name, provider, or model ID" },
+            ],
             operations: {
                 list: {
                     method: "GET",
                     path: "",
                     pathBuilder: (input, config) => `${base(input, config)}/models`,
                     operationPolicy: { risk: "read", retryPolicy: "safe" },
-                    queryParams: { ...listQ, active_only: "active_only" },
+                    queryParams: { ...listQ, active_only: "active_only", search: "search" },
                     responseExtractor: aiEvalsListExtract,
                     description: "List models",
                 },