@microsoft/m365-copilot-eval 1.3.0-preview.1 → 1.4.0-preview.1

package/README.md

@@ -5,21 +5,28 @@
 A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to your agent, get responses, and automatically score them with Azure AI Evaluation metrics (relevance, coherence, groundedness).
 - Send a batch (or interactive set) of prompts to a configured chat API endpoint.
 - Collect agent responses and evaluate them locally using Azure AI Evaluation SDK.
-- Metrics produced per prompt:
-- - Relevance (1–5)
-- - Coherence (1–5)
-- - Groundedness (1–5)
-- - Tool Call Accuracy (1–5)
-- - Citations (0–1)
+- The CLI supports 7 evaluator types. Evaluators marked with ⭐ are **enabled by default**.
+
+| Evaluator | Type | Scale | Default Threshold | Default |
+|-----------|------|-------|-------------------|---------|
+| **Relevance** ⭐ | LLM-based | 1-5 | 3 | Yes |
+| **Coherence** ⭐ | LLM-based | 1-5 | 3 | Yes |
+| **Groundedness** | LLM-based | 1-5 | 3 | No |
+| **ToolCallAccuracy** | LLM-based | 1-5 | 3 | No |
+| **Citations** | Count-based | >= 0 | 1 | No |
+| **ExactMatch** | String match | boolean | N/A | No |
+| **PartialMatch** | String match | 0.0-1.0 | 0.5 | No |
 - Multiple input modes: command‑line list, JSON file, interactive.
 - Multiple output formats: console (colorized), JSON, CSV, HTML (auto‑opens report).
 
 ## 📋 Prerequisites
 
+- **M365 Copilot License** for your tenant
 - **M365 Copilot Agent** deployed to your tenant (can be created with [M365 Agents Toolkit](https://learn.microsoft.com/en-us/microsoft-365/developer/overview-m365-agents-toolkit) or any other method)
 - **Node.js 24.12.0+** (check: `node --version`)
 - **Environment file** with your credentials and agent ID (see [Environment Setup](#-environment-setup) below)
-- **Your Tenant ID, Azure OpenAI endpoint, and API key** (see [Getting Variables](#-getting-variables) below)
+- **Your Tenant ID** - get your tenant id using the instructions [here](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id) 
+- **Azure OpenAI endpoint, and API key** (see [Getting Variables](#-getting-variables) below)
 
 > Note: Authentication is currently supported on Windows only. Support for other operating systems is coming soon.
 
@@ -123,16 +130,29 @@ You need both the endpoint URL and API key from your Azure OpenAI resource for "
 **How to obtain:**
 
 1. Go to [Azure Portal](https://portal.azure.com)
-2. Navigate to your Azure OpenAI service
-   - **Path:** Portal → All Services → Search "OpenAI" → Select your resource
-   - **Or create new:** Portal → Create a resource → Search "OpenAI"
-3. In the **Overview** section, copy the **Endpoint** value
-   - Format: `https://YOUR-RESOURCE-NAME.openai.azure.com/`
-   - This is your `AZURE_AI_OPENAI_ENDPOINT`
-4. In the left sidebar, click **Keys and Endpoint**
-5. Copy **KEY 1** or **KEY 2**
-   - This is your `AZURE_AI_API_KEY`
-6. Add both values to your `.env.dev` file as shown in the [Setup Steps](#setup-steps) above
+2. Open Azure Portal. Search OpenAI in the search bar and select Azure OpenAi. 
+![Azure Portal search bar showing Azure OpenAI service](docs/images/image.png)
+3. once you select Azure OpenAi, then Create an AI Foundry Resource. 
+![Azure OpenAI service page with Create AI Foundry Resource button](docs/images/image-1.png)
+4. On the Create Foundry Resource, fill in the details and click 'Review + Create'.
+![Create AI Foundry Resource form with Review + Create button](docs/images/image-2.png)
+5. Once the resource deployed, go to foundry portal 
+![Resource deployment complete with link to AI Foundry portal](docs/images/image-3.png)
+6. At this point, you should be able to deploy an LLM model. 
+7. Select Models + Endpoints on the left rail
+![AI Foundry portal left navigation with Models + Endpoints selected](docs/images/image-4.png)
+8. Select Deploy Model -> Deploy base model (we recommend gpt-4o-mini model)
+![Deploy Model dropdown showing Deploy base model option](docs/images/image-5.png)
+9. Select Confirm, then select Customize
+![Model deployment confirmation dialog with Customize button](docs/images/image-6.png)
+10. Click on Customize and change the capacity to 50K tokens per minute
+![Model deployment customization showing token capacity setting](docs/images/image-7.png)
+![Token capacity set to 50K tokens per minute](docs/images/image-8.png)
+11. Hit deploy and wait for a few minutes for the model to deploy.
+12. Once the deployment finishes, you are redirected to the API endpoint and API_Key page.
+13. Copy the following values from that page.
+![API endpoint and API key values on the model deployment page](docs/images/image-10.png)
+14. Add all of these values to your `.env.dev` file as shown in the [Setup Steps](#setup-steps) above
 
 **Required model:** Ensure you have `gpt-4o-mini` (or similar) deployed in your Azure OpenAI resource.
 
@@ -165,64 +185,110 @@ runevals --env dev
 
 ---
 
-## 📝 Creating Prompts Files
+## 📝 Eval Document Format
 
-The CLI auto-discovers prompts files in your project:
+The eval document schema is versioned independently from the CLI, following [Semantic Versioning](https://semver.org/).
 
-### Auto-Discovery
+- **Schema location**: [`schema/v1/eval-document.schema.json`](schema/v1/eval-document.schema.json)
+- **Schema changelog**: [`schema/CHANGELOG.md`](schema/CHANGELOG.md)
+
+> **New in Schema v1.2.0**: Multi-turn conversation threads — test context persistence across multiple turns within a shared conversation session. Each thread supports 1-20 turns.
+
+> **New in Schema v1.1.0**: Per-prompt evaluator overrides with `evaluators_mode` (`extend`/`replace`), file-level `default_evaluators`, and `ExactMatch`/`PartialMatch` evaluators.
+
+### Getting Started
 
-When you run `runevals`, it searches:
+The CLI auto-discovers prompts files in your project. When you run `runevals`, it searches:
 1. Current directory: `prompts.json`, `evals.json`, `tests.json`
 2. `./evals/` subdirectory: `prompts.json`, `evals.json`, `tests.json`
 
-**Example project structure:**
-```
-my-agent/
-├── .env.local              # Your credentials
-├── evals/
-│   └── evals.json         # Your test prompts (auto-discovered!)
-└── .evals/
-    └── 2025-12-03_14-30-45.html  # Generated reports
-```
+**No prompts file?** The CLI will offer to create a starter file with example prompts for you.
 
-### Starter File Creation
+A minimal eval document:
 
-If no file is found:
+```json
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "prompt": "What is Microsoft 365?",
+      "expected_response": "Microsoft 365 is a cloud-based productivity suite..."
+    }
+  ]
+}
 ```
-⚠️  No prompts file found in current directory or ./evals/
 
-Create a starter evals file with sample prompts? (Y/n):
-```
+### Evaluator Configuration
 
-Answering "Y" creates `./evals/evals.json` with 2 starter prompts:
+Use `default_evaluators` to set file-level defaults, and per-item `evaluators` with `evaluators_mode` to customize:
 
 ```json
-[
-  {
-    "prompt": "What is Microsoft 365?",
-    "expected_response": "Microsoft 365 is a cloud-based productivity suite..."
+{
+  "schemaVersion": "1.2.0",
+  "default_evaluators": {
+    "Relevance": {},
+    "Coherence": {}
   },
-  {
-    "prompt": "How can I share a file in Teams?",
-    "expected_response": "You can share a file in Teams by uploading it..."
-  }
-]
+  "items": [
+    {
+      "prompt": "What is Microsoft Graph?",
+      "expected_response": "A unified API endpoint for Microsoft services.",
+      "evaluators": {
+        "Citations": { "citation_format": "mixed" }
+      },
+      "evaluators_mode": "extend"
+    },
+    {
+      "name": "Expense policy flow",
+      "turns": [
+        {
+          "prompt": "I spent $250 on dinner. Is that okay?",
+          "expected_response": "The per-diem meal allowance is $200."
+        },
+        {
+          "prompt": "What should I do about the overage?",
+          "expected_response": "Request manager approval.",
+          "evaluators": {
+            "ExactMatch": { "case_sensitive": false }
+          },
+          "evaluators_mode": "replace"
+        }
+      ]
+    }
+  ]
+}
 ```
 
-Edit this file with your own prompts and run again!
+**How evaluator modes work in this example:**
 
-### Manual Creation
+| Item | `evaluators_mode` | Active Evaluators | Why |
+|------|-------------------|-------------------|-----|
+| Single-turn (Graph) | `extend` | Relevance, Coherence, Citations | Per-prompt Citations **merged** with defaults |
+| Multi-turn turn 1 (dinner) | _(none)_ | Relevance, Coherence | **Inherits** file-level defaults |
+| Multi-turn turn 2 (overage) | `replace` | ExactMatch | Per-turn ExactMatch **replaces** defaults entirely |
 
-Create `./evals/prompts.json`:
+### Evaluator Modes
 
-```json
-[
-  {
-    "prompt": "Your test prompt here",
-    "expected_response": "Expected agent response"
-  }
-]
-```
+| Mode | Behavior |
+|------|----------|
+| `"extend"` (default) | Per-item evaluators **merge** with defaults. Both run. |
+| `"replace"` | Per-item evaluators **replace** defaults entirely. Only per-item evaluators run. |
+| _(none)_ | Inherits file-level `default_evaluators`, or system defaults (Relevance, Coherence) if not set. |
+
+See `schema/v1/examples/` in the package for more examples including per-turn evaluator overrides, mixed single/multi-turn files, and output format.
+
+### Auto-Upgrade Behavior
+
+When the CLI loads an eval document:
+
+- **Legacy documents** (missing `schemaVersion`): Automatically upgraded with a timestamped backup (e.g., `file.json.bak.20260205143052`)
+- **Older versions** (same major version): `schemaVersion` field updated without backup
+- **Invalid documents**: CLI exits with an error message and guidance to review the schema changelog
+- **Future versions**: CLI rejects with a message suggesting a CLI update
+
+### Version Compatibility
+
+Within a major version (e.g., 1.x.x), we aim to maintain backward compatibility for documents that conform to the published schema for their version. Compatibility does not extend to undeclared or ad-hoc fields outside the schema definition; review the [schema changelog](schema/CHANGELOG.md) when upgrading between minor versions.
 
 ## 🎯 Usage Examples
 
@@ -271,6 +337,10 @@ runevals --log-level info
 runevals --log-level warning
 runevals --log-level error
 
+# Parallel prompt execution control
+runevals --concurrency 5 --prompts-file ./evals/evals.json
+runevals --concurrency 1000 --prompts-file ./evals/evals.json   # Python CLI clamps to 5
+
 # Custom output location in your project
 runevals --output ./reports/results.html
 ```
@@ -309,10 +379,9 @@ npm run eval:dev
 ## 📊 Output Formats
 
 Results are automatically saved to `./evals/YYYY-MM-DD_HH-MM-SS.html` with:
-- **Relevance** score (1-5)
-- **Coherence** score (1-5)  
-- **Groundedness** score (1-5)
-- Per-prompt details and aggregate metrics
+- Per-prompt and per-turn evaluation scores from configured evaluators
+- Aggregate statistics across all evaluated items
+- Multi-turn thread summaries (turns passed/failed, overall status)
 
 Other formats:
 ```bash
@@ -410,43 +479,6 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
 
-## Schema Versioning
-
-The eval document schema is versioned independently from the CLI, following [Semantic Versioning](https://semver.org/). This allows external consumers to depend on a stable contract without coupling to CLI release cycles.
-
-- **Schema location**: [`schema/v1/eval-document.schema.json`](schema/v1/eval-document.schema.json)
-- **Schema changelog**: [`schema/CHANGELOG.md`](schema/CHANGELOG.md)
-- **Consumer quickstart**: [`specs/wi-6081652-dataset-schema-versioning/quickstart.md`](specs/wi-6081652-dataset-schema-versioning/quickstart.md)
-
-### Eval Document Format
-
-Eval documents should include a `schemaVersion` field:
-
-```json
-{
-  "schemaVersion": "1.0.0",
-  "items": [
-    {
-      "prompt": "What is Microsoft 365?",
-      "expected_response": "Microsoft 365 is a cloud-based productivity suite."
-    }
-  ]
-}
-```
-
-### Auto-Upgrade Behavior
-
-When the CLI loads an eval document:
-
-- **Legacy documents** (missing `schemaVersion`): Automatically upgraded with a timestamped backup (e.g., `file.json.bak.20260205143052`)
-- **Older versions** (same major version): `schemaVersion` field updated without backup
-- **Invalid documents**: CLI exits with an error message and guidance to review the schema changelog
-- **Future versions**: CLI rejects with a message suggesting a CLI update
-
-### Version Compatibility
-
-Within a major version (e.g., 1.x.x), backward compatibility is guaranteed. Documents valid against 1.0.0 will remain valid against 1.1.0, 1.2.0, etc.
-
 ## Trademarks
 
 This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft

package/package.json

@@ -1,9 +1,9 @@
 {
   "name": "@microsoft/m365-copilot-eval",
-  "version": "1.3.0-preview.1",
+  "version": "1.4.0-preview.1",
   "minCliVersion": "1.0.1-preview.1",
   "description": "Zero-config Node.js wrapper for M365 Copilot Agent Evaluations CLI (Python-based Azure AI Evaluation SDK)",
-  "publishDate": "2026-04-01",
+  "publishDate": "2026-04-22",
   "main": "src/clients/node-js/lib/index.js",
   "type": "module",
   "bin": {
@@ -14,6 +14,7 @@
     "build": "npm run prettier:check && npm run clean && npm run lint",
     "clean": "rimraf node_modules/.cache dist coverage",
     "test": "node --test tests/clients/node-js/**/*.test.js",
+    "test:coverage": "c8 --reporter=cobertura --reporter=text node --test tests/clients/node-js/**/*.test.js",
     "install-credprovider": "artifacts-npm-credprovider && npm ci",
     "set-publish-date": "node scripts/set-publish-date.js",
     "prepublishOnly": "node scripts/set-publish-date.js",
@@ -41,9 +42,10 @@
   },
   "dependencies": {
     "commander": "^12.1.0",
+    "dotenv": "^16.0.0",
+    "https-proxy-agent": "^7.0.5",
     "node-fetch": "^3.3.2",
-    "tar": "^7.5.4",
-    "https-proxy-agent": "^7.0.5"
+    "tar": "^7.5.4"
   },
   "devDependencies": {
     "@microsoft/eslint-config-msgraph": "^5.0.0",
@@ -52,6 +54,7 @@
     "@vitest/coverage-istanbul": "^3.0.0",
     "@vitest/coverage-v8": "^3.0.0",
     "@vitest/ui": "^3.0.0",
+    "c8": "^11.0.0",
     "eslint": "^9.7.0",
     "eslint-config-prettier": "^10.0.0",
     "eslint-plugin-jsdoc": "^50.1.0",

package/schema/v1/eval-document.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://raw.githubusercontent.com/microsoft/M365-Copilot-Agent-Evals/refs/heads/main/schema/v1/eval-document.schema.json",
   "title": "M365 Copilot Eval Document",
-  "description": "Schema for evaluation documents used by M365 Copilot Agent Evals CLI. Version 1.1.0.",
+  "description": "Schema for evaluation documents used by M365 Copilot Agent Evals CLI. Supports single-turn and multi-turn evaluations.",
   "type": "object",
   "required": ["schemaVersion", "items"],
   "additionalProperties": true,
@@ -28,9 +28,12 @@
     "items": {
       "type": "array",
       "minItems": 1,
-      "description": "Array of evaluation items (prompts and optionally responses with scores)",
+      "description": "Array of evaluation items: single-turn evaluations or multi-turn threads",
       "items": {
-        "$ref": "#/$defs/EvalItem"
+        "oneOf": [
+          { "$ref": "#/$defs/SingleTurnEvaluation" },
+          { "$ref": "#/$defs/MultiTurnThread" }
+        ]
       }
     }
   },
@@ -60,7 +63,7 @@
         "evaluatedAt": {
           "type": "string",
           "format": "date-time",
-          "description": "ISO 8601 timestamp when evaluation was performed (output documents)"
+          "description": "ISO 8601 timestamp when evaluation was performed"
         },
         "tags": {
           "type": "array",
@@ -88,11 +91,11 @@
         }
       }
     },
-    "EvalItem": {
+    "SingleTurnEvaluation": {
       "type": "object",
-      "description": "A single evaluation item containing a prompt and optionally a response with scores",
+      "description": "A standalone single-turn prompt-response evaluation",
       "required": ["prompt"],
-      "additionalProperties": true,
+      "additionalProperties": false,
       "properties": {
         "prompt": {
           "type": "string",
@@ -105,7 +108,7 @@
         },
         "response": {
           "type": "string",
-          "description": "Actual response from the agent (present in output documents)"
+          "description": "Actual response from the agent"
         },
         "context": {
           "type": "string",
@@ -136,6 +139,135 @@
           "additionalProperties": true,
           "description": "Extension point for custom item-level fields"
         }
+      },
+      "not": {
+        "required": ["turns"]
+      }
+    },
+    "MultiTurnThread": {
+      "type": "object",
+      "description": "A multi-turn conversation thread with ordered turns sharing conversation context",
+      "required": ["turns"],
+      "additionalProperties": false,
+      "properties": {
+        "name": {
+          "type": "string",
+          "description": "Human-readable name for the thread"
+        },
+        "description": {
+          "type": "string",
+          "description": "Description of what this thread tests"
+        },
+        "turns": {
+          "type": "array",
+          "minItems": 1,
+          "maxItems": 20,
+          "items": { "$ref": "#/$defs/Turn" },
+          "description": "Ordered array of conversation turns"
+        },
+        "conversation_id": {
+          "type": "string",
+          "description": "Unique identifier for this conversation thread"
+        },
+        "summary": {
+          "$ref": "#/$defs/ThreadSummary",
+          "description": "Aggregate statistics for the thread"
+        },
+        "extensions": {
+          "type": "object",
+          "additionalProperties": true,
+          "description": "Extension point for custom thread-level fields"
+        }
+      },
+      "not": {
+        "required": ["prompt"]
+      }
+    },
+    "Turn": {
+      "type": "object",
+      "description": "A single turn within a multi-turn thread",
+      "required": ["prompt"],
+      "additionalProperties": false,
+      "properties": {
+        "prompt": {
+          "type": "string",
+          "minLength": 1,
+          "description": "The user message for this turn"
+        },
+        "expected_response": {
+          "type": "string",
+          "description": "Expected agent response for this turn"
+        },
+        "response": {
+          "type": "string",
+          "description": "Actual agent response"
+        },
+        "context": {
+          "type": "string",
+          "description": "Additional context for grounding evaluation"
+        },
+        "evaluators": {
+          "$ref": "#/$defs/EvaluatorMap",
+          "description": "Per-turn evaluator overrides"
+        },
+        "evaluators_mode": {
+          "type": "string",
+          "enum": ["extend", "replace"],
+          "default": "extend",
+          "description": "How per-turn evaluators combine with defaults"
+        },
+        "citations": {
+          "type": "array",
+          "items": {
+            "$ref": "#/$defs/Citation"
+          },
+          "description": "Citations included in the response"
+        },
+        "scores": {
+          "$ref": "#/$defs/ScoreCollection"
+        },
+        "status": {
+          "type": "string",
+          "enum": ["pass", "fail", "error"],
+          "description": "Overall status of this turn"
+        },
+        "error": {
+          "type": "string",
+          "description": "Error message if status is 'error'"
+        },
+        "extensions": {
+          "type": "object",
+          "additionalProperties": true,
+          "description": "Extension point for custom turn-level fields"
+        }
+      }
+    },
+    "ThreadSummary": {
+      "type": "object",
+      "description": "Aggregate statistics for a thread",
+      "required": ["turns_total", "turns_passed", "turns_failed", "overall_status"],
+      "additionalProperties": false,
+      "properties": {
+        "turns_total": {
+          "type": "integer",
+          "minimum": 1,
+          "description": "Total number of turns executed"
+        },
+        "turns_passed": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "Number of turns where all evaluators passed"
+        },
+        "turns_failed": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "Number of turns where any evaluator failed"
+        },
+        "overall_status": {
+          "type": "string",
+          "enum": ["pass", "partial", "fail"],
+          "description": "pass: all turns passed, partial: some failed, fail: all failed or error"
+        }
       }
     },
     "ScoreCollection": {

package/schema/v1/examples/invalid/multi-turn-empty-turns.json

@@ -0,0 +1,8 @@
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "turns": []
+    }
+  ]
+}

package/schema/v1/examples/invalid/multi-turn-has-both-prompt-and-turns.json

@@ -0,0 +1,13 @@
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "prompt": "This item has both prompt and turns",
+      "turns": [
+        {
+          "prompt": "Turn 1"
+        }
+      ]
+    }
+  ]
+}

package/schema/v1/examples/invalid/multi-turn-missing-prompt.json

@@ -0,0 +1,12 @@
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "turns": [
+        {
+          "expected_response": "This turn is missing a prompt."
+        }
+      ]
+    }
+  ]
+}

package/schema/v1/examples/invalid/multi-turn-typo-in-turn.json

@@ -0,0 +1,13 @@
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "turns": [
+        {
+          "prompt": "Hello",
+          "expeceted_response": "Typo in field name"
+        }
+      ]
+    }
+  ]
+}

package/schema/v1/examples/invalid/multi-turn-unknown-evaluator.json

@@ -0,0 +1,15 @@
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "turns": [
+        {
+          "prompt": "Hello",
+          "evaluators": {
+            "TaskCompletionEvaluator": {}
+          }
+        }
+      ]
+    }
+  ]
+}

package/schema/v1/examples/valid/mixed-single-and-multi-turn.json

@@ -0,0 +1,30 @@
+{
+  "schemaVersion": "1.2.0",
+  "default_evaluators": {
+    "Relevance": {},
+    "Coherence": {}
+  },
+  "items": [
+    {
+      "prompt": "What is Microsoft Graph API?",
+      "expected_response": "Microsoft Graph API is a unified endpoint for accessing Microsoft services."
+    },
+    {
+      "name": "Canadian Employee HR Inquiry",
+      "turns": [
+        {
+          "prompt": "I'm a Canadian employee based in Toronto.",
+          "expected_response": "Got it! I can help with Canada-specific HR questions."
+        },
+        {
+          "prompt": "Is July 4th a holiday for me?",
+          "expected_response": "July 4th is not a statutory holiday in Canada. However, July 1st (Canada Day) is."
+        }
+      ]
+    },
+    {
+      "prompt": "How do I authenticate with Microsoft Graph?",
+      "expected_response": "You can authenticate using OAuth 2.0 or client credentials flow."
+    }
+  ]
+}