@microsoft/m365-copilot-eval 1.2.1-preview.1 → 1.4.0-preview.1

package/README.md

@@ -5,21 +5,28 @@
 A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to your agent, get responses, and automatically score them with Azure AI Evaluation metrics (relevance, coherence, groundedness).
 - Send a batch (or interactive set) of prompts to a configured chat API endpoint.
 - Collect agent responses and evaluate them locally using Azure AI Evaluation SDK.
-- Metrics produced per prompt:
-- - Relevance (1–5)
-- - Coherence (1–5)
-- - Groundedness (1–5)
-- - Tool Call Accuracy (1–5)
-- - Citations (0–1)
+- The CLI supports 7 evaluator types. Evaluators marked with ⭐ are **enabled by default**.
+
+| Evaluator | Type | Scale | Default Threshold | Default |
+|-----------|------|-------|-------------------|---------|
+| **Relevance** ⭐ | LLM-based | 1-5 | 3 | Yes |
+| **Coherence** ⭐ | LLM-based | 1-5 | 3 | Yes |
+| **Groundedness** | LLM-based | 1-5 | 3 | No |
+| **ToolCallAccuracy** | LLM-based | 1-5 | 3 | No |
+| **Citations** | Count-based | >= 0 | 1 | No |
+| **ExactMatch** | String match | boolean | N/A | No |
+| **PartialMatch** | String match | 0.0-1.0 | 0.5 | No |
 - Multiple input modes: command‑line list, JSON file, interactive.
 - Multiple output formats: console (colorized), JSON, CSV, HTML (auto‑opens report).
 
 ## 📋 Prerequisites
 
+- **M365 Copilot License** for your tenant
 - **M365 Copilot Agent** deployed to your tenant (can be created with [M365 Agents Toolkit](https://learn.microsoft.com/en-us/microsoft-365/developer/overview-m365-agents-toolkit) or any other method)
 - **Node.js 24.12.0+** (check: `node --version`)
 - **Environment file** with your credentials and agent ID (see [Environment Setup](#-environment-setup) below)
-- **Your Tenant ID, Azure OpenAI endpoint, and API key** (see [Getting Variables](#-getting-variables) below)
+- **Your Tenant ID** - get your tenant id using the instructions [here](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id) 
+- **Azure OpenAI endpoint, and API key** (see [Getting Variables](#-getting-variables) below)
 
 > Note: Authentication is currently supported on Windows only. Support for other operating systems is coming soon.
 
@@ -123,16 +130,29 @@ You need both the endpoint URL and API key from your Azure OpenAI resource for "
 **How to obtain:**
 
 1. Go to [Azure Portal](https://portal.azure.com)
-2. Navigate to your Azure OpenAI service
-   - **Path:** Portal → All Services → Search "OpenAI" → Select your resource
-   - **Or create new:** Portal → Create a resource → Search "OpenAI"
-3. In the **Overview** section, copy the **Endpoint** value
-   - Format: `https://YOUR-RESOURCE-NAME.openai.azure.com/`
-   - This is your `AZURE_AI_OPENAI_ENDPOINT`
-4. In the left sidebar, click **Keys and Endpoint**
-5. Copy **KEY 1** or **KEY 2**
-   - This is your `AZURE_AI_API_KEY`
-6. Add both values to your `.env.dev` file as shown in the [Setup Steps](#setup-steps) above
+2. Open Azure Portal. Search OpenAI in the search bar and select Azure OpenAi. 
+![Azure Portal search bar showing Azure OpenAI service](docs/images/image.png)
+3. once you select Azure OpenAi, then Create an AI Foundry Resource. 
+![Azure OpenAI service page with Create AI Foundry Resource button](docs/images/image-1.png)
+4. On the Create Foundry Resource, fill in the details and click 'Review + Create'.
+![Create AI Foundry Resource form with Review + Create button](docs/images/image-2.png)
+5. Once the resource deployed, go to foundry portal 
+![Resource deployment complete with link to AI Foundry portal](docs/images/image-3.png)
+6. At this point, you should be able to deploy an LLM model. 
+7. Select Models + Endpoints on the left rail
+![AI Foundry portal left navigation with Models + Endpoints selected](docs/images/image-4.png)
+8. Select Deploy Model -> Deploy base model (we recommend gpt-4o-mini model)
+![Deploy Model dropdown showing Deploy base model option](docs/images/image-5.png)
+9. Select Confirm, then select Customize
+![Model deployment confirmation dialog with Customize button](docs/images/image-6.png)
+10. Click on Customize and change the capacity to 50K tokens per minute
+![Model deployment customization showing token capacity setting](docs/images/image-7.png)
+![Token capacity set to 50K tokens per minute](docs/images/image-8.png)
+11. Hit deploy and wait for a few minutes for the model to deploy.
+12. Once the deployment finishes, you are redirected to the API endpoint and API_Key page.
+13. Copy the following values from that page.
+![API endpoint and API key values on the model deployment page](docs/images/image-10.png)
+14. Add all of these values to your `.env.dev` file as shown in the [Setup Steps](#setup-steps) above
 
 **Required model:** Ensure you have `gpt-4o-mini` (or similar) deployed in your Azure OpenAI resource.
 
@@ -165,64 +185,110 @@ runevals --env dev
 
 ---
 
-## 📝 Creating Prompts Files
+## 📝 Eval Document Format
 
-The CLI auto-discovers prompts files in your project:
+The eval document schema is versioned independently from the CLI, following [Semantic Versioning](https://semver.org/).
 
-### Auto-Discovery
+- **Schema location**: [`schema/v1/eval-document.schema.json`](schema/v1/eval-document.schema.json)
+- **Schema changelog**: [`schema/CHANGELOG.md`](schema/CHANGELOG.md)
+
+> **New in Schema v1.2.0**: Multi-turn conversation threads — test context persistence across multiple turns within a shared conversation session. Each thread supports 1-20 turns.
+
+> **New in Schema v1.1.0**: Per-prompt evaluator overrides with `evaluators_mode` (`extend`/`replace`), file-level `default_evaluators`, and `ExactMatch`/`PartialMatch` evaluators.
+
+### Getting Started
 
-When you run `runevals`, it searches:
+The CLI auto-discovers prompts files in your project. When you run `runevals`, it searches:
 1. Current directory: `prompts.json`, `evals.json`, `tests.json`
 2. `./evals/` subdirectory: `prompts.json`, `evals.json`, `tests.json`
 
-**Example project structure:**
-```
-my-agent/
-├── .env.local              # Your credentials
-├── evals/
-│   └── evals.json         # Your test prompts (auto-discovered!)
-└── .evals/
-    └── 2025-12-03_14-30-45.html  # Generated reports
-```
+**No prompts file?** The CLI will offer to create a starter file with example prompts for you.
 
-### Starter File Creation
+A minimal eval document:
 
-If no file is found:
+```json
+{
+  "schemaVersion": "1.2.0",
+  "items": [
+    {
+      "prompt": "What is Microsoft 365?",
+      "expected_response": "Microsoft 365 is a cloud-based productivity suite..."
+    }
+  ]
+}
 ```
-⚠️  No prompts file found in current directory or ./evals/
 
-Create a starter evals file with sample prompts? (Y/n):
-```
+### Evaluator Configuration
 
-Answering "Y" creates `./evals/evals.json` with 2 starter prompts:
+Use `default_evaluators` to set file-level defaults, and per-item `evaluators` with `evaluators_mode` to customize:
 
 ```json
-[
-  {
-    "prompt": "What is Microsoft 365?",
-    "expected_response": "Microsoft 365 is a cloud-based productivity suite..."
+{
+  "schemaVersion": "1.2.0",
+  "default_evaluators": {
+    "Relevance": {},
+    "Coherence": {}
   },
-  {
-    "prompt": "How can I share a file in Teams?",
-    "expected_response": "You can share a file in Teams by uploading it..."
-  }
-]
+  "items": [
+    {
+      "prompt": "What is Microsoft Graph?",
+      "expected_response": "A unified API endpoint for Microsoft services.",
+      "evaluators": {
+        "Citations": { "citation_format": "mixed" }
+      },
+      "evaluators_mode": "extend"
+    },
+    {
+      "name": "Expense policy flow",
+      "turns": [
+        {
+          "prompt": "I spent $250 on dinner. Is that okay?",
+          "expected_response": "The per-diem meal allowance is $200."
+        },
+        {
+          "prompt": "What should I do about the overage?",
+          "expected_response": "Request manager approval.",
+          "evaluators": {
+            "ExactMatch": { "case_sensitive": false }
+          },
+          "evaluators_mode": "replace"
+        }
+      ]
+    }
+  ]
+}
 ```
 
-Edit this file with your own prompts and run again!
+**How evaluator modes work in this example:**
 
-### Manual Creation
+| Item | `evaluators_mode` | Active Evaluators | Why |
+|------|-------------------|-------------------|-----|
+| Single-turn (Graph) | `extend` | Relevance, Coherence, Citations | Per-prompt Citations **merged** with defaults |
+| Multi-turn turn 1 (dinner) | _(none)_ | Relevance, Coherence | **Inherits** file-level defaults |
+| Multi-turn turn 2 (overage) | `replace` | ExactMatch | Per-turn ExactMatch **replaces** defaults entirely |
 
-Create `./evals/prompts.json`:
+### Evaluator Modes
 
-```json
-[
-  {
-    "prompt": "Your test prompt here",
-    "expected_response": "Expected agent response"
-  }
-]
-```
+| Mode | Behavior |
+|------|----------|
+| `"extend"` (default) | Per-item evaluators **merge** with defaults. Both run. |
+| `"replace"` | Per-item evaluators **replace** defaults entirely. Only per-item evaluators run. |
+| _(none)_ | Inherits file-level `default_evaluators`, or system defaults (Relevance, Coherence) if not set. |
+
+See `schema/v1/examples/` in the package for more examples including per-turn evaluator overrides, mixed single/multi-turn files, and output format.
+
+### Auto-Upgrade Behavior
+
+When the CLI loads an eval document:
+
+- **Legacy documents** (missing `schemaVersion`): Automatically upgraded with a timestamped backup (e.g., `file.json.bak.20260205143052`)
+- **Older versions** (same major version): `schemaVersion` field updated without backup
+- **Invalid documents**: CLI exits with an error message and guidance to review the schema changelog
+- **Future versions**: CLI rejects with a message suggesting a CLI update
+
+### Version Compatibility
+
+Within a major version (e.g., 1.x.x), we aim to maintain backward compatibility for documents that conform to the published schema for their version. Compatibility does not extend to undeclared or ad-hoc fields outside the schema definition; review the [schema changelog](schema/CHANGELOG.md) when upgrading between minor versions.
 
 ## 🎯 Usage Examples
 
@@ -265,10 +331,22 @@ runevals --prompts "What is Microsoft Graph?" --expected "Gateway to M365 data"
 # Interactive mode (enter prompts interactively)
 runevals --interactive
 
+# Canonical logging verbosity
+runevals --log-level debug
+runevals --log-level info
+runevals --log-level warning
+runevals --log-level error
+
+# Parallel prompt execution control
+runevals --concurrency 5 --prompts-file ./evals/evals.json
+runevals --concurrency 1000 --prompts-file ./evals/evals.json   # Python CLI clamps to 5
+
 # Custom output location in your project
 runevals --output ./reports/results.html
 ```
 
+> **⚠️ Debug log safety notice:** The `--log-level debug` option is opt-in and may include raw API payloads and response data in console output. Redaction is pattern-based (API keys, tokens, passwords, long mixed-case strings) and **will not catch arbitrary PII or custom credentials** embedded in prompts or responses. Do not share debug-level output publicly without manual review.
+
 ### Optional: Add Shortcuts to package.json
 
 You can add shortcuts (npm scripts) to your agent project's `package.json`:
@@ -301,10 +379,9 @@ npm run eval:dev
 ## 📊 Output Formats
 
 Results are automatically saved to `./evals/YYYY-MM-DD_HH-MM-SS.html` with:
-- **Relevance** score (1-5)
-- **Coherence** score (1-5)  
-- **Groundedness** score (1-5)
-- Per-prompt details and aggregate metrics
+- Per-prompt and per-turn evaluation scores from configured evaluators
+- Aggregate statistics across all evaluated items
+- Multi-turn thread summaries (turns passed/failed, overall status)
 
 Other formats:
 ```bash
@@ -320,8 +397,7 @@ runevals --output results.csv
 ```bash
 Options:
   -V, --version                 output version number
-  -v, --verbose                 show detailed processing steps
-  -q, --quiet                   minimal output
+  --log-level [level]           log level: debug|info|warning|error (bare flag -> info)
   --prompts <prompts...>        inline prompts to evaluate
   --expected <responses...>     expected responses (with --prompts)
   --prompts-file <file>         JSON file with prompts
@@ -360,7 +436,7 @@ runevals cache-info
 
 # Clear and rebuild
 runevals cache-clear
-runevals --init-only --verbose
+runevals --init-only --log-level debug
 ```
 
 ### Network/Proxy Issues
@@ -369,7 +445,7 @@ runevals --init-only --verbose
 export HTTPS_PROXY=http://proxy:8080
 
 # Retry with verbose output
-runevals --init-only --verbose
+runevals --init-only --log-level debug
 ```
 
 ### Permission Issues
@@ -403,43 +479,6 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
 
-## Schema Versioning
-
-The eval document schema is versioned independently from the CLI, following [Semantic Versioning](https://semver.org/). This allows external consumers to depend on a stable contract without coupling to CLI release cycles.
-
-- **Schema location**: [`schema/v1/eval-document.schema.json`](schema/v1/eval-document.schema.json)
-- **Schema changelog**: [`schema/CHANGELOG.md`](schema/CHANGELOG.md)
-- **Consumer quickstart**: [`specs/wi-6081652-dataset-schema-versioning/quickstart.md`](specs/wi-6081652-dataset-schema-versioning/quickstart.md)
-
-### Eval Document Format
-
-Eval documents should include a `schemaVersion` field:
-
-```json
-{
-  "schemaVersion": "1.0.0",
-  "items": [
-    {
-      "prompt": "What is Microsoft 365?",
-      "expected_response": "Microsoft 365 is a cloud-based productivity suite."
-    }
-  ]
-}
-```
-
-### Auto-Upgrade Behavior
-
-When the CLI loads an eval document:
-
-- **Legacy documents** (missing `schemaVersion`): Automatically upgraded with a timestamped backup (e.g., `file.json.bak.20260205143052`)
-- **Older versions** (same major version): `schemaVersion` field updated without backup
-- **Invalid documents**: CLI exits with an error message and guidance to review the schema changelog
-- **Future versions**: CLI rejects with a message suggesting a CLI update
-
-### Version Compatibility
-
-Within a major version (e.g., 1.x.x), backward compatibility is guaranteed. Documents valid against 1.0.0 will remain valid against 1.1.0, 1.2.0, etc.
-
 ## Trademarks
 
 This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft

package/package.json

@@ -1,9 +1,9 @@
 {
   "name": "@microsoft/m365-copilot-eval",
-  "version": "1.2.1-preview.1",
+  "version": "1.4.0-preview.1",
   "minCliVersion": "1.0.1-preview.1",
   "description": "Zero-config Node.js wrapper for M365 Copilot Agent Evaluations CLI (Python-based Azure AI Evaluation SDK)",
-  "publishDate": "2026-03-23",
+  "publishDate": "2026-04-22",
   "main": "src/clients/node-js/lib/index.js",
   "type": "module",
   "bin": {
@@ -14,6 +14,7 @@
     "build": "npm run prettier:check && npm run clean && npm run lint",
     "clean": "rimraf node_modules/.cache dist coverage",
     "test": "node --test tests/clients/node-js/**/*.test.js",
+    "test:coverage": "c8 --reporter=cobertura --reporter=text node --test tests/clients/node-js/**/*.test.js",
     "install-credprovider": "artifacts-npm-credprovider && npm ci",
     "set-publish-date": "node scripts/set-publish-date.js",
     "prepublishOnly": "node scripts/set-publish-date.js",
@@ -41,9 +42,10 @@
   },
   "dependencies": {
     "commander": "^12.1.0",
+    "dotenv": "^16.0.0",
+    "https-proxy-agent": "^7.0.5",
     "node-fetch": "^3.3.2",
-    "tar": "^7.5.4",
-    "https-proxy-agent": "^7.0.5"
+    "tar": "^7.5.4"
   },
   "devDependencies": {
     "@microsoft/eslint-config-msgraph": "^5.0.0",
@@ -52,6 +54,7 @@
     "@vitest/coverage-istanbul": "^3.0.0",
     "@vitest/coverage-v8": "^3.0.0",
     "@vitest/ui": "^3.0.0",
+    "c8": "^11.0.0",
     "eslint": "^9.7.0",
     "eslint-config-prettier": "^10.0.0",
     "eslint-plugin-jsdoc": "^50.1.0",

package/schema/CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to the eval document schema will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.0.0...schema-v1.1.0) (2026-03-30)
+
+
+### Features
+
+* **WI-6855059:** add agentName/cliVersion to schema, fix duplicate prompt loss, include default_evaluators in output ([#181](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/181)) ([9321474](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/93214746144e9d11f507433eff185aefac4a858a))
+* **WI-6855059:** implement per-prompt evaluator configuration ([#168](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/168)) ([eface7e](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/eface7e7041b118681cd4c68582fe903640bf6c0))
+
 ## [1.0.0] - 2026-02-19
 
 ### Added