@microsoft/m365-copilot-eval 1.0.1-preview.1

package/src/clients/cli/readme.md

@@ -0,0 +1,301 @@
+# M365 Copilot Agent Evaluations (CLI)
+
+This CLI evaluates DA responses from a configured M365 Copilot endpoint (copilotApi) and scores them locally using Azure AI Evaluation services.
+
+Current evaluation metrics:
+- Relevance (1–5)
+- Coherence (1–5)
+- Groundedness (1–5)
+- Citations (count with pass/fail based on presence)
+- Tool Call Accuracy (1-5, evaluates correct tool usage when tools are invoked)
+
+## 📋 Prerequisites
+
+- Python 3.13.10+ - https://www.python.org/downloads/
+- Azure subscription with Azure OpenAI access (for evaluation metrics only)
+
+## 🚀 Quick Start (Run CLI)
+
+### 1. Clone and Install Dependencies
+
+```bash
+git clone https://github.com/microsoft/M365-Copilot-Agent-Evals.git
+cd M365-Copilot-Agent-Evals/src/clients/cli
+python -m venv .venv # Create virtual Python environment.
+.\.venv\Scripts\Activate.ps1 # Activate the virtual environment.
+pip install -r requirements.txt
+```
+
+### 2. Set Up Environment Variables
+
+Create a `.env` file in the `src/clients/cli` directory (or export them). Choose **one** auth path for the Copilot API: either provide a pre-issued token or use interactive WAM auth (Windows).
+
+```bash
+# Azure OpenAI (evaluation models)
+AZURE_AI_OPENAI_ENDPOINT="<Your_Azure_AI_OpenAI_Endpoint>"
+AZURE_AI_API_KEY="<azure-openai-key>"
+AZURE_AI_API_VERSION="2024-12-01-preview"
+AZURE_AI_MODEL_NAME="gpt-4o-mini"
+
+# Copilot Chat API (response generation)
+COPILOT_API_ENDPOINT="https://substrate.office.com/m365Copilot"   # CLI appends /chat
+X_SCENARIO_HEADER="<scenario-header>"                             # e.g, officeweb
+
+# Auth option A: static access token (no prompt)
+COPILOT_API_ACCESS_TOKEN="<access-token>"
+
+# Auth option B: interactive WAM auth (used if COPILOT_API_ACCESS_TOKEN is empty)
+M365_EVAL_CLIENT_ID="<app-registration-client-id>"
+TENANT_ID="<aad-tenant-id>"
+COPILOT_SCOPES="https://substrate.office.com/sydney/.default"
+
+# Optional: default agent id (overridable via --agent-id)
+AGENT_ID="00000000-0000-0000-0000-000000000000"
+```
+
+### 3. Run the Agent Evaluation
+
+The CLI supports multiple ways to run evaluations:
+
+#### Basic Usage (with default prompt set)
+```bash
+python main.py
+```
+
+#### Command Line Prompts
+```bash
+# Single prompt and expected response
+python main.py --prompts "What is Microsoft Graph?" --expected "Microsoft Graph is a gateway to data and intelligence in Microsoft 365."
+
+# Multiple prompts and expected responses
+python main.py --prompts "What is Microsoft Graph?" "How does authentication work?" --expected "Microsoft Graph is a gateway..." "Authentication works by..."
+
+# Override the agent configured in environment variables
+python main.py --agent-id "00000000-0000-0000-0000-000000000000" --prompts "What is Microsoft Graph?"
+```
+
+#### Using Prompts from File
+```bash
+# JSON file with prompts and expected responses
+python main.py --prompts-file samples/example_prompts.json
+
+# Output to JSON file
+python main.py --prompts-file samples/example_prompts.json --output results.json
+
+# Output to CSV file
+python main.py --prompts-file samples/example_prompts.json --output results.csv
+
+# Output to HTML file (opens in browser)
+python main.py --prompts-file samples/example_prompts.json --output report.html
+```
+
+#### Interactive Mode
+```bash
+# Enter prompts interactively
+python main.py --interactive
+```
+
+#### Additional Options
+```bash
+# Verbose output (shows detailed processing steps)
+python main.py --verbose
+
+# Quiet mode (minimal output)
+python main.py --quiet
+
+# Get help and see all options
+python main.py --help
+
+# Specify / override the Agent ID (takes precedence over AZURE_AI_AGENT_ID env var)
+python main.py --agent-id "00000000-0000-0000-0000-000000000000"
+
+# Citation format options
+python main.py --citation-format oai_unicode      # Default: New OAI format
+python main.py --citation-format legacy_bracket   # Old [^i^] format
+```
+
+#### File Format Examples
+
+**JSON Format 1 (Array of Objects):**
+```json
+[
+  {
+    "prompt": "What is Microsoft Graph?",
+    "expected_response": "Microsoft Graph is a gateway to data and intelligence in Microsoft 365."
+  },
+  {
+    "prompt": "How do I authenticate with Microsoft Graph?",
+    "expected_response": "You can authenticate using OAuth 2.0..."
+  }
+]
+```
+
+**JSON Format 2 (Separate Arrays):**
+```json
+{
+  "prompts": [
+    "What is Microsoft Graph?",
+    "How do I authenticate with Microsoft Graph?"
+  ],
+  "expected_responses": [
+    "Microsoft Graph is a gateway to data and intelligence in Microsoft 365.",
+    "You can authenticate using OAuth 2.0..."
+  ]
+}
+```
+
+## 🔧 Tool Call Accuracy Evaluation
+
+The CLI now includes advanced tool call accuracy evaluation that analyzes how effectively the agent uses available tools:
+
+### What It Evaluates
+- **Tool Selection**: Whether the agent chooses appropriate tools for the given task
+- **Parameter Accuracy**: Correctness of arguments passed to tool functions
+- **Tool Usage Patterns**: Overall effectiveness of tool invocation strategies
+
+### How It Works
+1. **Response Analysis**: Extracts tool calls and results from conversation telemetry
+2. **Tool Definitions**: Captures available tools from conversation metadata
+3. **Accuracy Assessment**: Uses Azure AI Evaluation SDK's ToolCallAccuracyEvaluator
+4. **Score Calculation**: Returns 1-5 score with pass/fail threshold (default: 3)
+
+### Enhanced Data Extraction
+The tool now extracts detailed information from agent responses:
+- **Message Flow**: Complete chronological sequence of tool calls and results
+- **Tool Definitions**: Available tools and their schemas
+- **Internal Filtering**: Removes framework-internal tools for cleaner analysis
+
+### Example Output
+```bash
+📊 Aggregate Statistics (3 prompts):
+════════════════════════════════════════════════════════════
+Tool Call Accuracy:
+  Pass Rate: 66.7% (2/3 passed)
+  Avg Score: 0.75
+  Threshold: 0.5
+```
+
+**Note**: Tool Call Accuracy evaluation only applies when the agent response includes tool invocations. For text-only responses, this metric will not be computed.
+
+## 🔧 Configuration
+
+### Getting Azure AI Foundry Configuration Values
+
+#### 1. Azure OpenAI Endpoint (`AZURE_AI_OPENAI_ENDPOINT`)
+
+1. In [Azure AI Foundry](https://ai.azure.com/)
+2. Go to **Models + endpoints** under My assets
+3. Select your model. Create one if you don't have one available - Use `gpt-4o-mini` for better compatibility with the evaluators
+4. Copy the endpoint URL (format: `https://your-resource.openai.azure.com/`)
+
+#### 2. Azure OpenAI API Key (`AZURE_AI_API_KEY`)
+
+1. In [Azure AI Foundry](https://ai.azure.com/)
+2. Go to **Models + endpoints** under My assets
+3. Select your model. Create one if you don't have one available - Use `gpt-4o-mini` for better compatibility with the evaluators
+4. Copy the key
+
+#### 3. Azure OpenAI Model Name (`AZURE_AI_MODEL_NAME`)
+
+1. In [Azure AI Foundry](https://ai.azure.com/)
+2. Go to **Models + endpoints** under My assets
+3. Select your model. Create one if you don't have one available - Use `gpt-4o-mini` for better compatibility with the evaluators
+4. Copy the model/deployment name (e.g., "gpt-4o-mini", "gpt-4", "gpt-35-turbo")
+
+#### 4. API Version (`AZURE_AI_API_VERSION`)
+
+Use the API version: `2024-12-01-preview`
+
+For the most current version, check the [Azure OpenAI API reference](https://docs.microsoft.com/en-us/azure/cognitive-services/openai/reference).
+
+
+## 📁 Project Structure
+
+```
+M365-Copilot-Agent-Evals/
+├── src/
+│   └── clients/
+│       └── cli/
+│           ├── main.py              # Main evaluation script
+│           ├── response_extractor.py # Enhanced response parsing and tool extraction
+│           ├── generate_report.py   # HTML report generation with aggregates
+│           ├── requirements.txt     # Python dependencies
+│           ├── readme.md           # This file
+│           ├── CHANGELOG.md        # Version history and changes
+│           ├── .env.template       # Environment variables template
+│           ├── .env                # Environment variables (create this)
+│           ├── custom_evaluators/  # Custom evaluation modules
+│           │   ├── __init__.py
+│           │   └── CitationsEvaluator.py  # Citation detection evaluator
+│           └── samples/            # Example prompt files
+│               ├── example_prompts.json
+│               └── prompts_*.json  # Various test scenarios
+├── tests/                          # Test files
+├── CODE_OF_CONDUCT.md             # Code of conduct
+├── LICENSE                        # License file
+├── README.md                      # Root project README
+├── SECURITY.md                    # Security guidelines
+└── SUPPORT.md                     # Support information
+```
+
+## 📊 Features
+
+- **Chat Invocation**: Sends prompts to the Sydney chat API
+- **Evaluation Metrics**: Relevance, Coherence, Groundedness, Citations, Tool Call Accuracy
+  - **Multi-format Citation Detection**: Supports both new OAI Unicode and legacy bracket formats
+  - **Tool Call Analysis**: Extracts and evaluates tool invocations from conversation telemetry
+- **Enhanced Response Extraction**: Detailed parsing of tool calls, results, and message flow
+- **Aggregate Statistics**: Summary metrics across multiple prompts with pass/fail rates
+- **Colorized Console Output**
+- **Multiple Output Formats**: JSON, CSV, HTML with aggregate dashboards
+- **Flexible Prompt Input**: Command line, file, interactive
+
+## 🔒 Security Best Practices
+
+- ✅ All sensitive information is stored in environment variables
+- ✅ No hardcoded credentials in source code
+- ✅ `.env` files are excluded from version control
+
+## 🔑 Authentication Requirements
+
+- Copilot API: Either a static `COPILOT_API_ACCESS_TOKEN` **or** WAM interactive auth via `M365_EVAL_CLIENT_ID` and `TENANT_ID` (optional `COPILOT_SCOPES`)
+- Evaluators: Azure OpenAI key (`AZURE_AI_API_KEY`)
+
+## 📚 Useful Resources
+
+- [Azure AI Foundry Documentation](https://docs.microsoft.com/en-us/azure/ai-services/ai-foundry/)
+- [Azure AI Projects Python SDK](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/README.md)
+- [Azure AI Evaluation Documentation](https://docs.microsoft.com/en-us/azure/ai-services/ai-foundry/how-to/evaluate-models)
+- [Azure OpenAI Service Documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/openai/)
+- [Azure Identity Library](https://docs.microsoft.com/en-us/python/api/overview/azure/identity-readme)
+
+## 🐛 Troubleshooting
+
+### Common Issues
+
+1. **Authentication Errors**: Ensure your Azure credentials are properly configured and you have access to the resources.
+
+2. **HTTP 401 / 403**: If using a static token, verify `COPILOT_API_ACCESS_TOKEN` and required headers. If using WAM, confirm `M365_EVAL_CLIENT_ID`, `X_SCENARIO_HEADER`, `TENANT_ID`, and `COPILOT_SCOPES` are correct.
+
+3. **Endpoint Issues**: Confirm `COPILOT_API_ENDPOINT` (base URL without `/chat`) is reachable (try `curl` / `Invoke-WebRequest`).
+
+### Getting Help
+
+- Check the [Azure AI Foundry troubleshooting guide](https://docs.microsoft.com/en-us/azure/ai-services/ai-foundry/troubleshooting)
+- Review the Azure AI Projects SDK [troubleshooting section](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/README.md#troubleshooting)
+
+## 📋 Version History
+
+For detailed information about changes, new features, and breaking changes, see [CHANGELOG.md](CHANGELOG.md).
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Ensure all sensitive information uses environment variables
+5. Submit a pull request
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

package/src/clients/cli/requirements.txt

@@ -0,0 +1,10 @@
+ansible-core==2.19.0
+azure-ai-evaluation==1.10.0
+azure-ai-projects==1.0.0
+msal[broker]>=1.34,<2
+msal-extensions>=1.3.1
+pip==25.3
+python-dotenv==1.1.1
+markdown==3.8.2
+promptflow>=1.18.1
+questionary>=2.1.1