@microsoft/m365-copilot-eval 1.4.0-preview.1 → 1.6.0-preview.1

package/README.md

@@ -1,8 +1,8 @@
 # M365 Copilot Agent Evaluations
 
-> **🔒 PRIVATE PREVIEW:** This tool is currently in private preview. And the instructions below are for Private Preview.
+> **PUBLIC PREVIEW:** This tool is currently in public preview; refer to the instructions below to get started.
 
-A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to your agent, get responses, and automatically score them with Azure AI Evaluation metrics (relevance, coherence, groundedness).
+A CLI for evaluating M365 Copilot agents. Send prompts to your agent, get responses, and automatically score them with Azure AI Evaluation metrics (relevance, coherence, groundedness).
 - Send a batch (or interactive set) of prompts to a configured chat API endpoint.
 - Collect agent responses and evaluate them locally using Azure AI Evaluation SDK.
 - The CLI supports 7 evaluator types. Evaluators marked with ⭐ are **enabled by default**.
@@ -12,7 +12,7 @@ A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to
 | **Relevance** ⭐ | LLM-based | 1-5 | 3 | Yes |
 | **Coherence** ⭐ | LLM-based | 1-5 | 3 | Yes |
 | **Groundedness** | LLM-based | 1-5 | 3 | No |
-| **ToolCallAccuracy** | LLM-based | 1-5 | 3 | No |
+| **Similarity** | LLM-based | 1-5 | 3 | No |
 | **Citations** | Count-based | >= 0 | 1 | No |
 | **ExactMatch** | String match | boolean | N/A | No |
 | **PartialMatch** | String match | 0.0-1.0 | 0.5 | No |
@@ -24,8 +24,10 @@ A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to
 - **M365 Copilot License** for your tenant
 - **M365 Copilot Agent** deployed to your tenant (can be created with [M365 Agents Toolkit](https://learn.microsoft.com/en-us/microsoft-365/developer/overview-m365-agents-toolkit) or any other method)
 - **Node.js 24.12.0+** (check: `node --version`)
+- **Python 3.13.x** is downloaded automatically. If the download fails (e.g., network restrictions), set `PYTHON_PATH` to a local Python 3.13.x installation (see [Troubleshooting](#-troubleshooting))
 - **Environment file** with your credentials and agent ID (see [Environment Setup](#-environment-setup) below)
 - **Your Tenant ID** - get your tenant id using the instructions [here](https://learn.microsoft.com/en-us/azure/azure-portal/get-subscription-tenant-id) 
+- Admin approval to run WORKIQ Client App for your tenant [here](https://github.com/microsoft/work-iq/blob/main/ADMIN-INSTRUCTIONS.md)
 - **Azure OpenAI endpoint, and API key** (see [Getting Variables](#-getting-variables) below)
 
 > Note: Authentication is currently supported on Windows only. Support for other operating systems is coming soon.
@@ -66,6 +68,8 @@ M365_TITLE_ID="T_your-title-id-here"  # Auto-generated by ATK
 # .env.local.user (NOT checked in — secrets go here)
 AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
 AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
+AZURE_AI_API_VERSION="2024-12-01-preview"  # default
+AZURE_AI_MODEL_NAME="gpt-4o-mini"           # recommended
 TENANT_ID="<your-tenant-id>"
 ```
 
@@ -90,7 +94,7 @@ M365_AGENT_ID="your-agent-id"  # e.g., U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4
 AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
 AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
 AZURE_AI_API_VERSION="2024-12-01-preview"  # default
-AZURE_AI_MODEL_NAME="gpt-4o-mini"           # default
+AZURE_AI_MODEL_NAME="gpt-4o-mini"           # recommended
 TENANT_ID="<your-tenant-id>"
 ```
 
@@ -457,6 +461,20 @@ runevals cache-dir
 chmod -R u+w $(runevals cache-dir)
 ```
 
+### Custom Python Runtime (PYTHON_PATH)
+
+If the automatic Python download fails (e.g., network restrictions, unsupported platform), provide your own Python installation:
+
+```bash
+# Windows
+set PYTHON_PATH=C:\Python313\python.exe
+
+# macOS/Linux
+export PYTHON_PATH=/usr/local/bin/python3.13
+```
+
+Python 3.13.x is the tested version. If a different version is found, you'll be prompted to confirm before proceeding. In CI/CD, a version mismatch fails automatically.
+
 ## 📚 Advanced Documentation
 
 - **[CI/CD Integration](./CICD_CACHE_GUIDE.md)** - GitHub Actions, Azure DevOps caching

package/package.json

@@ -1,9 +1,9 @@
 {
   "name": "@microsoft/m365-copilot-eval",
-  "version": "1.4.0-preview.1",
+  "version": "1.6.0-preview.1",
   "minCliVersion": "1.0.1-preview.1",
   "description": "Zero-config Node.js wrapper for M365 Copilot Agent Evaluations CLI (Python-based Azure AI Evaluation SDK)",
-  "publishDate": "2026-04-22",
+  "publishDate": "2026-05-07",
   "main": "src/clients/node-js/lib/index.js",
   "type": "module",
   "bin": {
@@ -80,8 +80,9 @@
     "README.md",
     "LICENSE"
   ],
+  "homepage": "https://github.com/microsoft/m365-copilot-eval",
   "repository": {
     "type": "git",
-    "url": "https://github.com/microsoft/M365-Copilot-Agent-Evals.git"
+    "url": "https://github.com/microsoft/m365-copilot-eval.git"
   }
 }

package/schema/CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to the eval document schema will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.2.0...schema-v1.3.0) (2026-04-30)
+
+
+### Features
+
+* Added similarity evaluator for compatibility with MCS Evals. ([#228](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/228)) ([0fe8315](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/0fe8315abc8e0422d1ac9117fe9f29195f29044f))
+
+## [1.2.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.1.0...schema-v1.2.0) (2026-04-22)
+
+
+### Features
+
+* **schema:** add multi-turn evaluation support (v1.2.0) ([#208](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/208)) ([a5ad22b](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/a5ad22bb4f6ac8ba548dc7f431ace073fa5970ce))
+
 ## [1.1.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.0.0...schema-v1.1.0) (2026-03-30)
 
 

package/schema/v1/eval-document.schema.json

@@ -287,9 +287,9 @@
           "$ref": "#/$defs/EvalScore",
           "description": "Groundedness score (1-5)"
         },
-        "toolCallAccuracy": {
+        "similarity": {
           "$ref": "#/$defs/EvalScore",
-          "description": "Tool call accuracy score (1-5)"
+          "description": "Similarity score (1-5)"
         },
         "citations": {
           "$ref": "#/$defs/CitationScore",
@@ -427,7 +427,7 @@
       "type": "object",
       "description": "Map of evaluator names to their configuration options",
       "propertyNames": {
-        "enum": ["Relevance", "Coherence", "Groundedness", "ToolCallAccuracy", "Citations", "ExactMatch", "PartialMatch"]
+        "enum": ["Relevance", "Coherence", "Groundedness", "Similarity", "Citations", "ExactMatch", "PartialMatch"]
       },
       "additionalProperties": {
         "$ref": "#/$defs/EvaluatorOptions"

package/schema/version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.2.0",
+  "version": "1.3.0",
   "releaseDate": "2026-04-02",
   "schemaId": "https://raw.githubusercontent.com/microsoft/M365-Copilot-Agent-Evals/refs/heads/main/schema/v1/eval-document.schema.json",
   "description": "M365 Copilot Eval Document Schema"

package/src/clients/cli/agent_selector.py

@@ -0,0 +1,74 @@
+"""Interactive agent selection and agent-id utilities."""
+
+from typing import Any, Dict, List, Optional, Tuple
+
+import questionary
+
+from cli_logging.cli_logger import emit_structured_log
+from cli_logging.logging_utils import Operation
+
+
+def normalize_agent_id(agent_id):
+    """Append '.declarativeAgent' if agent_id has no '.', else return unchanged.
+
+    Returns the input unchanged when it is None/empty or already contains a dot.
+    """
+    if not agent_id:
+        return agent_id
+    return agent_id if '.' in agent_id else f"{agent_id}.declarativeAgent"
+
+
+def select_agent_interactively(agents: List[Dict[str, Any]]) -> Tuple[Optional[str], Optional[str]]:
+    """
+    Display an interactive agent selector using questionary.
+
+    Args:
+        agents: List of agent dictionaries.
+
+    Returns:
+        Tuple of (agent_id, agent_name) or (None, None) if cancelled/skipped
+    """
+    if not agents:
+        return None, None
+
+    # Build id→name lookup and choices
+    id_to_name: Dict[str, str] = {}
+    choices = []
+    sorted_agents = sorted(agents, key=lambda a: a.get("name", ""))
+    for agent in sorted_agents:
+        agent_name = agent.get("name", "Unknown")
+        agent_id = (agent.get("gptId") or "").strip()
+        if not agent_id:
+            emit_structured_log("warning", f"Skipping agent '{agent_name}': missing or empty gptId.", operation=Operation.FETCH_AGENTS)
+            continue
+        agent_description = agent.get("description")
+        agent_is_owner = agent.get('isOwner')
+        agent_provider = agent.get("provider")
+        id_to_name[agent_id] = agent_name
+
+        # Format the display text
+        if agent_provider:
+            title = f"{agent_name} - {agent_provider} ({agent_id})"
+        else:
+            title = f"{agent_name} ({agent_id})"
+        segments = [title]
+        if agent_is_owner:
+            segments.append(f"IsOwner: {agent_is_owner}")
+        if agent_description:
+            segments.append(agent_description)
+        display_text = " - ".join(segments)
+
+        choices.append(questionary.Choice(title=display_text, value=agent_id))
+
+    if not choices:
+        return None, None
+
+    # Display the selection prompt
+    selected_agent = questionary.select(
+        "Select an agent to evaluate:",
+        choices=choices,
+        use_shortcuts=len(choices) <= 35,
+        use_arrow_keys=True
+    ).ask()
+
+    return selected_agent, id_to_name.get(selected_agent) if selected_agent else None

package/src/clients/cli/api_clients/A2A/a2a_client.py

@@ -8,7 +8,7 @@ import re
 import urllib.error
 import urllib.request
 import uuid
-from typing import Any, Dict, List, Optional, Tuple
+from typing import Any, Callable, Dict, List, Optional, Tuple
 
 from api_clients.base_agent_client import BaseAgentClient
 from cli_logging.console_diagnostics import emit_structured_log
@@ -35,6 +35,7 @@ class A2AClient(BaseAgentClient):
         access_token: str,
         logger: Optional[logging.Logger] = None,
         diagnostic_records: Optional[List[Dict[str, Any]]] = None,
+        token_refresh_fn: Optional[Callable[[], str]] = None,
     ) -> None:
         """
         Args:
@@ -42,11 +43,15 @@ class A2AClient(BaseAgentClient):
             access_token: Bearer token for A2A authentication.
             logger: Logger to use. Defaults to a module-level logger if not provided.
             diagnostic_records: List to accumulate structured log entries.
+            token_refresh_fn: Optional callable that returns a fresh access token string.
+                When provided, a single HTTP 401 response will trigger a token refresh
+                and one automatic retry, making the refresh invisible to the caller.
         """
         self._endpoint = a2a_endpoint.rstrip("/")
         self._access_token = access_token
         self._logger = logger or logging.getLogger(__name__)
         self._diagnostic_records = diagnostic_records
+        self._token_refresh_fn = token_refresh_fn
         self._resolved_agent_url: Optional[str] = None
 
     # ------------------------------------------------------------------ #
@@ -61,7 +66,7 @@ class A2AClient(BaseAgentClient):
         """Fetch agents from the A2A discovery endpoint.
 
         Calls GET {endpoint}/.agents. Each A2A agent card is normalized to
-        include 'gptId', 'name', and 'description' so it is compatible with
+        include 'gptId', 'name', and 'provider' so it is compatible with
         the shared select_agent_interactively selector.
 
         Returns an empty list if the endpoint is unreachable or returns an
@@ -79,8 +84,14 @@ class A2AClient(BaseAgentClient):
             )
             req = urllib.request.Request(agents_url, headers=headers, method="GET")
             with urllib.request.urlopen(req, timeout=_REQUEST_TIMEOUT_SECS) as resp:
-                data = json.loads(resp.read().decode("utf-8"))
-            agents = data if isinstance(data, list) else data.get("agents", [])
+                agents = json.loads(resp.read().decode("utf-8"))
+            emit_structured_log(
+                "debug",
+                f"[A2A] Available agents response: {json.dumps(agents)}",
+                Operation.FETCH_AGENTS,
+                logger=self._logger,
+                diagnostic_records=self._diagnostic_records
+            )
             return [self._normalize_agent_card(a) for a in agents]
         except urllib.error.HTTPError as e:
             emit_structured_log(
@@ -104,22 +115,11 @@ class A2AClient(BaseAgentClient):
     @staticmethod
     def _normalize_agent_card(agent: Dict[str, Any]) -> Dict[str, Any]:
         """Normalize an A2A agent card to the shape expected by the selector.
-
-        A2A agent cards use a 'url' field rather than a discrete ID.  The
-        agent ID is extracted as the last path segment of that URL, falling
-        back to the agent name when the URL is absent.
         """
-        agent_url = agent.get("url", "")
-        agent_id = (
-            agent_url.rstrip("/").rsplit("/", 1)[-1]
-            if agent_url
-            else agent.get("name", "")
-        )
         return {
-            "gptId": agent_id,
-            "name": agent.get("name", agent_id),
-            "description": agent.get("description", ""),
-            "isOwner": False,
+            "gptId": agent.get("agentId"),
+            "name": agent.get("name"),
+            "provider": agent.get("provider")
         }
 
     def send_prompt(
@@ -266,6 +266,12 @@ class A2AClient(BaseAgentClient):
     ) -> tuple[Dict[str, Any], Dict[str, Any]]:
         """Send a JSON-RPC message to the agent and parse the response.
 
+        When a ``token_refresh_fn`` was supplied at construction time and the
+        server responds with HTTP 401 (Unauthorized), the token is refreshed
+        automatically and the request is retried exactly once.  This keeps
+        long-running eval sessions alive beyond the initial token lifetime
+        without requiring any user interaction.
+
         Returns:
             A tuple of (result_dict, raw_result) where result_dict is the
             normalized response dict (raw_response_text, display_response_text,
@@ -280,15 +286,51 @@ class A2AClient(BaseAgentClient):
             with urllib.request.urlopen(req, timeout=_REQUEST_TIMEOUT_SECS) as resp:
                 raw = resp.read().decode("utf-8", errors="replace")
         except urllib.error.HTTPError as e:
-            body = ""
-            try:
-                body = e.read().decode("utf-8", errors="replace")
-            except Exception:
-                pass
-            raise RuntimeError(
-                f"A2A request failed (HTTP {e.code} {e.reason})."
-                + (f" Body: {body[:500]}" if body else "")
-            ) from e
+            if e.code == 401 and self._token_refresh_fn is not None:
+                emit_structured_log(
+                    "info",
+                    "[A2A] Access token expired (HTTP 401); refreshing token and retrying.",
+                    Operation.AUTHENTICATE,
+                    logger=self._logger,
+                    diagnostic_records=self._diagnostic_records,
+                )
+                new_token = self._token_refresh_fn()
+                if not new_token:
+                    raise RuntimeError(
+                        "A2A request failed (HTTP 401 Unauthorized) and token refresh returned no token."
+                    ) from e
+                self._access_token = new_token
+                headers["Authorization"] = f"Bearer {self._access_token}"
+                retry_req = urllib.request.Request(
+                    agent_url, data=payload, headers=headers, method="POST"
+                )
+                try:
+                    with urllib.request.urlopen(retry_req, timeout=_REQUEST_TIMEOUT_SECS) as resp:
+                        raw = resp.read().decode("utf-8", errors="replace")
+                except urllib.error.HTTPError as retry_e:
+                    body = ""
+                    try:
+                        body = retry_e.read().decode("utf-8", errors="replace")
+                    except Exception:
+                        pass
+                    raise RuntimeError(
+                        f"A2A request failed (HTTP {retry_e.code} {retry_e.reason}) after token refresh."
+                        + (f" Body: {body[:500]}" if body else "")
+                    ) from retry_e
+                except urllib.error.URLError as retry_e:
+                    raise RuntimeError(
+                        f"A2A connection error after token refresh: {getattr(retry_e, 'reason', str(retry_e))}"
+                    ) from retry_e
+            else:
+                body = ""
+                try:
+                    body = e.read().decode("utf-8", errors="replace")
+                except Exception:
+                    pass
+                raise RuntimeError(
+                    f"A2A request failed (HTTP {e.code} {e.reason})."
+                    + (f" Body: {body[:500]}" if body else "")
+                ) from e
         except urllib.error.URLError as e:
             raise RuntimeError(
                 f"A2A connection error: {getattr(e, 'reason', str(e))}"
@@ -334,15 +376,39 @@ class A2AClient(BaseAgentClient):
             state = result.get("status", {}).get("state")
             if state == "completed":
                 msg = result.get("status", {}).get("message") or {}
+                all_parts = list(msg.get("parts", []))
+                for artifact in result.get("artifacts", []):
+                    all_parts.extend(artifact.get("parts", []))
                 text = "\n".join(
                     p.get("text", "")
-                    for p in msg.get("parts", [])
+                    for p in all_parts
                     if p.get("kind") == "text"
                 )
                 attributions = msg.get("metadata", {}).get("attributions", [])
-            elif state in ("failed", "canceled"):
+            elif state in ("failed", "canceled", "rejected"):
+                status_msg = result.get("status", {}).get("message") or {}
+                detail = "\n".join(
+                    p.get("text", "")
+                    for p in status_msg.get("parts", [])
+                    if p.get("kind") == "text"
+                ).strip()
+                suffix = f" Detail: {detail}" if detail else ""
+                raise RuntimeError(
+                    f"A2A task {state}. Task id: {result.get('id')}{suffix}"
+                )
+            elif state in ("input_required", "auth_required"):
+                requirement = {
+                    "input_required": "user input",
+                    "auth_required": "authentication",
+                }.get(state, state.replace("_", " "))
+                raise RuntimeError(
+                    f"A2A task requires {requirement} and cannot proceed automatically."
+                    f" Task id: {result.get('id')}"
+                )
+            elif state in ("submitted", "working"):
                 raise RuntimeError(
-                    f"A2A task {state}. Task id: {result.get('id')}"
+                    f"A2A task is still {state}; synchronous send returned before completion."
+                    f" Task id: {result.get('id')}"
                 )
             else:
                 raise RuntimeError(

package/src/clients/cli/api_clients/base_agent_client.py

@@ -44,7 +44,6 @@ class BaseAgentClient(ABC):
             The conversation_context should be passed to the next turn
             in a multi-turn conversation, or discarded for single-turn.
             The context structure is implementation-specific:
-              - Sydney/REST: {"conversation_id": str}
               - A2A: {"context_id": str}
             Returns None as context when no conversation state is established.
         """

package/src/clients/cli/auth/auth_handler.py

@@ -13,7 +13,7 @@ https://github.com/AzureAD/microsoft-authentication-extensions-for-python
 import os
 import platform
 import logging
-from typing import Optional
+from typing import Callable, Optional
 from pathlib import Path
 import jwt
 from msal import PublicClientApplication
@@ -260,3 +260,23 @@ class AuthHandler:
             return oid
         except jwt.DecodeError as e:
             raise ValueError(f"Failed to decode token: {e}")
+
+
+def make_token_refresh_fn(auth_handler: "AuthHandler") -> Callable[[], str]:
+    """Return a callable that silently refreshes the A2A access token.
+
+    On a 401 response the caller invokes this function.  It first attempts a
+    silent refresh (using the MSAL refresh token) and falls back to interactive
+    authentication only when a silent refresh is not possible.  The returned
+    string is the new access token; an empty string signals failure.
+
+    Args:
+        auth_handler: An initialized AuthHandler instance to use for token acquisition.
+
+    Returns:
+        A zero-argument callable that returns a fresh access token string.
+    """
+    def _refresh() -> str:
+        result = auth_handler.acquire_token_interactive() or {}
+        return result.get("access_token") or ""
+    return _refresh

package/src/clients/cli/cli_args.py

@@ -0,0 +1,136 @@
+"""CLI argument parsing and version-check bypass logic."""
+
+import argparse
+import os
+
+from cli_logging.cli_logger import emit_structured_log
+from cli_logging.logging_utils import Operation
+from common import MAX_CONCURRENCY, RunConfig
+from agent_selector import normalize_agent_id
+
+
+# Flags that should bypass remote min-version enforcement.
+# --help is not needed here because argparse exits before runtime checks.
+VERSION_CHECK_BYPASS_FLAGS = (
+    "signout",
+)
+
+
+def should_bypass_min_version_check(config: RunConfig) -> bool:
+    """Return True if the current invocation should skip min-version checks."""
+    return any(getattr(config, flag, False) for flag in VERSION_CHECK_BYPASS_FLAGS)
+
+
+def parse_arguments():
+    """Parse command line arguments."""
+    parser = argparse.ArgumentParser(
+        description="M365 Copilot Agent Evaluation CLI",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Run with default prompts
+  python main.py
+
+  # Run with custom prompts
+  python main.py --prompts "What is Microsoft Graph?" --expected "Microsoft Graph is a gateway..."
+
+  # Run with prompts from file
+  python main.py --prompts-file prompts.json
+
+  # Interactive mode
+  python main.py --interactive
+
+  # Save results to JSON
+  python main.py --output results.json
+
+  # Save results to CSV
+  python main.py --output results.csv
+
+  # Save results to HTML and open in browser
+  python main.py --output report.html
+
+  # Debug-level diagnostics
+  python main.py --log-level debug
+
+  # Sign out and clear cached authentication tokens
+  python main.py --signout
+        """
+    )
+    
+    # Input options (mutually exclusive)
+    input_group = parser.add_mutually_exclusive_group()
+    input_group.add_argument(
+        '--prompts', 
+        nargs='+', 
+        help='List of prompts to evaluate'
+    )
+    input_group.add_argument(
+        '--prompts-file', 
+        type=str, 
+        help='JSON file containing prompts and expected responses'
+    )
+    input_group.add_argument(
+        '--interactive', 
+        action='store_true', 
+        help='Interactive mode to enter prompts'
+    )
+    
+    # Expected responses (only used with --prompts)
+    parser.add_argument(
+        '--expected', 
+        nargs='+', 
+        help='List of expected responses (must match number of prompts)'
+    )
+    
+    # Agent ID (--m365-agent-id is primary, --agent-id kept for backward compatibility)
+    parser.add_argument(
+        '--m365-agent-id', '--agent-id',
+        type=str,
+        default=os.environ.get("M365_AGENT_ID") or os.environ.get("AGENT_ID"),
+        help='Agent ID (default from M365_AGENT_ID environment variable)'
+    )
+
+    # Output options
+    parser.add_argument(
+        '--output', 
+        type=str, 
+        help='Output file path. Format is determined by file extension: .json, .csv, .html. If not provided, results are printed to console.'
+    )
+    
+    # Behavior options
+    parser.add_argument(
+        '--log-level',
+        nargs='?',
+        const='info',
+        action='append',
+        help='Set log verbosity: debug, info, warning, error. Bare --log-level resolves to info.'
+    )
+
+    parser.add_argument(
+        '--signout',
+        action='store_true',
+        help='Sign out and clear cached authentication tokens'
+    )
+
+    parser.add_argument(
+        '--concurrency',
+        type=int,
+        default=MAX_CONCURRENCY,
+        help=f'Number of parallel workers for prompt processing (1-{MAX_CONCURRENCY}, default: {MAX_CONCURRENCY})'
+    )
+    
+    args = parser.parse_args()
+
+    args.m365_agent_id = normalize_agent_id(args.m365_agent_id)
+
+    if args.concurrency < 1:
+        parser.error('--concurrency must be an integer >= 1.')
+    if args.concurrency > MAX_CONCURRENCY:
+        emit_structured_log(
+            "warning",
+            f"--concurrency {args.concurrency} exceeds max {MAX_CONCURRENCY}; clamping to {MAX_CONCURRENCY}.",
+            operation=Operation.SETUP,
+        )
+        args.concurrency = MAX_CONCURRENCY
+
+    return args

package/src/clients/cli/cli_logging/cli_logger.py

@@ -0,0 +1,33 @@
+"""Shared CLI logger instance and structured-log convenience wrapper.
+
+Every module in the CLI layer that needs to emit diagnostics imports from here
+instead of main.py, which avoids circular-import issues.
+"""
+
+import logging
+import sys
+from typing import Any, Dict, List
+
+from cli_logging.console_diagnostics import emit_structured_log as _emit_structured_log
+from cli_logging.logging_utils import LOG_LEVEL_MAP, Operation
+
+CLI_LOGGER_NAME = "m365.eval.cli"
+CLI_LOGGER = logging.getLogger(CLI_LOGGER_NAME)
+DIAGNOSTIC_RECORDS: List[Dict[str, Any]] = []
+
+
+def configure_cli_logging(effective_log_level: str) -> None:
+    if not CLI_LOGGER.handlers:
+        handler = logging.StreamHandler(sys.stdout)
+        handler.setFormatter(logging.Formatter("%(message)s"))
+        CLI_LOGGER.addHandler(handler)
+        CLI_LOGGER.propagate = False
+    CLI_LOGGER.setLevel(LOG_LEVEL_MAP[effective_log_level])
+
+
+def emit_structured_log(level: str, message: str, operation: str = Operation.EVALUATE) -> None:
+    _emit_structured_log(
+        level, message, operation,
+        logger=CLI_LOGGER,
+        diagnostic_records=DIAGNOSTIC_RECORDS,
+    )

package/src/clients/cli/cli_logging/console_diagnostics.py

@@ -8,6 +8,7 @@ from cli_logging.logging_utils import (
     STRUCTURED_LOG_FIELDS,
     Operation,
     format_structured_log_entry,
+    redact_sensitive_content,
 )
 
 _ANSI_COLORS = {
@@ -102,6 +103,7 @@ def emit_structured_log(
     if diagnostic_records is not None:
         diagnostic_records.append(entry)
     try:
-        logger.log(getattr(logging, level.upper(), logging.INFO), render_diagnostic(entry))
+        rendered, _ = redact_sensitive_content(render_diagnostic(entry))
+        logger.log(getattr(logging, level.upper(), logging.INFO), rendered)
     except Exception:
         pass