@microsoft/m365-copilot-eval 1.0.1-preview.1

package/LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

package/README.md

@@ -0,0 +1,415 @@
+# M365 Copilot Agent Evaluations
+
+> **🔒 PRIVATE PREVIEW:** This tool is currently in private preview. And the instructions below are for Private Preview.
+
+A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to your agent, get responses, and automatically score them with Azure AI Evaluation metrics (relevance, coherence, groundedness).
+- Send a batch (or interactive set) of prompts to a configured chat API endpoint.
+- Collect agent responses and evaluate them locally using Azure AI Evaluation SDK.
+- Metrics produced per prompt:
+- - Relevance (1–5)
+- - Coherence (1–5)
+- - Groundedness (1–5)
+- Multiple input modes: command‑line list, JSON file, interactive.
+- Multiple output formats: console (colorized), JSON, CSV, HTML (auto‑opens report).
+
+## 📋 Prerequisites
+
+- **M365 Copilot Agent** deployed to your tenant (can be created with [M365 Agents Toolkit](https://learn.microsoft.com/en-us/microsoft-365/developer/overview-m365-agents-toolkit) or any other method)
+- **Node.js 24.12.0+** (check: `node --version`)
+- **Environment file** with your credentials and agent ID (see [Environment Setup](#-environment-setup) below)
+- **Your Tenant ID, Azure OpenAI endpoint, and API key** (see [Getting Variables](#-getting-variables) below)
+
+> Note: Authentication is currently supported on Windows only. Support for other operating systems is coming soon.
+
+## 🔧 Environment Setup
+
+### Install the Tool
+
+1. Go to the releases → https://github.com/microsoft/M365-Copilot-Agent-Evals/releases and click on the most latest release.
+2. Click on the `Source code (tar.gz)`
+3. This should download the package to your device.
+4. Go to the folder where this tar.gz file is. This will now be your project root folder.
+5. Run `npm install -g <filename.tar.gz>` e.g., `npm install -g M365-Copilot-Agent-Evals-<version>.tar.gz`
+
+### Setup Steps
+
+Now, set up where you'll store your environment variables:
+
+**Are you using M365 Agents Toolkit (ATK)?**
+- ✅ **Yes** → You already have `.env.local` in your project with `M365_TITLE_ID`. You'll add Azure OpenAI variables to this file.
+- ✅ **No** → Create a new `env/.env.dev` file in your project directory. You'll add all variables there.
+
+The CLI loads environment variables from multiple sources (in order of precedence):
+
+1. **`.env.local`** in current directory (auto-detected, ideal for ATK projects)
+2. **`env/.env.{environment}`** via `--env` flag (e.g., `--env dev` loads `env/.env.dev`)
+3. **System environment variables**
+
+#### Option 1: For M365 Agents Toolkit (ATK) Projects
+
+If you're working in an ATK project, you already have `.env.local` with `M365_TITLE_ID`. Just add your Azure credentials and tenant ID:
+
+```bash
+# .env.local (existing ATK project file)
+# Already present from ATK:
+M365_TITLE_ID="T_your-title-id-here"  # Auto-generated by ATK
+
+# You'll add these (see Getting Variables section below):
+AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
+AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
+TENANT_ID="<your-tenant-id>"
+```
+
+#### Option 2: For Non-ATK Projects
+
+Create `env/.env.dev` in your project directory:
+
+```bash
+# env/.env.dev (new file you create)
+# Your agent ID (Optional):
+M365_AGENT_ID="your-agent-id"  # e.g., U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4
+
+# You'll add these (see Getting Variables section below):
+AZURE_AI_OPENAI_ENDPOINT="<your-azure-openai-endpoint>"
+AZURE_AI_API_KEY="<your-api-key-from-azure-portal>"
+TENANT_ID="<your-tenant-id>"
+```
+
+#### Optional Overrides
+```bash
+AZURE_AI_API_VERSION="2024-12-01-preview"  # default
+AZURE_AI_MODEL_NAME="gpt-4o-mini"           # default
+```
+
+You can also override the agent ID at runtime: `runevals --agent-id "custom-id"`
+
+---
+
+## 🔑 Getting Variables
+
+Now that you know what's needed, here's how to get the required values:
+
+### 1. Tenant ID
+
+Your Azure Active Directory (AAD) tenant ID.
+
+**How to obtain:**
+
+1. Go to [Azure Portal](https://portal.azure.com)
+2. Search for "Azure Active Directory" or "Microsoft Entra ID"
+3. In the Overview section, you'll see **Tenant ID**
+4. Copy this value - this is your `TENANT_ID`
+
+Alternatively, if you have the Azure CLI installed:
+```bash
+az account show --query tenantId
+```
+
+### 2. Agent ID (Only for MSIT)
+- If you have created your agent using Agents Toolkit, then the agent-id is the M365_TITLE_ID in .env.local file
+- If you did not, then you can get your agent-id by 
+```
+1. Open aka.ms/devui in your browser
+2. Click on `Configuration`
+3. In the dialog that opens, click on `Untitled Config`
+4. Click on the `Payload` tab
+5. If you scroll down on this tab, you will see `DA (Declarative Agent)`
+6. In this dropdown, you will see all the agents that are installed for you.
+7. Select the agent that you want to evaluate.
+8. Copy the `gpts.id` value before .declarativeAgent.
+9. This is your `agent-id`. It would look like `U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4`
+```
+
+### 3. Azure OpenAI Endpoint and API Key
+
+You need both the endpoint URL and API key from your Azure OpenAI resource for "LLM as a Judge" evaluations.
+
+**How to obtain:**
+
+1. Go to [Azure Portal](https://portal.azure.com)
+2. Navigate to your Azure OpenAI service
+   - **Path:** Portal → All Services → Search "OpenAI" → Select your resource
+   - **Or create new:** Portal → Create a resource → Search "OpenAI"
+3. In the **Overview** section, copy the **Endpoint** value
+   - Format: `https://YOUR-RESOURCE-NAME.openai.azure.com/`
+   - This is your `AZURE_AI_OPENAI_ENDPOINT`
+4. In the left sidebar, click **Keys and Endpoint**
+5. Copy **KEY 1** or **KEY 2**
+   - This is your `AZURE_AI_API_KEY`
+6. Add both values to your `.env.dev` file as shown in the [Setup Steps](#setup-steps) above
+
+**Required model:** Ensure you have `gpt-4o-mini` (or similar) deployed in your Azure OpenAI resource.
+
+**Security tip:** Store keys and endpoints securely and never commit to source control.
+
+## 🚀 Quick Start
+
+Now that you have your environment variables set up, you're ready to run evaluations!
+
+> **Important:** Run this tool FROM your M365 agent project directory (where your agent code lives), **not** from this repository. You don't need to clone or download this repo.
+
+```bash
+# Navigate to YOUR agent project directory
+cd /path/to/your-agent-project
+
+# Run evaluations (auto-discovers .env.local for ATK projects)
+runevals
+
+# Or specify an environment file
+runevals --env dev
+```
+
+**No prompts file?** If you don't have a prompts file yet, the tool will offer to create a starter file with example prompts for you.
+
+**Environment file lookup:**
+- Checks `.env.local` first (ATK projects)
+- Then checks `env/.env.{name}` if `--env {name}` is specified
+- Prompts file auto-discovery works the same for all projects
+
+
+---
+
+## 📝 Creating Prompts Files
+
+The CLI auto-discovers prompts files in your project:
+
+### Auto-Discovery
+
+When you run `runevals`, it searches:
+1. Current directory: `prompts.json`, `evals.json`, `tests.json`
+2. `./evals/` subdirectory: `prompts.json`, `evals.json`, `tests.json`
+
+**Example project structure:**
+```
+my-agent/
+├── .env.local              # Your credentials
+├── evals/
+│   └── evals.json         # Your test prompts (auto-discovered!)
+└── .evals/
+    └── 2025-12-03_14-30-45.html  # Generated reports
+```
+
+### Starter File Creation
+
+If no file is found:
+```
+⚠️  No prompts file found in current directory or ./evals/
+
+Create a starter evals file with sample prompts? (Y/n):
+```
+
+Answering "Y" creates `./evals/evals.json` with 2 starter prompts:
+
+```json
+[
+  {
+    "prompt": "What is Microsoft 365?",
+    "expected_response": "Microsoft 365 is a cloud-based productivity suite..."
+  },
+  {
+    "prompt": "How can I share a file in Teams?",
+    "expected_response": "You can share a file in Teams by uploading it..."
+  }
+]
+```
+
+Edit this file with your own prompts and run again!
+
+### Manual Creation
+
+Create `./evals/prompts.json`:
+
+```json
+[
+  {
+    "prompt": "Your test prompt here",
+    "expected_response": "Expected agent response"
+  }
+]
+```
+
+## 🎯 Usage Examples
+
+> **Remember:** All commands below assume you're running them FROM your agent project directory, **not** from this repository.
+
+### What to Expect
+
+When you run an evaluation from your agent project directory, you'll see:
+```bash
+🚀 M365 Copilot Agent Evaluations CLI
+
+📂 Loading environment: dev
+🤖 Agent ID (from M365_TITLE_ID): T_my-agent.declarativeAgent
+📄 Using prompts file: ./evals/evals.json
+
+📊 Running evaluations...
+
+─────────────────────────────────────────────────────────────
+
+✓ Evals completed successfully!
+
+Results saved to: ./evals/2025-12-03_14-30-45.html
+```
+
+**Commands to run from your project root:**
+
+```bash
+# Use .env.local (checked in current dir, then env/ folder)
+runevals
+
+# Use env/.env.dev configuration
+runevals --env dev
+
+# Use specific prompts file in your project
+runevals --prompts-file ./evals/my-tests.json
+
+# Inline prompts (no file needed, useful for quick tests)
+runevals --prompts "What is Microsoft Graph?" --expected "Gateway to M365 data"
+
+# Interactive mode (enter prompts interactively)
+runevals --interactive
+
+# Custom output location in your project
+runevals --output ./reports/results.html
+```
+
+### Optional: Add Shortcuts to package.json
+
+You can add shortcuts (npm scripts) to your agent project's `package.json`:
+
+```json
+{
+  "scripts": {
+    "eval": "runevals",
+    "eval:local": "runevals --env local",
+    "eval:dev": "runevals --env dev"
+  }
+}
+```
+
+Then use shorter commands:
+
+```bash
+# Uses .env.local (ATK default)
+npm run eval
+
+# Uses env/.env.local
+npm run eval:local
+
+# Uses env/.env.dev
+npm run eval:dev
+```
+
+**Production note:** For production environments, use CI/CD pipelines instead of local `npm run` commands. See [CICD_CACHE_GUIDE.md](CICD_CACHE_GUIDE.md) for examples.
+
+## 📊 Output Formats
+
+Results are automatically saved to `./evals/YYYY-MM-DD_HH-MM-SS.html` with:
+- **Relevance** score (1-5)
+- **Coherence** score (1-5)  
+- **Groundedness** score (1-5)
+- Per-prompt details and aggregate metrics
+
+Other formats:
+```bash
+# JSON output
+runevals --output results.json
+
+# CSV output
+runevals --output results.csv
+```
+
+## 🔧 Command Reference
+
+```bash
+Options:
+  -V, --version                 output version number
+  -v, --verbose                 show detailed processing steps
+  -q, --quiet                   minimal output
+  --prompts <prompts...>        inline prompts to evaluate
+  --expected <responses...>     expected responses (with --prompts)
+  --prompts-file <file>         JSON file with prompts
+  -o, --output <file>           output file (JSON, CSV, or HTML)
+  -i, --interactive             interactive prompt entry mode
+  --agent-id <id>               override agent ID
+  --env <environment>           environment name (default: dev)
+  --init-only                   just setup, don't run evals
+  -h, --help                    display help
+
+Cache Commands:
+  cache-info                    show cache statistics
+  cache-clear                   remove cached Python runtime
+  cache-dir                     print cache directory path
+```
+
+## ❓ Troubleshooting
+
+### Pre-cache Python Environment (Optional)
+
+If you want to set up the Python environment ahead of time without running evaluations:
+
+```bash
+runevals --init-only
+```
+
+This is useful for:
+- Pre-warming the cache in CI/CD pipelines
+- Testing the setup without running evaluations
+- Troubleshooting installation issues
+
+### Cache Issues
+```bash
+# View cache info
+runevals cache-info
+
+# Clear and rebuild
+runevals cache-clear
+runevals --init-only --verbose
+```
+
+### Network/Proxy Issues
+```bash
+# Set proxy
+export HTTPS_PROXY=http://proxy:8080
+
+# Retry with verbose output
+runevals --init-only --verbose
+```
+
+### Permission Issues
+```bash
+# Check cache directory
+runevals cache-dir
+
+# Fix permissions (Unix/macOS)
+chmod -R u+w $(runevals cache-dir)
+```
+
+## 📚 Advanced Documentation
+
+- **[CI/CD Integration](./CICD_CACHE_GUIDE.md)** - GitHub Actions, Azure DevOps caching
+- **[Testing Guide](./.github/TESTING_GUIDE.md)** - Cross-platform testing procedures
+- **[Python CLI Guide](./PYTHON_CLI.md)** - Direct Python usage (without Node.js)
+
+
+## Contributing
+
+This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
+the rights to use your contribution. For details, visit [Contributor License Agreements](https://cla.opensource.microsoft.com).
+
+When you submit a pull request, a CLA bot will automatically determine whether you need to provide
+a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
+provided by the bot. You will only need to do this once across all repos using our CLA.
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+## Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
+trademarks or logos is subject to and must follow
+[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/legal/intellectualproperty/trademarks/usage/general).
+Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
+Any use of third-party trademarks or logos are subject to those third-party's policies.

package/TERMS.txt

@@ -0,0 +1,65 @@
+MICROSOFT SOFTWARE LICENSE TERMS
+M365 Copilot Agent Eval Tool
+________________________________________
+IF YOU LIVE IN (OR ARE A BUSINESS WITH A PRINCIPAL PLACE OF BUSINESS IN) THE UNITED STATES, PLEASE READ THE “BINDING ARBITRATION AND CLASS ACTION WAIVER” SECTION BELOW. IT AFFECTS HOW DISPUTES ARE RESOLVED.
+________________________________________
+These license terms are an agreement between you and Microsoft Corporation. They apply to the software named above and any Microsoft services or software updates (except to the extent such services or updates are accompanied by new or additional terms, in which case those different terms apply prospectively and do not alter your or Microsoft’s rights relating to pre-updated software or services). IF YOU COMPLY WITH THESE LICENSE TERMS, YOU HAVE THE RIGHTS BELOW. BY USING THE SOFTWARE, YOU ACCEPT THESE TERMS.
+1.	INSTALLATION AND USE RIGHTS.
+a)	General. You may install and use any number of copies of the software on your devices.
+b)	Included Microsoft Applications. The software may include other Microsoft applications. These license terms apply to those included applications, if any, unless other license terms are provided with the other Microsoft applications.
+c)	Work or School Accounts. You can sign into the software with a work or school email address. If you do, you agree that the owner of the domain associated with your email address may control and administer your account, and access and process your data, including the contents of your communications and files. You further agree that your use of the software may be subject to: i) your organization’s guidelines and policies regarding the use of the software; and ii) the agreements Microsoft has with you or your organization, and in such case these terms may not apply.  If you already have a Microsoft account and you use a separate work or school email address to access the software, you may be prompted to update the email address associated with your Microsoft account to continue accessing the software.
+d)	Third Party Components. The software may include third party components with separate legal notices or governed by other agreements, as may be described in the ThirdPartyNotices file(s) accompanying the software.
+
+e)	Microsoft Services Agreement. Some features of the software provide access to, or rely on, online services. The use of those services (but not the software) is governed by the separate terms and privacy policies in the Microsoft Services Agreement at http://go.microsoft.com/fwlink/?linkid=398923. Please read them. The services may not be available in all regions.
+
+2.	TIME-SENSITIVE SOFTWARE.
+a)	Period. This agreement is effective on your acceptance and terminates on the earlier of (i) 30 days following first availability of a commercial release of the software or (ii) upon termination by Microsoft. Microsoft may extend this agreement in its discretion.
+b)	Notice. You may receive periodic reminder notices of this date through the software.
+c)	Access to data. You may not be able to access data used in the software when it stops running.
+3.	SCOPE OF LICENSE. The software is licensed, not sold. Microsoft reserves all other rights. Unless applicable law gives you more rights despite this limitation, you will not (and have no right to):
+a)	work around any technical limitations in the software that only allow you to use it in certain ways;
+b)	reverse engineer, decompile or disassemble the software, or otherwise attempt to derive the source code for the software, except and to the extent required by third party licensing terms governing use of certain open source components that may be included in the software;
+c)	remove, minimize, block, or modify any notices of Microsoft or its suppliers in the software;
+d)	use the software for commercial, non-profit, or revenue-generating activities unless you have commercial use rights under a separate agreement;
+e)	use the software in any way that is against the law or to create or propagate malware; or
+f)	share, publish, distribute, or lease the software, provide the software as a stand-alone offering for others to use, or transfer the software or this agreement to any third party.
+4.	PRE-RELEASE SOFTWARE. The software is a pre-release version. It may not operate correctly. It may be different from the commercially released version. Customer further acknowledges that the tool is provided in preview and that Customer’s use of any preview features is governed by the Preview Terms within the Product Terms (Microsoft Azure Legal Information | Microsoft Azure, Preview Terms Of Use | Microsoft Azure), as well as the applicable preview provisions of the Data Protection Addendum (DPA)
+5.	FEEDBACK. If you give feedback about the software to Microsoft, you give to Microsoft, without charge, the right to use, share and commercialize your feedback in any way and for any purpose. You will not give feedback that is subject to a license that requires Microsoft to license its software or documentation to third parties because Microsoft includes your feedback in them. These rights survive this agreement.
+6.	DATA. This software may interact with other Microsoft products that collect data that is transmitted to Microsoft. To learn more about how Microsoft processes personal data we collect, please see the Microsoft Privacy Statement at https://go.microsoft.com/fwlink/?LinkId=248681. Customer is solely responsible for determining what data is submitted for evaluation and for ensuring that such data is handled in compliance with applicable laws, internal policies, and contractual obligations.[A1.1]
+7.	EXPORT RESTRICTIONS. You must comply with all domestic and international export laws and regulations that apply to the software, which include restrictions on destinations, end users, and end use. For further information on export restrictions, visit https://aka.ms/exporting.
+8.	SUPPORT SERVICES. Microsoft is not obligated under this agreement to provide any support services for the software. Any support provided is “as is”, “with all faults”, and without warranty of any kind. 
+9.	UPDATES. The software may periodically check for updates, and download and install them for you. You may obtain updates only from Microsoft or authorized sources. Microsoft may need to update your system to provide you with updates. You agree to receive these automatic updates without any additional notice. Updates may not include or support all existing software features, services, or peripheral devices.
+10.	BINDING ARBITRATION AND CLASS ACTION WAIVER. This Section applies if you live in (or, if a business, your principal place of business is in) the United States.  If you and Microsoft have a dispute, you and Microsoft agree to try for 60 days to resolve it informally. If you and Microsoft can’t, you and Microsoft agree to binding individual arbitration before the American Arbitration Association under the Federal Arbitration Act (“FAA”), and not to sue in court in front of a judge or jury. Instead, a neutral arbitrator will decide. Class action lawsuits, class-wide arbitrations, private attorney-general actions, and any other proceeding where someone acts in a representative capacity are not allowed; nor is combining individual proceedings without the consent of all parties. The complete Arbitration Agreement contains more terms and is at https://aka.ms/arb-agreement-4. You and Microsoft agree to these terms.
+11.	TERMINATION. Without prejudice to any other rights, Microsoft may terminate this agreement if you fail to comply with any of its terms or conditions. In such event, you must destroy all copies of the software and all of its component parts.
+12.	ENTIRE AGREEMENT. This agreement, and any other terms Microsoft may provide for supplements, updates, or third-party applications, is the entire agreement for the software.
+13.	APPLICABLE LAW AND PLACE TO RESOLVE DISPUTES. If you acquired the software in the United States or Canada, the laws of the state or province where you live (or, if a business, where your principal place of business is located) govern the interpretation of this agreement, claims for its breach, and all other claims (including consumer protection, unfair competition, and tort claims), regardless of conflict of laws principles, except that the FAA governs everything related to arbitration. If you acquired the software in any other country, its laws apply, except that the FAA governs everything related to arbitration. If U.S. federal jurisdiction exists, you and Microsoft consent to exclusive jurisdiction and venue in the federal court in King County, Washington for all disputes heard in court (excluding arbitration). If not, you and Microsoft consent to exclusive jurisdiction and venue in the Superior Court of King County, Washington for all disputes heard in court (excluding arbitration).
+14.	CONSUMER RIGHTS; REGIONAL VARIATIONS. This agreement describes certain legal rights. You may have other rights, including consumer rights, under the laws of your state, province, or country. Separate and apart from your relationship with Microsoft, you may also have rights with respect to the party from which you acquired the software. This agreement does not change those other rights if the laws of your state, province, or country do not permit it to do so. For example, if you acquired the software in one of the below regions, or mandatory country law applies, then the following provisions apply to you:
+a)	Australia. You have statutory guarantees under the Australian Consumer Law and nothing in this agreement is intended to affect those rights.
+b)	Canada. If you acquired this software in Canada, you may stop receiving updates by turning off the automatic update feature, disconnecting your device from the Internet (if and when you re-connect to the Internet, however, the software will resume checking for and installing updates), or uninstalling the software. The product documentation, if any, may also specify how to turn off updates for your specific device or software.
+c)	Germany and Austria.
+i.	Warranty. The properly licensed software will perform substantially as described in any Microsoft materials that accompany the software. However, Microsoft gives no contractual guarantee in relation to the licensed software.
+ii.	Limitation of Liability. In case of intentional conduct, gross negligence, claims based on the Product Liability Act, as well as, in case of death or personal or physical injury, Microsoft is liable according to the statutory law.
+Subject to the foregoing clause ii., Microsoft will only be liable for slight negligence if Microsoft is in breach of such material contractual obligations, the fulfillment of which facilitate the due performance of this agreement, the breach of which would endanger the purpose of this agreement and the compliance with which a party may constantly trust in (so-called "cardinal obligations"). In other cases of slight negligence, Microsoft will not be liable for slight negligence.
+
+15.	CONFIDENTIAL INFORMATION. The software, including its user interface, features and documentation, is confidential and proprietary to Microsoft and its suppliers.
+c)	Use. For five years after installation of the software or its commercial release, whichever is first, you may not disclose confidential information to third parties. You may disclose confidential information only to your employees and consultants who need to know the information. You must have written agreements with them that protect the confidential information at least as much as this agreement.
+c)	Survival. Your duty to protect confidential information survives this agreement.
+c)	Exclusions. You may disclose confidential information in response to a judicial or governmental order. You must first give written notice to Microsoft to allow it to seek a protective order or otherwise protect the information. Confidential information does not include information that:
+1.	becomes publicly known through no wrongful act;
+2.	you received from a third party who did not breach confidentiality obligations to Microsoft or its suppliers; or
+3.	you developed independently.
+
+16.	DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED “AS IS.” YOU BEAR THE RISK OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES, OR CONDITIONS. TO THE EXTENT PERMITTED UNDER APPLICABLE LAWS, MICROSOFT EXCLUDES ALL IMPLIED WARRANTIES, INCLUDING MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
+17.	LIMITATION ON AND EXCLUSION OF DAMAGES. IF YOU HAVE ANY BASIS FOR RECOVERING DAMAGES DESPITE THE PRECEDING DISCLAIMER OF WARRANTY, YOU CAN RECOVER FROM MICROSOFT AND ITS SUPPLIERS ONLY DIRECT DAMAGES UP TO U.S. $5.00. YOU CANNOT RECOVER ANY OTHER DAMAGES, INCLUDING CONSEQUENTIAL, LOST PROFITS, SPECIAL, INDIRECT OR INCIDENTAL DAMAGES.
+This limitation applies to (a) anything related to the software, services, content (including code) on third party Internet sites, or third party applications; and (b) claims for breach of contract, warranty, guarantee, or condition; strict liability, negligence, or other tort; or any other claim; in each case to the extent permitted by applicable law.
+It also applies even if Microsoft knew or should have known about the possibility of the damages. The above limitation or exclusion may not apply to you because your state, province, or country may not allow the exclusion or limitation of incidental, consequential, or other damages.
+
+Please note: As this software is distributed in Canada, some of the clauses in this agreement are provided below in French.
+Remarque: Ce logiciel étant distribué au Canada, certaines des clauses dans ce contrat sont fournies ci-dessous en français.
+EXONÉRATION DE GARANTIE. Le logiciel visé par une licence est offert « tel quel ». Toute utilisation de ce logiciel est à votre seule risque et péril. Microsoft n’accorde aucune autre garantie expresse. Vous pouvez bénéficier de droits additionnels en vertu du droit local sur la protection des consommateurs, que ce contrat ne peut modifier. La ou elles sont permises par le droit locale, les garanties implicites de qualité marchande, d’adéquation à un usage particulier et d’absence de contrefaçon sont exclues.
+LIMITATION DES DOMMAGES-INTÉRÊTS ET EXCLUSION DE RESPONSABILITÉ POUR LES DOMMAGES. Vous pouvez obtenir de Microsoft et de ses fournisseurs une indemnisation en cas de dommages directs uniquement à hauteur de 5,00 $ US. Vous ne pouvez prétendre à aucune indemnisation pour les autres dommages, y compris les dommages spéciaux, indirects ou accessoires et pertes de bénéfices.
+Cette limitation concerne:
+•	tout ce qui est relié au logiciel, aux services ou au contenu (y compris le code) figurant sur des sites Internet tiers ou dans des programmes tiers; et
+•	les réclamations au titre de violation de contrat ou de garantie, ou au titre de responsabilité stricte, de négligence ou d’une autre faute dans la limite autorisée par la loi en vigueur.
+Elle s’applique également, même si Microsoft connaissait ou devrait connaître l’éventualité d’un tel dommage. Si votre pays n’autorise pas l’exclusion ou la limitation de responsabilité pour les dommages indirects, accessoires ou de quelque nature que ce soit, il se peut que la limitation ou l’exclusion ci-dessus ne s’appliquera pas à votre égard.
+EFFET JURIDIQUE. Le présent contrat décrit certains droits juridiques. Vous pourriez avoir d’autres droits prévus par les lois de votre pays. Le présent contrat ne modifie pas les droits que vous confèrent les lois de votre pays si celles-ci ne le permettent pas.

package/package.json

@@ -0,0 +1,82 @@
+{
+  "name": "@microsoft/m365-copilot-eval",
+  "version": "1.0.1-preview.1",
+  "description": "Zero-config Node.js wrapper for M365 Copilot Agent Evaluations CLI (Python-based Azure AI Evaluation SDK)",
+  "publishDate": "2026-01-21",
+  "main": "src/clients/node-js/lib/index.js",
+  "type": "module",
+  "bin": {
+    "runevals": "./src/clients/node-js/bin/runevals.js"
+  },
+  "scripts": {
+    "postinstall": "node -e \"console.log('By installing this package, you agree to:\\n'); require('fs').readFile('./TERMS.txt', 'utf8', (err, content) => { if (!err) console.log(content); else console.error('Read about the Terms for M365 Copilot Agent Eval Tool here - aka.ms/evaltoolterms'); });\"",
+    "prebuild": "node scripts/generate-defaults.js",
+    "build": "npm run prettier:check && npm run clean && npm run lint",
+    "clean": "rimraf node_modules/.cache dist coverage",
+    "test": "node --test tests/clients/node-js/**/*.test.js",
+    "set-publish-date": "node scripts/set-publish-date.js",
+    "prepublishOnly": "node scripts/set-publish-date.js",
+    "prettier:base": "prettier --parser babel",
+    "prettier:check": "npm run prettier:base -- --check \"src/clients/node-js/**/*.js\"",
+    "prettier:write": "npm run prettier:base -- --write \"src/clients/node-js/**/*.js\"",
+    "lint": "npm run lint:eslint",
+    "lint:eslint": "eslint --quiet \"src/clients/node-js/**/*.js\"",
+    "lint:eslint:fix": "npm run lint:eslint -- --fix",
+    "lint:eslint:loud": "eslint \"src/clients/node-js/**/*.js\"",
+    "prepare": "husky"
+  },
+  "keywords": [
+    "m365",
+    "copilot",
+    "evaluation",
+    "azure-ai",
+    "agent",
+    "python-wrapper"
+  ],
+  "author": "Microsoft",
+  "license": "MICROSOFT SOFTWARE LICENSE TERMS",
+  "engines": {
+    "node": ">=22.21.1"
+  },
+  "dependencies": {
+    "commander": "^12.1.0",
+    "node-fetch": "^3.3.2",
+    "tar": "^7.4.3",
+    "https-proxy-agent": "^7.0.5"
+  },
+  "devDependencies": {
+    "@microsoft/eslint-config-msgraph": "^5.0.0",
+    "@tony.ganchev/eslint-plugin-header": "^3.1.2",
+    "@vitest/browser": "^3.0.0",
+    "@vitest/coverage-istanbul": "^3.0.0",
+    "@vitest/coverage-v8": "^3.0.0",
+    "@vitest/ui": "^3.0.0",
+    "eslint": "^9.7.0",
+    "eslint-config-prettier": "^10.0.0",
+    "eslint-plugin-jsdoc": "^50.1.0",
+    "eslint-plugin-prefer-arrow": "^1.2.3",
+    "eslint-plugin-prettier": "^5.1.3",
+    "eslint-plugin-simple-import-sort": "^12.0.0",
+    "husky": "^9.0.11",
+    "lerna": "^9.0.3",
+    "prettier": "^3.2.4",
+    "rimraf": "^6.0.0",
+    "sinon": "^21.0.1",
+    "vitest": "^3.0.0",
+    "webdriverio": "^9.0.1"
+  },
+  "files": [
+    "src/clients/node-js/bin/",
+    "src/clients/node-js/lib/",
+    "src/clients/node-js/config/",
+    "src/clients/cli/**/*.py",
+    "src/clients/cli/requirements.txt",
+    "src/clients/cli/samples/",
+    "README.md",
+    "TERMS.txt"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/microsoft/M365-Copilot-Agent-Evals.git"
+  }
+}

package/src/clients/cli/auth/__init__.py

@@ -0,0 +1 @@
+# Authentication module for token acquisition