@microsoft/m365-copilot-eval 1.0.1-preview.1 → 1.1.0-preview.1

package/LICENSE

@@ -1,21 +1,65 @@
-    MIT License
+MICROSOFT SOFTWARE LICENSE TERMS
+M365 Copilot Agent Eval Tool
+________________________________________
+IF YOU LIVE IN (OR ARE A BUSINESS WITH A PRINCIPAL PLACE OF BUSINESS IN) THE UNITED STATES, PLEASE READ THE “BINDING ARBITRATION AND CLASS ACTION WAIVER” SECTION BELOW. IT AFFECTS HOW DISPUTES ARE RESOLVED.
+________________________________________
+These license terms are an agreement between you and Microsoft Corporation. They apply to the software named above and any Microsoft services or software updates (except to the extent such services or updates are accompanied by new or additional terms, in which case those different terms apply prospectively and do not alter your or Microsoft’s rights relating to pre-updated software or services). IF YOU COMPLY WITH THESE LICENSE TERMS, YOU HAVE THE RIGHTS BELOW. BY USING THE SOFTWARE, YOU ACCEPT THESE TERMS.
+1.	INSTALLATION AND USE RIGHTS.
+a)	General. You may install and use any number of copies of the software on your devices.
+b)	Included Microsoft Applications. The software may include other Microsoft applications. These license terms apply to those included applications, if any, unless other license terms are provided with the other Microsoft applications.
+c)	Work or School Accounts. You can sign into the software with a work or school email address. If you do, you agree that the owner of the domain associated with your email address may control and administer your account, and access and process your data, including the contents of your communications and files. You further agree that your use of the software may be subject to: i) your organization’s guidelines and policies regarding the use of the software; and ii) the agreements Microsoft has with you or your organization, and in such case these terms may not apply.  If you already have a Microsoft account and you use a separate work or school email address to access the software, you may be prompted to update the email address associated with your Microsoft account to continue accessing the software.
+d)	Third Party Components. The software may include third party components with separate legal notices or governed by other agreements, as may be described in the ThirdPartyNotices file(s) accompanying the software.
 
-    Copyright (c) Microsoft Corporation.
+e)	Microsoft Services Agreement. Some features of the software provide access to, or rely on, online services. The use of those services (but not the software) is governed by the separate terms and privacy policies in the Microsoft Services Agreement at http://go.microsoft.com/fwlink/?linkid=398923. Please read them. The services may not be available in all regions.
 
-    Permission is hereby granted, free of charge, to any person obtaining a copy
-    of this software and associated documentation files (the "Software"), to deal
-    in the Software without restriction, including without limitation the rights
-    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-    copies of the Software, and to permit persons to whom the Software is
-    furnished to do so, subject to the following conditions:
+2.	TIME-SENSITIVE SOFTWARE.
+a)	Period. This agreement is effective on your acceptance and terminates on the earlier of (i) 30 days following first availability of a commercial release of the software or (ii) upon termination by Microsoft. Microsoft may extend this agreement in its discretion.
+b)	Notice. You may receive periodic reminder notices of this date through the software.
+c)	Access to data. You may not be able to access data used in the software when it stops running.
+3.	SCOPE OF LICENSE. The software is licensed, not sold. Microsoft reserves all other rights. Unless applicable law gives you more rights despite this limitation, you will not (and have no right to):
+a)	work around any technical limitations in the software that only allow you to use it in certain ways;
+b)	reverse engineer, decompile or disassemble the software, or otherwise attempt to derive the source code for the software, except and to the extent required by third party licensing terms governing use of certain open source components that may be included in the software;
+c)	remove, minimize, block, or modify any notices of Microsoft or its suppliers in the software;
+d)	use the software for commercial, non-profit, or revenue-generating activities unless you have commercial use rights under a separate agreement;
+e)	use the software in any way that is against the law or to create or propagate malware; or
+f)	share, publish, distribute, or lease the software, provide the software as a stand-alone offering for others to use, or transfer the software or this agreement to any third party.
+4.	PRE-RELEASE SOFTWARE. The software is a pre-release version. It may not operate correctly. It may be different from the commercially released version. Customer further acknowledges that the tool is provided in preview and that Customer’s use of any preview features is governed by the Preview Terms within the Product Terms (Microsoft Azure Legal Information | Microsoft Azure, Preview Terms Of Use | Microsoft Azure), as well as the applicable preview provisions of the Data Protection Addendum (DPA)
+5.	FEEDBACK. If you give feedback about the software to Microsoft, you give to Microsoft, without charge, the right to use, share and commercialize your feedback in any way and for any purpose. You will not give feedback that is subject to a license that requires Microsoft to license its software or documentation to third parties because Microsoft includes your feedback in them. These rights survive this agreement.
+6.	DATA. This software may interact with other Microsoft products that collect data that is transmitted to Microsoft. To learn more about how Microsoft processes personal data we collect, please see the Microsoft Privacy Statement at https://go.microsoft.com/fwlink/?LinkId=248681. Customer is solely responsible for determining what data is submitted for evaluation and for ensuring that such data is handled in compliance with applicable laws, internal policies, and contractual obligations.[A1.1]
+7.	EXPORT RESTRICTIONS. You must comply with all domestic and international export laws and regulations that apply to the software, which include restrictions on destinations, end users, and end use. For further information on export restrictions, visit https://aka.ms/exporting.
+8.	SUPPORT SERVICES. Microsoft is not obligated under this agreement to provide any support services for the software. Any support provided is “as is”, “with all faults”, and without warranty of any kind. 
+9.	UPDATES. The software may periodically check for updates, and download and install them for you. You may obtain updates only from Microsoft or authorized sources. Microsoft may need to update your system to provide you with updates. You agree to receive these automatic updates without any additional notice. Updates may not include or support all existing software features, services, or peripheral devices.
+10.	BINDING ARBITRATION AND CLASS ACTION WAIVER. This Section applies if you live in (or, if a business, your principal place of business is in) the United States.  If you and Microsoft have a dispute, you and Microsoft agree to try for 60 days to resolve it informally. If you and Microsoft can’t, you and Microsoft agree to binding individual arbitration before the American Arbitration Association under the Federal Arbitration Act (“FAA”), and not to sue in court in front of a judge or jury. Instead, a neutral arbitrator will decide. Class action lawsuits, class-wide arbitrations, private attorney-general actions, and any other proceeding where someone acts in a representative capacity are not allowed; nor is combining individual proceedings without the consent of all parties. The complete Arbitration Agreement contains more terms and is at https://aka.ms/arb-agreement-4. You and Microsoft agree to these terms.
+11.	TERMINATION. Without prejudice to any other rights, Microsoft may terminate this agreement if you fail to comply with any of its terms or conditions. In such event, you must destroy all copies of the software and all of its component parts.
+12.	ENTIRE AGREEMENT. This agreement, and any other terms Microsoft may provide for supplements, updates, or third-party applications, is the entire agreement for the software.
+13.	APPLICABLE LAW AND PLACE TO RESOLVE DISPUTES. If you acquired the software in the United States or Canada, the laws of the state or province where you live (or, if a business, where your principal place of business is located) govern the interpretation of this agreement, claims for its breach, and all other claims (including consumer protection, unfair competition, and tort claims), regardless of conflict of laws principles, except that the FAA governs everything related to arbitration. If you acquired the software in any other country, its laws apply, except that the FAA governs everything related to arbitration. If U.S. federal jurisdiction exists, you and Microsoft consent to exclusive jurisdiction and venue in the federal court in King County, Washington for all disputes heard in court (excluding arbitration). If not, you and Microsoft consent to exclusive jurisdiction and venue in the Superior Court of King County, Washington for all disputes heard in court (excluding arbitration).
+14.	CONSUMER RIGHTS; REGIONAL VARIATIONS. This agreement describes certain legal rights. You may have other rights, including consumer rights, under the laws of your state, province, or country. Separate and apart from your relationship with Microsoft, you may also have rights with respect to the party from which you acquired the software. This agreement does not change those other rights if the laws of your state, province, or country do not permit it to do so. For example, if you acquired the software in one of the below regions, or mandatory country law applies, then the following provisions apply to you:
+a)	Australia. You have statutory guarantees under the Australian Consumer Law and nothing in this agreement is intended to affect those rights.
+b)	Canada. If you acquired this software in Canada, you may stop receiving updates by turning off the automatic update feature, disconnecting your device from the Internet (if and when you re-connect to the Internet, however, the software will resume checking for and installing updates), or uninstalling the software. The product documentation, if any, may also specify how to turn off updates for your specific device or software.
+c)	Germany and Austria.
+i.	Warranty. The properly licensed software will perform substantially as described in any Microsoft materials that accompany the software. However, Microsoft gives no contractual guarantee in relation to the licensed software.
+ii.	Limitation of Liability. In case of intentional conduct, gross negligence, claims based on the Product Liability Act, as well as, in case of death or personal or physical injury, Microsoft is liable according to the statutory law.
+Subject to the foregoing clause ii., Microsoft will only be liable for slight negligence if Microsoft is in breach of such material contractual obligations, the fulfillment of which facilitate the due performance of this agreement, the breach of which would endanger the purpose of this agreement and the compliance with which a party may constantly trust in (so-called "cardinal obligations"). In other cases of slight negligence, Microsoft will not be liable for slight negligence.
 
-    The above copyright notice and this permission notice shall be included in all
-    copies or substantial portions of the Software.
+15.	CONFIDENTIAL INFORMATION. The software, including its user interface, features and documentation, is confidential and proprietary to Microsoft and its suppliers.
+c)	Use. For five years after installation of the software or its commercial release, whichever is first, you may not disclose confidential information to third parties. You may disclose confidential information only to your employees and consultants who need to know the information. You must have written agreements with them that protect the confidential information at least as much as this agreement.
+c)	Survival. Your duty to protect confidential information survives this agreement.
+c)	Exclusions. You may disclose confidential information in response to a judicial or governmental order. You must first give written notice to Microsoft to allow it to seek a protective order or otherwise protect the information. Confidential information does not include information that:
+1.	becomes publicly known through no wrongful act;
+2.	you received from a third party who did not breach confidentiality obligations to Microsoft or its suppliers; or
+3.	you developed independently.
 
-    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-    SOFTWARE
+16.	DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED “AS IS.” YOU BEAR THE RISK OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES, OR CONDITIONS. TO THE EXTENT PERMITTED UNDER APPLICABLE LAWS, MICROSOFT EXCLUDES ALL IMPLIED WARRANTIES, INCLUDING MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
+17.	LIMITATION ON AND EXCLUSION OF DAMAGES. IF YOU HAVE ANY BASIS FOR RECOVERING DAMAGES DESPITE THE PRECEDING DISCLAIMER OF WARRANTY, YOU CAN RECOVER FROM MICROSOFT AND ITS SUPPLIERS ONLY DIRECT DAMAGES UP TO U.S. $5.00. YOU CANNOT RECOVER ANY OTHER DAMAGES, INCLUDING CONSEQUENTIAL, LOST PROFITS, SPECIAL, INDIRECT OR INCIDENTAL DAMAGES.
+This limitation applies to (a) anything related to the software, services, content (including code) on third party Internet sites, or third party applications; and (b) claims for breach of contract, warranty, guarantee, or condition; strict liability, negligence, or other tort; or any other claim; in each case to the extent permitted by applicable law.
+It also applies even if Microsoft knew or should have known about the possibility of the damages. The above limitation or exclusion may not apply to you because your state, province, or country may not allow the exclusion or limitation of incidental, consequential, or other damages.
+
+Please note: As this software is distributed in Canada, some of the clauses in this agreement are provided below in French.
+Remarque: Ce logiciel étant distribué au Canada, certaines des clauses dans ce contrat sont fournies ci-dessous en français.
+EXONÉRATION DE GARANTIE. Le logiciel visé par une licence est offert « tel quel ». Toute utilisation de ce logiciel est à votre seule risque et péril. Microsoft n’accorde aucune autre garantie expresse. Vous pouvez bénéficier de droits additionnels en vertu du droit local sur la protection des consommateurs, que ce contrat ne peut modifier. La ou elles sont permises par le droit locale, les garanties implicites de qualité marchande, d’adéquation à un usage particulier et d’absence de contrefaçon sont exclues.
+LIMITATION DES DOMMAGES-INTÉRÊTS ET EXCLUSION DE RESPONSABILITÉ POUR LES DOMMAGES. Vous pouvez obtenir de Microsoft et de ses fournisseurs une indemnisation en cas de dommages directs uniquement à hauteur de 5,00 $ US. Vous ne pouvez prétendre à aucune indemnisation pour les autres dommages, y compris les dommages spéciaux, indirects ou accessoires et pertes de bénéfices.
+Cette limitation concerne:
+•	tout ce qui est relié au logiciel, aux services ou au contenu (y compris le code) figurant sur des sites Internet tiers ou dans des programmes tiers; et
+•	les réclamations au titre de violation de contrat ou de garantie, ou au titre de responsabilité stricte, de négligence ou d’une autre faute dans la limite autorisée par la loi en vigueur.
+Elle s’applique également, même si Microsoft connaissait ou devrait connaître l’éventualité d’un tel dommage. Si votre pays n’autorise pas l’exclusion ou la limitation de responsabilité pour les dommages indirects, accessoires ou de quelque nature que ce soit, il se peut que la limitation ou l’exclusion ci-dessus ne s’appliquera pas à votre égard.
+EFFET JURIDIQUE. Le présent contrat décrit certains droits juridiques. Vous pourriez avoir d’autres droits prévus par les lois de votre pays. Le présent contrat ne modifie pas les droits que vous confèrent les lois de votre pays si celles-ci ne le permettent pas.

package/README.md

@@ -25,11 +25,8 @@ A **zero-configuration** CLI for evaluating M365 Copilot agents. Send prompts to
 
 ### Install the Tool
 
-1. Go to the releases → https://github.com/microsoft/M365-Copilot-Agent-Evals/releases and click on the most latest release.
-2. Click on the `Source code (tar.gz)`
-3. This should download the package to your device.
-4. Go to the folder where this tar.gz file is. This will now be your project root folder.
-5. Run `npm install -g <filename.tar.gz>` e.g., `npm install -g M365-Copilot-Agent-Evals-<version>.tar.gz`
+1. Make sure you have Node.js
+2. Run `npm install @microsoft/m365-copilot-eval`
 
 ### Setup Steps
 
@@ -105,20 +102,10 @@ Alternatively, if you have the Azure CLI installed:
 az account show --query tenantId
 ```
 
-### 2. Agent ID (Only for MSIT)
+### 2. Agent ID 
 - If you have created your agent using Agents Toolkit, then the agent-id is the M365_TITLE_ID in .env.local file
-- If you did not, then you can get your agent-id by 
-```
-1. Open aka.ms/devui in your browser
-2. Click on `Configuration`
-3. In the dialog that opens, click on `Untitled Config`
-4. Click on the `Payload` tab
-5. If you scroll down on this tab, you will see `DA (Declarative Agent)`
-6. In this dropdown, you will see all the agents that are installed for you.
-7. Select the agent that you want to evaluate.
-8. Copy the `gpts.id` value before .declarativeAgent.
-9. This is your `agent-id`. It would look like `U_0dc4a8a2-b95f-edac-91c8-d802023ec2d4`
-```
+- If you don't know your agent-id, the tool offers agent selection when you try to submit a job. The agent selection has both the name, description, agent-id so that you can select the right agent.
+
 
 ### 3. Azure OpenAI Endpoint and API Key
 
@@ -413,3 +400,9 @@ trademarks or logos is subject to and must follow
 [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/legal/intellectualproperty/trademarks/usage/general).
 Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
 Any use of third-party trademarks or logos are subject to those third-party's policies.
+
+## Terms of Use
+
+By using this tool, you agree to the [Microsoft Software License Terms](https://aka.ms/evaltoolterms).
+
+See [LICENSE](./LICENSE) for the full license text.

package/package.json

@@ -1,15 +1,14 @@
 {
   "name": "@microsoft/m365-copilot-eval",
-  "version": "1.0.1-preview.1",
+  "version": "1.1.0-preview.1",
   "description": "Zero-config Node.js wrapper for M365 Copilot Agent Evaluations CLI (Python-based Azure AI Evaluation SDK)",
-  "publishDate": "2026-01-21",
+  "publishDate": "2026-02-03",
   "main": "src/clients/node-js/lib/index.js",
   "type": "module",
   "bin": {
     "runevals": "./src/clients/node-js/bin/runevals.js"
   },
   "scripts": {
-    "postinstall": "node -e \"console.log('By installing this package, you agree to:\\n'); require('fs').readFile('./TERMS.txt', 'utf8', (err, content) => { if (!err) console.log(content); else console.error('Read about the Terms for M365 Copilot Agent Eval Tool here - aka.ms/evaltoolterms'); });\"",
     "prebuild": "node scripts/generate-defaults.js",
     "build": "npm run prettier:check && npm run clean && npm run lint",
     "clean": "rimraf node_modules/.cache dist coverage",
@@ -41,7 +40,7 @@
   "dependencies": {
     "commander": "^12.1.0",
     "node-fetch": "^3.3.2",
-    "tar": "^7.4.3",
+    "tar": "^7.5.4",
     "https-proxy-agent": "^7.0.5"
   },
   "devDependencies": {
@@ -73,7 +72,7 @@
     "src/clients/cli/requirements.txt",
     "src/clients/cli/samples/",
     "README.md",
-    "TERMS.txt"
+    "LICENSE"
   ],
   "repository": {
     "type": "git",

package/src/clients/cli/requirements.txt

@@ -3,7 +3,7 @@ azure-ai-evaluation==1.10.0
 azure-ai-projects==1.0.0
 msal[broker]>=1.34,<2
 msal-extensions>=1.3.1
-pip==25.3
+PyJWT>=2.11.0
 python-dotenv==1.1.1
 markdown==3.8.2
 promptflow>=1.18.1

package/src/clients/node-js/bin/runevals.js

@@ -8,7 +8,7 @@ import { ensurePythonRuntime, getCacheDir } from '../lib/python-runtime.js';
 import { ensureVenv, executePythonCli } from '../lib/venv-manager.js';
 import { getCacheStats, clearCache, formatBytes } from '../lib/cache-utils.js';
 import { checkPackageExpiry } from '../lib/expiry-check.js';
-import config from '../config/default.js';
+import { ProgressReporter } from '../lib/progress.js';
 
 // Check package expiry (exits if expired, warns if close to expiry)
 checkPackageExpiry();
@@ -26,11 +26,22 @@ const PYTHON_CLI_DIR = path.join(__dirname, '..', '..', 'cli');
 const MAIN_SCRIPT = path.join(PYTHON_CLI_DIR, 'main.py');
 const REQUIREMENTS_FILE = path.join(PYTHON_CLI_DIR, 'requirements.txt');
 
+/**
+ * Display usage terms notice
+ * Called before running evaluations (but not for --init-only, cache commands, or --signout)
+ * This notice MUST be displayed even in quiet mode per legal requirements (FR-006)
+ */
+function displayUsageTerms() {
+  console.log('By using this tool, you agree to the Terms of Use: https://aka.ms/evaltoolterms\n');
+}
+
 /**
  * Set default environment constants that cannot be overridden
  * This ensures these values are always set regardless of .env files
+ * Config is loaded lazily so --init-only works without build-time config
  */
-function setDefaultEnvironmentConstants() {
+async function setDefaultEnvironmentConstants() {
+  const config = (await import('../config/default.js')).default;
   process.env.M365_EVAL_CLIENT_ID = config.copilotApi.m365EvalClientId;
   process.env.COPILOT_API_ENDPOINT = config.copilotApi.copilotApiEndpoint;
   process.env.COPILOT_SCOPES = config.copilotApi.copilotScopes;
@@ -89,7 +100,7 @@ function loadEnvFile(envFilePath) {
 /**
  * Check for required environment variables and provide helpful guidance
  */
-function validateEnvironmentVariables(envName, detectedVars) {
+function validateEnvironmentVariables(envName, detectedVars, quiet = false) {
   const required = [
     { key: 'TENANT_ID', description: 'Your Tenant ID' },
     { key: 'AZURE_AI_OPENAI_ENDPOINT', description: 'Azure OpenAI endpoint URL' },
@@ -106,11 +117,12 @@ function validateEnvironmentVariables(envName, detectedVars) {
     return true; // All required vars present
   }
 
-  // Show error with helpful guidance
-  console.error('\n❌ Missing required environment variables:\n');
+  // Show error with helpful guidance (skip output in quiet mode, but still return false)
+  if (!quiet) {
+    console.error('\n❌ Missing required environment variables:\n');
 
-  const envFile = envName ? `env/.env.${envName}` : '.env.local or env/env.local';
-  console.error(`Create ${envFile} with:\n`);
+    const envFile = envName ? `env/.env.${envName}` : '.env.local or env/env.local';
+    console.error(`Create ${envFile} with:\n`);
 
   for (const req of missing) {
     console.error(`  ${req.key}="<your-${req.description.toLowerCase().replace(/\s+/g, '-')}>"`);
@@ -124,8 +136,8 @@ function validateEnvironmentVariables(envName, detectedVars) {
     }
   }
 
-  console.error(`\nThen run: npx runevals${envName ? ` --env ${envName}` : ''}\n`);
-  console.error('📖 See README.md for complete setup guide.\n');
+  console.error('\n📖 Setup guide: https://www.npmjs.com/package/@microsoft/m365-copilot-eval?activeTab=readme\n');
+  }
 
   return false;
 }
@@ -146,29 +158,36 @@ function constructAgentId(envVars) {
 
 /**
  * Initialize the Python environment (download, venv, pip install)
+ * @param {boolean} [verbose=false] - Enable verbose output
+ * @param {boolean} [quiet=false] - Suppress progress output
  */
-async function initializePythonEnvironment(verbose = false) {
+async function initializePythonEnvironment(verbose = false, quiet = false) {
+  // Create progress reporter (silent if quiet mode)
+  const reporter = new ProgressReporter({ verbose, quiet });
+  const onProgress = reporter.createCallback();
+
   try {
-    // Step 1: Ensure Python runtime is available
-    await ensurePythonRuntime(verbose);
-    
-    // Step 2: Ensure venv with dependencies is set up
-    await ensureVenv(REQUIREMENTS_FILE, verbose);
-    
+    // Step 1: Ensure Python runtime is available (handles download + extract phases)
+    await ensurePythonRuntime(verbose, onProgress);
+
+    // Step 2: Ensure venv with dependencies is set up (handles venv + deps phases)
+    await ensureVenv(REQUIREMENTS_FILE, verbose, onProgress);
+
+    // Show completion summary
+    reporter.complete();
+
   } catch (error) {
-    console.error('\n❌ Failed to initialize Python environment:');
-    console.error(error.message);
-    
+    // Error is already displayed by reporter.failPhase() via callback
     if (verbose) {
       console.error('\nFull error:', error);
     }
-    
+
     console.error('\nTroubleshooting:');
     console.error('  - Check your internet connection');
     console.error('  - If behind a proxy, set HTTP_PROXY/HTTPS_PROXY environment variables');
     console.error('  - For SSL issues, set NODE_EXTRA_CA_CERTS or PIP_CERT');
     console.error('  - Run with --verbose for detailed output');
-    
+
     process.exit(1);
   }
 }
@@ -177,11 +196,8 @@ async function initializePythonEnvironment(verbose = false) {
  * Main CLI entry point
  */
 async function main() {
-  // Set default environment constants.
-  setDefaultEnvironmentConstants();
-  
   const program = new Command();
-  
+
   program
     .name('runevals')
     .description('M365 Copilot Agent Evaluations CLI - Zero-config Python evaluation tool')
@@ -200,11 +216,11 @@ async function main() {
     .option('--cache-clear', 'clear the cache (removes Python runtime and venv)')
     .option('--cache-dir', 'print the cache directory path')
     .option('--signout', 'sign out and clear cached authentication tokens');
-  
+
   program.parse(process.argv);
   const options = program.opts();
-  
-  // Handle cache commands first (they don't need environment validation)
+
+  // Handle cache commands first (they don't need environment validation or config)
   if (options.cacheInfo) {
     console.log('🗂️  Cache Information\n');
     const stats = await getCacheStats();
@@ -245,15 +261,37 @@ async function main() {
     process.exit(0);
   }
   
-  // Load environment file if specified
+  // Initialize Python environment (do this early for --init-only)
+  // Skip env file loading for --init-only since it's not needed
+  if (!options.quiet) {
+    console.log('🚀 M365 Copilot Agent Evaluations CLI\n');
+  }
+  await initializePythonEnvironment(options.verbose, options.quiet);
+
+  // If --init-only, stop here (no config or env files needed)
+  if (options.initOnly) {
+    if (!options.quiet) {
+      console.log('\n✅ Python environment initialized successfully!\n');
+      console.log('⚠️  Note: Configure environment variables before running evaluations.');
+      console.log('📖 See README.md for complete setup guide.\n');
+    }
+    return;
+  }
+
+  // === From here on, we're running actual evals - load config and env files ===
+
+  displayUsageTerms();
+  // Load build-time config
+  await setDefaultEnvironmentConstants();
+
+  // Load environment files
   const envVars = {};
   let resolvedAgentId = options.agentId;
-  
+
   // Check for .env.local in current directory (ATK projects)
-  // First check for .env.local directly in current directory
   let localEnvPath = path.join(process.cwd(), '.env.local');
   let localEnvFound = false;
-  
+
   if (fs.existsSync(localEnvPath)) {
     if (!options.quiet && options.verbose) {
       console.log(`📂 Loading .env.local from current directory`);
@@ -262,8 +300,8 @@ async function main() {
     Object.assign(envVars, localEnvVars);
     localEnvFound = true;
   }
-  
-  // If not found, check for env.local in env subfolder of current directory
+
+  // If not found, check for env.local in env subfolder
   if (!localEnvFound) {
     localEnvPath = path.join(process.cwd(), 'env', 'env.local');
     if (fs.existsSync(localEnvPath)) {
@@ -275,12 +313,12 @@ async function main() {
       localEnvFound = true;
     }
   }
-  
+
   if (options.env) {
     // First check current directory's env folder
     let envFilePath = path.join(process.cwd(), 'env', `.env.${options.env}`);
     let envFileFound = false;
-    
+
     if (fs.existsSync(envFilePath)) {
       if (!options.quiet) {
         console.log(`📂 Loading environment: ${options.env} from current directory env folder`);
@@ -291,7 +329,7 @@ async function main() {
     } else {
       // Fallback to package's env directory
       envFilePath = path.join(__dirname, '..', 'env', `.env.${options.env}`);
-      
+
       if (fs.existsSync(envFilePath)) {
         if (!options.quiet) {
           console.log(`📂 Loading environment: ${options.env} from package env folder`);
@@ -301,7 +339,7 @@ async function main() {
         envFileFound = true;
       }
     }
-    
+
     if (envFileFound) {
       // Auto-construct agent ID if not explicitly provided
       if (!resolvedAgentId) {
@@ -317,7 +355,7 @@ async function main() {
       console.warn(`   Continuing with system environment variables...\n`);
     }
   }
-  
+
   // Auto-construct agent ID from loaded env vars if not explicitly provided
   if (!resolvedAgentId && Object.keys(envVars).length > 0) {
     resolvedAgentId = constructAgentId(envVars);
@@ -325,34 +363,23 @@ async function main() {
       console.log(`🤖 Agent ID (from M365_TITLE_ID): ${resolvedAgentId}`);
     }
   }
-  
+
   // Fallback to M365_AGENT_ID env var if still not resolved
   if (!resolvedAgentId) {
     resolvedAgentId = process.env.M365_AGENT_ID;
   }
-  
-  // Validate required environment variables (skip for init-only and cache commands)
-  if (!options.initOnly && !options.quiet) {
-    if (!validateEnvironmentVariables(options.env, envVars)) {
-      process.exit(1);
+
+  // Validate required environment variables (always validate, quiet just suppresses output)
+  if (!validateEnvironmentVariables(options.env, envVars, options.quiet)) {
+    if (options.quiet) {
+      console.error('📖 Setup guide: https://www.npmjs.com/package/@microsoft/m365-copilot-eval?activeTab=readme\n');
     }
+    process.exit(1);
   }
-  
-  // Initialize Python environment
-  console.log('🚀 M365 Copilot Agent Evaluations CLI\n');
-  await initializePythonEnvironment(options.verbose);
-  
-  // If --init-only, stop here
-  if (options.initOnly) {
-    console.log('\n✅ Python environment initialized successfully!\n');
-    console.log('⚠️  Note: Configure environment variables before running evaluations.');
-    console.log('📖 See README.md for complete setup guide.\n');
-    return;
-  }
-  
+
   // Build arguments to pass to Python CLI
   const pythonArgs = [];
-  
+
   if (options.verbose) pythonArgs.push('--verbose');
   if (options.quiet) pythonArgs.push('--quiet');
   if (options.interactive) pythonArgs.push('--interactive');

package/src/clients/node-js/config/default.js

@@ -2,7 +2,7 @@
  * Build-time injected default values
  * DO NOT EDIT - This file is auto-generated during build.
  *
- * Generated: 2026-01-21T21:10:04.863Z
+ * Generated: 2026-02-03T22:27:55.106Z
  *
  * @copyright Microsoft Corporation. All rights reserved.
  * @license MIT