testronaut 1.1.1 → 1.2.0

package/CONTRIBUTING.md

@@ -0,0 +1,194 @@
+# Contributing to 🧑‍🚀 Testronaut
+
+Thanks for your interest in contributing to 🧑‍🚀 **Testronaut**! 🚀  
+This document outlines how to set up your environment, run tests, and write code and documentation that align with the project’s standards.
+
+---
+
+## 🧭 Project Overview
+
+🧑‍🚀 **Testronaut** is an open-source CLI for autonomous end-to-end testing.  
+It provides mission-driven testing flows, LLM provider integrations (OpenAI, Gemini, etc.), and a growing ecosystem of helpers and automation tools.
+
+This guide applies to all contributions — whether you're fixing a bug, improving documentation, or adding new features.
+
+---
+
+## 🧩 Local Development Setup
+
+1. **Clone the repository**
+    ```bash
+    git clone https://github.com/<your-org-or-user>/testronaut.git
+    cd testronaut
+    ```
+    Install dependencies
+    ```bash
+    npm install
+    ```
+    Run the CLI locally
+
+    ```bash
+    node ./bin/testronaut.js init
+    ```
+    Run tests
+    ```bash
+    npm test
+    # or watch mode
+    npm run test:watch
+    ```
+
+
+## 🧪 Testing Guidelines
+  We use [Vitest](https://vitest.dev/) for unit testing.
+
+### Test Structure
+- Unit tests → tests/initHelpersTests/ and similar subfolders
+  - Pure, no side effects
+  - Target helpers and utilities directly
+  - Fast and deterministic
+
+
+
+### Commands
+  ```bash
+  # Run all tests
+  npm test
+
+  # Watch mode
+  npm run test:watch
+
+  # Coverage report
+  npx vitest run --coverage
+  ```
+
+### Principles
+- Each helper or core module should have at least one test file.
+- Tests should verify behavior, not implementation details.
+- Prefer fixtures over mocks when possible.
+
+## 💬 Comment & Documentation Standards
+Testronaut uses structured, meaningful comments to make the codebase self-documenting.
+
+### 1. File Headers
+Every source file should start with a doc header that summarizes:
+
+- Purpose: What the file is responsible for
+- Responsibilities: Major actions or logic areas
+- Side effects: I/O or network operations
+- Related tests: Folder or suite where its tests live
+- Used by: Other modules that import it
+
+Example:
+```js
+
+/**
+ * ExampleModule.js
+ * ----------------
+ * Purpose:
+ *   Handles mission parsing and validation for CLI runs.
+ *
+ * Design goals:
+ *   - Pure functions where possible
+ *   - Unit-tested via Vitest
+ *
+ * Related tests:
+ *   Located in `tests/exampleModuleTests/`
+ *
+ * Used by:
+ *   ./cli/runMission.js
+ */
+```
+
+### 2. Inline Comments
+- Focus on why rather than what the code is doing.
+- Use NOTE: or TODO: prefixes to highlight rationale or future work.
+- Avoid restating code syntax (“// read file”).
+
+Example:
+```js
+// NOTE: We skip writing .env if it already exists to avoid overwriting secrets.
+```
+
+### 3. Section Headers
+Use clear dividers for major logical steps in longer scripts (e.g., CLI commands):
+```js
+// ─────────────────────────────
+// STEP 3: Merge defaults safely
+// ─────────────────────────────
+```
+
+### 4. Function Documentation (JSDoc)
+Add JSDoc blocks for all exported functions and for internal ones with nontrivial logic:
+```js
+/**
+ * Generates default configuration for new projects.
+ *
+ * @param {string} rootBasename - The current directory name.
+ * @returns {{initialized: boolean, outputDir: string}}
+ */
+export function defaultConfig(rootBasename) { ... }
+```
+These comments improve IDE autocompletion and make code self-descriptive.
+
+### 5. Test References
+If a file has full coverage:
+- Add one Related tests: line at the top (e.g., tests/initHelpersTests/).
+- Don’t link to individual test files unless the logic is unusually complex.
+
+## 🧱 Code Style
+- Language: Modern JavaScript (ES Modules only)
+- Linting: Use ESLint + Prettier (follow repo config)
+- Imports: Order as → Node built-ins → Third-party → Local
+- Async: Prefer async/await over callbacks
+- File Names: Use camelCase for utilities, PascalCase for React components (if applicable)
+
+## 🧰 Folder Conventions
+| Folder | Purpose |
+|---|---|
+/bin/	| CLI logic, utilities, helpers
+/core/ | central functionality, turn loop, and agent management
+/tests/	| All Vitest unit tests
+/runner/	| Mission runner and suite management
+/openAI/ |	OpenAI specific helper functions
+
+## ⚙️ Submitting Changes
+1. Create a new branch
+```bash
+git checkout -b feat/new-provider
+```
+
+2. Make your changes
+- Keep commits focused and descriptive
+- Follow documentation and testing standards
+
+3. Run 
+```bash
+npm test
+```
+
+4. Open a Pull Request
+- Reference related issues if applicable
+- Summarize what was added, fixed, or refactored
+
+## 🧑‍💻 Example of a “Good” PR Description
+> ### Summary
+> Adds Gemini provider support to the CLI init flow.
+> Includes unit tests for new helper functions in initHelpers.js.
+>
+> ### Details
+> - Added new prompt for selecting Gemini model
+> - Updated .env scaffold to include GEMINI_API_KEY
+> - Added tests in tests/initHelpersTests/makeEnvTemplate.test.js
+>
+> ### Checklist
+> - Unit tests pass (npm test)
+> - Comments and headers added
+> - No regressions in CLI behavior
+
+## 🤝 Code of Conduct
+We expect all contributors to follow the Code of Conduct.
+Be respectful, collaborative, and kind — we’re all here to build something great together.
+
+## 🧡 Thank You
+Your contributions make 🧑‍🚀 Testronaut better for everyone — from developers automating their first mission to teams running autonomous test fleets.
+We deeply appreciate your time, effort, and ideas.

package/bin/cli.js

@@ -104,7 +104,7 @@ if (args.includes('--init')) {
 
 Next steps:
   1. Get an API key from your AI provider
-  2. Add it to your .env file as OPENAI_API_KEY
+  2. Add it to your .env file
   3. Run your first mission:
        npx testronaut
 

package/bin/init.js

@@ -1,118 +1,202 @@
+/**
+ * Testronaut CLI Initialization
+ * -----------------------------
+ * Purpose:
+ *   Bootstraps a new Testronaut project by creating required folders,
+ *   writing a provider/model-aware config, and scaffolding a .env file.
+ *
+ * Responsibilities:
+ *   1) Ask the user for an LLM provider (OpenAI or Gemini).
+ *   2) Ask for a provider-specific model (keeps prior choice when re-run).
+ *   3) Write `testronaut-config.json` and an initial `.env` placeholder.
+ *   4) Ensure folder structure and create a welcome mission.
+ *
+ * Side effects:
+ *   - Writes files to disk: `testronaut-config.json`, `.env`,
+ *     `missions/`, `missions/mission_reports/`.
+ *   - Calls `createWelcomeMission()` on first init.
+ *
+ * Developer notes:
+ *   - Extend provider support by adding a new provider choice here and
+ *     updating `initHelpers.js` (model list + template).
+ *   - Keep this file orchestration-only; pure logic lives in helpers
+ *     (which are unit-tested).
+ */
+
 import fs from 'fs';
 import path from 'path';
 import prompts from 'prompts';
 import { createWelcomeMission } from './createWelcomeMission.js';
 
+import {
+  migrateLegacyConfig,
+  defaultConfig,
+  makeEnvTemplate,
+  openAIModels,
+  geminiModels,
+  pickInitialIndex
+} from './initHelpers.js';
+
 export async function initializeTestronautProject() {
+  // ─────────────────────────────────────────────
+  // STEP 0: Resolve important paths
+  // ─────────────────────────────────────────────
   const root = process.cwd();
   const missionsDir = path.join(root, 'missions');
   const reportsDir = path.join(missionsDir, 'mission_reports');
   const configPath = path.join(root, 'testronaut-config.json');
   const envPath = path.join(root, '.env');
 
+  // ─────────────────────────────────────────────
+  // STEP 1: Load existing config (if any) and normalize
+  // ─────────────────────────────────────────────
+  /** @type {any} */
   let config = {};
   if (fs.existsSync(configPath)) {
     config = JSON.parse(fs.readFileSync(configPath, 'utf-8'));
-
-    // --- Back-compat migration: old configs stored provider in `model`
-    if (!config.provider && config.model === 'openai') {
-      config.provider = 'openai';
-      // don't overwrite model yet; we'll re-ask below only if not initialized
-    }
+    // Normalize legacy structures (e.g., provider implied by model).
+    config = migrateLegacyConfig(config);
   }
 
-  // If not initialized before, walk through setup
+  // ─────────────────────────────────────────────
+  // STEP 2: Interactive setup (first run only)
+  // ─────────────────────────────────────────────
   if (!config.initialized) {
-    // 1) Provider
+    // 2a) Provider selection
     const { llmProvider } = await prompts({
       type: 'select',
       name: 'llmProvider',
       message: 'Which LLM provider would you like to use?',
       choices: [
         { title: 'OpenAI (GPT family)', value: 'openai' },
-        // Add others like Anthropic, Mistral, etc. later
-      ]
-    });
-
-    config.provider = llmProvider; // store provider explicitly
-
-    // 2) If OpenAI, ask for model
-    const { openaiModel } = await prompts({
-      type: 'select',
-      name: 'openaiModel',
-      message: 'Select an OpenAI model for agentic workflows (function/tool calling):',
-      choices: [
-        { title: 'GPT-4.1 (general purpose)', value: 'gpt-4.1' },
-        { title: 'GPT-4.1 mini (faster, cheaper)', value: 'gpt-4.1-mini' },
-        { title: 'GPT-4o (multimodal, tool use)', value: 'gpt-4o' },
-        { title: 'GPT-4o mini (speed/cost optimized)', value: 'gpt-4o-mini' },
-        { title: 'o3 (reasoning w/ native tool use)', value: 'o3' },
-        { title: 'o4-mini (reasoning, cost-effective)', value: 'o4-mini' },
-
-        // 👇 GPT-5 family
-        { title: 'GPT-5 (latest reasoning model)', value: 'gpt-5' },
-        { title: 'GPT-5 mini (faster)', value: 'gpt-5-mini' },
-        { title: 'GPT-5 nano (lightweight)', value: 'gpt-5-nano' },
+        { title: 'Google Gemini', value: 'gemini' },
+        // Future: add Anthropic, Mistral, etc.
       ],
-      initial: 1
+      // If user re-runs init before writing config, preselect known value.
+      initial: ['openai', 'gemini'].indexOf(config.provider ?? 'openai')
     });
 
-    config.model = openaiModel;
+    // Persist provider explicitly to config.
+    config.provider = llmProvider;
 
-    // Warn about GPT-5 availability
-    if (openaiModel.startsWith('gpt-5')) {
-      console.log(`
-    ⚠️  Warning: GPT-5 support may not be available for all users.
-      - Access depends on your OpenAI account/region
-      - Some capabilities may be gated or rate-limited
-      `);
+    // 2b) Provider-specific model selection
+    if (llmProvider === 'openai') {
+      const models = openAIModels();
+      const { openaiModel } = await prompts({
+        type: 'select',
+        name: 'openaiModel',
+        message: 'Select an OpenAI model for agentic workflows (function/tool calling):',
+        choices: [
+          { title: 'GPT-4.1 (general purpose)', value: 'gpt-4.1' },
+          { title: 'GPT-4.1 mini (faster, cheaper)', value: 'gpt-4.1-mini' },
+          { title: 'GPT-4o (multimodal, tool use)', value: 'gpt-4o' },
+          { title: 'GPT-4o mini (speed/cost optimized)', value: 'gpt-4o-mini' },
+          { title: 'o3 (reasoning w/ native tool use)', value: 'o3' },
+          { title: 'o4-mini (reasoning, cost-effective)', value: 'o4-mini' },
+          // GPT-5 family (availability varies)
+          { title: 'GPT-5 (latest reasoning model)', value: 'gpt-5' },
+          { title: 'GPT-5 mini (faster)', value: 'gpt-5-mini' },
+          { title: 'GPT-5 nano (lightweight)', value: 'gpt-5-nano' },
+        ],
+        // Keep prior selection highlighted if present; fallback to 4.1-mini.
+        initial: pickInitialIndex(models, config.model, 'gpt-4.1-mini')
+      });
+
+      config.model = openaiModel;
+
+      // Helpful heads-up: GPT-5 access varies by account/region.
+      if (openaiModel?.startsWith('gpt-5')) {
+        console.log(`
+⚠️  Warning: GPT-5 support may not be available for all users.
+    - Access depends on your OpenAI account/region
+    - Some capabilities may be gated or rate-limited
+`);
+      }
+    } else if (llmProvider === 'gemini') {
+      const models = geminiModels();
+      const { geminiModel } = await prompts({
+        type: 'select',
+        name: 'geminiModel',
+        message: 'Select a Google Gemini model:',
+        choices: [
+          { title: 'Gemini 2.5 Pro (general, high quality)', value: 'gemini-2.5-pro' },
+          { title: 'Gemini 2.5 Flash (fast, cost-efficient)', value: 'gemini-2.5-flash' },
+          { title: 'Gemini 2.5 Flash-8B (lightweight)', value: 'gemini-2.5-flash-8b' },
+        ],
+        // Keep prior selection if present; default to fast/cost-effective Flash.
+        initial: pickInitialIndex(models, config.model, 'gemini-2.5-flash')
+      });
+
+      config.model = geminiModel;
     }
 
-    // Default config fields
-    config.initialized = true;
-    config.outputDir = config.outputDir || 'missions/mission_reports';
-    config.projectName = config.projectName || path.basename(root);
-    config.maxTurns = config.maxTurns ?? 20;
-
+    // ─────────────────────────────────────────────
+    // STEP 3: Merge defaults without clobbering user-provided fields
+    // ─────────────────────────────────────────────
+    const defaults = defaultConfig(config.projectName || path.basename(root));
+    config.initialized = true; // mark as initialized so subsequent runs skip prompts
+    config.outputDir = config.outputDir || defaults.outputDir;
+    config.projectName = config.projectName || defaults.projectName;
+    config.maxTurns = config.maxTurns ?? defaults.maxTurns;
+
+    // ─────────────────────────────────────────────
+    // STEP 4: Persist config and scaffold .env (first run only)
+    // ─────────────────────────────────────────────
     fs.writeFileSync(configPath, JSON.stringify(config, null, 2));
     console.log('📝 Created testronaut-config.json');
 
-    if (llmProvider === 'openai') {
-      // Display .env instructions for OpenAI
-      if (!fs.existsSync(envPath)) {
-        const exampleEnv = `# Add your OpenAI API key below
-OPENAI_API_KEY=sk-...
-`;
+    // Only create .env if it doesn't exist to avoid overwriting secrets.
+    if (!fs.existsSync(envPath)) {
+      const exampleEnv = makeEnvTemplate(config.provider);
+      if (exampleEnv) {
         fs.writeFileSync(envPath, exampleEnv);
         console.log('🔐 Created .env file');
       }
+    }
 
+    // Provider-specific post-setup messages: quick path to first run.
+    if (config.provider === 'openai') {
       console.log(`
 🔧 Setup Complete!
 
-👉 Please ensure your .env file contains your OpenAI API key like this:
+👉 Ensure your .env file contains your OpenAI key:
 
     OPENAI_API_KEY=sk-...
 
-📌 The key must have permissions to call the OpenAI API.
+✅ Ready to write missions in \`missions/\`. Try:
+
+    testronaut welcome.mission.js
+`);
+    } else if (config.provider === 'gemini') {
+      console.log(`
+🔧 Setup Complete!
+
+👉 Ensure your .env file contains your Gemini key:
 
-✅ You’re now ready to write missions in \`missions/\` and run your first mission using:
+    GEMINI_API_KEY=AIza...
+
+📌 If you plan to use tools/function-calling or images, make sure your account has access to those features.
+
+✅ Ready to write missions in \`missions/\`. Try:
 
     testronaut welcome.mission.js
 `);
     }
 
+    // Create a small starter mission for a smooth first run.
     createWelcomeMission();
   } else {
+    // Already initialized: keep init idempotent and non-destructive.
     console.log('🔁 Already initialized. Skipping setup.');
   }
 
-  // Create directories if they don't exist
+  // ─────────────────────────────────────────────
+  // STEP 5: Ensure folder structure exists (safe to call every run)
+  // ─────────────────────────────────────────────
   if (!fs.existsSync(missionsDir)) {
     fs.mkdirSync(missionsDir);
     console.log('📁 Created missions/');
   }
-
   if (!fs.existsSync(reportsDir)) {
     fs.mkdirSync(reportsDir);
     console.log('📁 Created missions/mission_reports/');

package/bin/initHelpers.js

@@ -0,0 +1,140 @@
+/**
+ * initHelpers.js
+ * ---------------
+ * Purpose:
+ *   Collection of pure, unit-testable helper functions used by the Testronaut
+ *   CLI initialization process. These handle config normalization, defaults,
+ *   provider-specific templates, and list-based selections.
+ *
+ * Design goals:
+ *   - No side effects (no file system or prompts)
+ *   - All logic easily testable via Vitest
+ *
+ * Related tests:
+ *   Located in `tests/binTests/`
+ *   Each helper has corresponding unit tests (e.g., `migrateLegacyConfig.test.js`,
+ *   `makeEnvTemplate.test.js`, `pickInitialIndex.test.js`) that validate expected
+ *   behavior and regression safety.
+ *
+ * Used by:
+ *   ./init.js
+ *
+ * Example usage:
+ *   const defaults = defaultConfig(path.basename(process.cwd()));
+ *   const envTemplate = makeEnvTemplate('gemini');
+ *   const models = openAIModels();
+ */
+
+/**
+ * Ensures backward compatibility for older configs where
+ * provider information may have been implied by the model field.
+ *
+ * @param {object} cfg - Parsed testronaut-config.json object
+ * @returns {object} - Updated config with normalized provider field
+ */
+export function migrateLegacyConfig(cfg) {
+  const out = { ...cfg };
+  if (!out.provider && out.model === 'openai') {
+    out.provider = 'openai';
+  }
+  return out;
+}
+
+/**
+ * Generates default configuration values for a new project.
+ *
+ * @param {string} rootBasename - Name of the current working directory
+ * @returns {{initialized: boolean, outputDir: string, projectName: string, maxTurns: number}}
+ */
+export function defaultConfig(rootBasename) {
+  return {
+    initialized: true,
+    outputDir: 'missions/mission_reports',
+    projectName: rootBasename,
+    maxTurns: 20,
+  };
+}
+
+/**
+ * Returns a provider-specific .env template string.
+ * This scaffolds a safe placeholder API key for the user to fill in manually.
+ *
+ * @param {'openai'|'gemini'} provider - The chosen LLM provider
+ * @returns {string} A multiline template for the .env file
+ */
+export function makeEnvTemplate(provider) {
+  if (provider === 'openai') {
+    return `# Add your OpenAI API key below
+OPENAI_API_KEY=sk-...
+`;
+  }
+  if (provider === 'gemini') {
+    return `# Add your Google Gemini API key below
+GEMINI_API_KEY=AIza...
+`;
+  }
+  return '';
+}
+
+/**
+ * List of supported OpenAI model identifiers.
+ *
+ * @returns {string[]} Array of OpenAI model names
+ */
+export function openAIModels() {
+  return [
+    'gpt-4.1',
+    'gpt-4.1-mini',
+    'gpt-4o',
+    'gpt-4o-mini',
+    'o3',
+    'o4-mini',
+    'gpt-5',
+    'gpt-5-mini',
+    'gpt-5-nano',
+  ];
+}
+
+/**
+ * List of supported Google Gemini model identifiers.
+ *
+ * @returns {string[]} Array of Gemini model names
+ */
+export function geminiModels() {
+  return [
+    'gemini-2.5-pro',        // best reasoning, tool use
+    'gemini-2.5-flash',      // fast, cost-effective
+    'gemini-2.5-flash-lite', // lightweight / high throughput
+  ];
+}
+
+/**
+ * Determines whether a given model name belongs to a known provider.
+ *
+ * @param {'openai'|'gemini'} provider - The LLM provider
+ * @param {string} model - The model name to check
+ * @returns {boolean} True if the model is recognized for that provider
+ */
+export function isKnownModel(provider, model) {
+  return provider === 'openai'
+    ? openAIModels().includes(model)
+    : geminiModels().includes(model);
+}
+
+/**
+ * Finds the initial selection index for a list-based UI (e.g. prompts.js select).
+ * Prefers the current value if found, otherwise falls back to a known default,
+ * and finally to the first item if neither is present.
+ *
+ * @template T
+ * @param {T[]} list - Array of available options
+ * @param {T} current - Currently selected value (if any)
+ * @param {T} fallback - Fallback value if current is missing
+ * @returns {number} The index to use as `initial` in a selection prompt
+ */
+export function pickInitialIndex(list, current, fallback) {
+  const idx = list.indexOf(current);
+  if (idx >= 0) return idx;
+  const fb = list.indexOf(fallback);
+  return fb >= 0 ? fb : 0;
+}

package/core/agent.js

@@ -49,7 +49,7 @@ export async function runAgent(goals, missionName, maxTurns = 20, ) {
         missionName,
         submissionType: goal.submissionType || goal.label,
         submissionName: goal.submissionName || null,
-        status: result.success ? 'passed' : 'failed',
+        status: result?.success ? 'passed' : 'failed',
         steps: JSON.parse(JSON.stringify(steps)), // 👈 deep-clone to avoid refs
         startTime: Date.now() - 1, // set real values if you have them
         endTime: Date.now(),