testdriverai 7.2.35 → 7.2.37

package/docs/v7/aws-setup.mdx

@@ -13,6 +13,18 @@ graph LR
     B <--> C[Your AWS EC2 Instance]
 ```
 
+# Quickstart
+
+<Card 
+  title="Launch Stack" 
+  icon="aws" 
+  href="https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://v7-cloudformation-template.s3.us-east-2.amazonaws.com/cloudformation.yaml" 
+  horizontal 
+  arrow
+>
+  Deploy TestDriver infrastructure with one click
+</Card>
+
 ## Overview
 
 The setup process involves three main steps:
@@ -21,6 +33,7 @@ The setup process involves three main steps:
 2. **On-Demand Instance Spawning** — Use `spawn-runner.sh` to launch TestDriver instances when you need to run tests
 3. **CI/CD Integration** — Automate the lifecycle in GitHub Actions: spawn instance → run tests → terminate instance
 
+
 ## Prerequisites
 
 Before you begin, ensure you have:
@@ -31,7 +44,7 @@ Before you begin, ensure you have:
 - A GitHub repository for your tests
 
 <Tip>
-  The TestDriver Golden Image AMI ID is `ami-086b5b4b86d78987c`. Contact us to get access in your preferred AWS region.
+  The TestDriver Golden Image AMI ID is `ami-0504bf50fad62f312`. Contact us to get access in your preferred AWS region.
 </Tip>
 
 ## Step 1: Deploy CloudFormation Stack
@@ -43,57 +56,88 @@ Our CloudFormation template creates all the AWS infrastructure you need:
 - IAM roles and instance profiles
 - EC2 launch template for instance creation
 
-Download the template from the [TestDriver CLI repository](https://github.com/testdriverai/testdriverai/tree/main/setup/aws/cloudformation.yaml), then deploy:
-
-```bash
-aws cloudformation deploy \
-  --template-file setup/aws/cloudformation.yaml \
-  --stack-name testdriver-infrastructure \
-  --parameter-overrides \
-    ProjectTag=testdriver \
-    AllowedIngressCidr=0.0.0.0/0 \
-    InstanceType=c5.xlarge \
-    CreateKeyPair=true \
-  --capabilities CAPABILITY_IAM
-```
+<Tabs>
+  <Tab title="GUI">
+    Click the button below to launch the CloudFormation stack in your AWS Console:
+
+    <Card 
+      title="Launch Stack" 
+      icon="aws" 
+      href="https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://v7-cloudformation-template.s3.us-east-2.amazonaws.com/cloudformation.yaml" 
+      horizontal 
+      arrow
+    >
+      Deploy TestDriver infrastructure with one click
+    </Card>
+
+    Configure the stack parameters:
+    - **Stack name**: `testdriver-infrastructure` (or your preferred name)
+    - **ProjectTag**: `testdriver`
+    - **AllowedIngressCidr**: Your IP range (e.g., `203.0.113.0/24`)
+    - **InstanceType**: `c5.xlarge` (recommended)
+    - **CreateKeyPair**: `true`
+
+    <Warning>
+      **Security**: Replace `AllowedIngressCidr` with your specific IP ranges to restrict VPC access. Avoid using `0.0.0.0/0` in production.
+    </Warning>
+
+    ### Get Your Launch Template ID
+
+    After the stack creation completes, navigate to the **Outputs** tab to find your `LaunchTemplateId`:
+
+    ![Launch Template ID](/images/content/self-hosted/launchtemplateid.png)
+
+    <Tip>
+      **Save this ID** — you'll need it for spawning instances and CI configuration.
+    </Tip>
+  </Tab>
+  <Tab title="CLI">
+    Download the template from the [TestDriver CLI repository](https://github.com/testdriverai/testdriverai/blob/main/setup/aws/cloudformation.yaml), then deploy:
+
+    ```bash
+    aws cloudformation deploy \
+      --template-file setup/aws/cloudformation.yaml \
+      --stack-name testdriver-infrastructure \
+      --parameter-overrides \
+        ProjectTag=testdriver \
+        AllowedIngressCidr=0.0.0.0/0 \
+        InstanceType=c5.xlarge \
+        CreateKeyPair=true \
+      --capabilities CAPABILITY_IAM
+    ```
 
-<Warning>
-  **Security**: Replace `AllowedIngressCidr=0.0.0.0/0` with your specific IP ranges to restrict VPC access.
-</Warning>
+    <Warning>
+      **Security**: Replace `AllowedIngressCidr=0.0.0.0/0` with your specific IP ranges to restrict VPC access.
+    </Warning>
 
-### Get Your Launch Template ID
+    ### Get Your Launch Template ID
 
-After deployment completes, retrieve the launch template ID:
+    After deployment completes, retrieve the launch template ID:
 
-```bash
-aws cloudformation describe-stacks \
-  --stack-name testdriver-infrastructure \
-  --query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
-  --output text
-```
+    ```bash
+    aws cloudformation describe-stacks \
+      --stack-name testdriver-infrastructure \
+      --query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
+      --output text
+    ```
 
-<Tip>
-  **Save this ID** — you'll need it for spawning instances and CI configuration.
-</Tip>
+    <Tip>
+      **Save this ID** — you'll need it for spawning instances and CI configuration.
+    </Tip>
+  </Tab>
+</Tabs>
 
 ## Step 2: Spawn Test Instances
 
-Use the `spawn-runner.sh` script to launch instances on-demand. This script:
+Use the [`spawn-runner.sh`](https://github.com/testdriverai/testdriverai/blob/main/setup/aws/spawn-runner.sh) script to launch instances on-demand. This script:
 
 - Launches a new EC2 instance using your launch template
 - Waits for the instance to be ready
-- Completes the TestDriver handshake
+- Validates SSM connectivity
 - Returns instance details for test execution
 
 ```bash
-# Set environment variables
-export AWS_REGION=us-east-2
-export AMI_ID=ami-086b5b4b86d78987c
-export AWS_LAUNCH_TEMPLATE_ID=lt-your-template-id
-export RESOLUTION=1440x900  # Optional, default is 1440x900
-
-# Spawn an instance
-./setup/aws/spawn-runner.sh
+AWS_REGION=us-east-2 AMI_ID=ami-0504bf50fad62f312 AWS_LAUNCH_TEMPLATE_ID=lt-your-template-id sh setup/aws/spawn-runner.sh
 ```
 
 Output:
@@ -105,10 +149,28 @@ AWS_REGION=us-east-2
 
 ### Run Tests Against Your Instance
 
-With Vitest, specify the instance IP using the `TD_VM` environment variable:
+Specify the instance IP using the `TD_IP` environment variable:
 
 ```bash
-TD_API_KEY=your-api-key TD_VM=1.2.3.4 npx vitest run
+TD_API_KEY=your-api-key TD_IP=1.2.3.4 npx vitest run
+```
+
+In your test file, pass the IP to TestDriver:
+
+```javascript
+import { describe, it } from "vitest";
+import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
+
+describe("My Test", () => {
+  it("should run on self-hosted instance", async (context) => {
+    const testdriver = TestDriver(context, {
+      ip: process.env.TD_IP,
+    });
+
+    await testdriver.provision.chrome({ url: "https://example.com" });
+    // ... your test steps
+  });
+});
 ```
 
 ### Terminate Instances
@@ -176,7 +238,7 @@ jobs:
         run: npx vitest run
         env:
           TD_API_KEY: ${{ secrets.TD_API_KEY }}
-          TD_VM: ${{ steps.aws-setup.outputs.public-ip }}
+          TD_IP: ${{ steps.aws-setup.outputs.public-ip }}
 
       - name: Shutdown AWS Instance
         if: always()
@@ -198,9 +260,60 @@ jobs:
 | `AWS_SECRET_ACCESS_KEY` | AWS secret key | `wJalrXUtnFEMI/K7MDENG...` |
 | `AWS_REGION` | AWS region | `us-east-2` |
 | `AWS_LAUNCH_TEMPLATE_ID` | From CloudFormation output | `lt-07c53ce8349b958d1` |
-| `AMI_ID` | TestDriver AMI ID | `ami-086b5b4b86d78987c` |
+| `AMI_ID` | TestDriver AMI ID | `ami-0504bf50fad62f312` |
 | `TD_API_KEY` | Your TestDriver API key | From [console.testdriver.ai](https://console.testdriver.ai) |
 
+### Example Workflow
+
+For a complete real-world example with matrix-based test execution and dynamic test file discovery, see the [`self-hosted.yml`](https://github.com/testdriverai/testdriverai/blob/main/setup/aws/self-hosted.yml) workflow in the TestDriver repository.
+
+<Tip>
+  **Local Testing**: You can test your GitHub Actions workflow locally using [nektos/act](https://github.com/nektos/act). The `self-hosted.yml` workflow includes commented lines for installing the AWS CLI when running with `act`.
+</Tip>
+
+## Manual Instance Management
+
+You can manage TestDriver instances manually through the AWS Console or CLI. This is useful for debugging, development, or running tests outside of CI/CD.
+
+### Starting an Instance
+
+```bash
+aws ec2 run-instances \
+  --region us-east-2 \
+  --image-id ami-0504bf50fad62f312 \
+  --launch-template LaunchTemplateId=lt-your-template-id \
+  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=testdriver-manual}]'
+```
+
+### Stopping an Instance
+
+```bash
+aws ec2 stop-instances --region us-east-2 --instance-ids i-1234567890abcdef0
+```
+
+### Starting a Stopped Instance
+
+```bash
+aws ec2 start-instances --region us-east-2 --instance-ids i-1234567890abcdef0
+```
+
+### Terminating Instances
+
+```bash
+aws ec2 terminate-instances --region us-east-2 --instance-ids i-1234567890abcdef0
+```
+
+### Connecting to an Instance
+
+You can connect to running instances via:
+- **RDP** — Use the public IP on port 3389
+- **VNC** — Access via web browser at `http://<public-ip>:5900`
+- **AWS Console** — Use EC2 Instance Connect or Session Manager
+
+<Note>
+  Stopped instances retain their EBS volumes and can be restarted later. Terminated instances are permanently deleted. Always terminate instances when done to avoid storage costs.
+</Note>
+
 ## AMI Customization
 
 The TestDriver Golden Image comes pre-configured with:

package/interfaces/cli/lib/base.js

@@ -144,19 +144,35 @@ class BaseCommand extends Command {
     });
 
     // Handle process signals to ensure clean disconnection
-    const cleanupAndExit = () => {
-      if (this.agent?.sandbox) {
+    let isExiting = false;
+    const cleanupAndExit = async (signal) => {
+      if (isExiting) return;
+      isExiting = true;
+      
+      console.log(`\nReceived ${signal}, cleaning up...`);
+      
+      // Use the agent's exit method for proper cleanup
+      if (this.agent) {
         try {
-          this.agent.sandbox.close();
+          await this.agent.exit(true, false, false);
         } catch (err) {
-          // Ignore close errors
+          console.error("Error during cleanup:", err.message);
         }
+      } else {
+        // Fallback if no agent
+        if (this.agent?.sandbox) {
+          try {
+            this.agent.sandbox.close();
+          } catch (err) {
+            // Ignore close errors
+          }
+        }
+        process.exit(1);
       }
-      process.exit(1);
     };
 
-    process.on('SIGINT', cleanupAndExit);
-    process.on('SIGTERM', cleanupAndExit);
+    process.on('SIGINT', () => cleanupAndExit('SIGINT'));
+    process.on('SIGTERM', () => cleanupAndExit('SIGTERM'));
 
     // Handle unhandled promise rejections to prevent them from interfering with the exit flow
     // This is particularly important when JavaScript execution in VM contexts leaves dangling promises

package/interfaces/vitest-plugin.mjs

@@ -383,11 +383,23 @@ export async function cleanupTestDriver(testdriver) {
  * Handle process termination and mark test run as cancelled
  */
 async function handleProcessExit() {
+  logger.debug("handleProcessExit called");
+  logger.debug("testRun:", !!pluginState.testRun);
+  logger.debug("testRunId:", pluginState.testRunId);
+  logger.debug("testRunCompleted:", pluginState.testRunCompleted);
+  
   if (!pluginState.testRun || !pluginState.testRunId) {
+    logger.debug("No test run to cancel - skipping cleanup");
+    return;
+  }
+
+  // Prevent duplicate completion
+  if (pluginState.testRunCompleted) {
+    logger.debug("Test run already completed - skipping cancellation");
     return;
   }
 
-  logger.debug("Process interrupted, marking test run as cancelled...");
+  logger.debug("Marking test run as cancelled...");
 
   try {
     const stats = {
@@ -413,8 +425,10 @@ async function handleProcessExit() {
       completeData.platform = platform;
     }
 
+    logger.debug("Calling completeTestRun with:", JSON.stringify(completeData));
     await completeTestRun(completeData);
-    logger.debug("✅ Test run marked as cancelled");
+    pluginState.testRunCompleted = true;
+    logger.info("Test run marked as cancelled");
   } catch (error) {
     logger.error("Failed to mark test run as cancelled:", error.message);
   }
@@ -422,21 +436,77 @@ async function handleProcessExit() {
 
 // Set up process exit handlers
 let exitHandlersRegistered = false;
+let isExiting = false;
+let isCancelling = false; // Track if we're in the process of cancelling due to SIGINT/SIGTERM
 
 function registerExitHandlers() {
   if (exitHandlersRegistered) return;
   exitHandlersRegistered = true;
 
-  // Handle Ctrl+C
-  process.on("SIGINT", async () => {
-    await handleProcessExit();
-    process.exit(130); // Standard exit code for SIGINT
+  // Handle Ctrl+C - use 'once' and prepend to run before Vitest's handler
+  process.prependOnceListener("SIGINT", () => {
+    logger.debug("SIGINT received, cleaning up...");
+    if (isExiting) {
+      logger.debug("Already exiting, skipping duplicate handler");
+      return;
+    }
+    isExiting = true;
+    isCancelling = true; // Mark that we're cancelling
+    
+    // Temporarily override process.exit to prevent Vitest from exiting before we're done
+    const originalExit = process.exit;
+    let exitCalled = false;
+    let exitCode = 130;
+    
+    process.exit = (code) => {
+      if (!exitCalled) {
+        exitCalled = true;
+        exitCode = code ?? 130;
+        logger.debug(`process.exit(${exitCode}) called, waiting for cleanup...`);
+      }
+    };
+    
+    handleProcessExit()
+      .then(() => {
+        logger.debug("Cleanup completed successfully");
+      })
+      .catch((err) => {
+        logger.error("Error during SIGINT cleanup:", err.message);
+      })
+      .finally(() => {
+        logger.debug(`Exiting with code ${exitCode}`);
+        // Restore and call original exit
+        process.exit = originalExit;
+        process.exit(exitCode);
+      });
   });
 
   // Handle kill command
-  process.on("SIGTERM", async () => {
-    await handleProcessExit();
-    process.exit(143); // Standard exit code for SIGTERM
+  process.prependOnceListener("SIGTERM", () => {
+    logger.debug("SIGTERM received, cleaning up...");
+    if (isExiting) return;
+    isExiting = true;
+    isCancelling = true;
+    
+    const originalExit = process.exit;
+    let exitCode = 143;
+    
+    process.exit = (code) => {
+      exitCode = code ?? 143;
+    };
+    
+    handleProcessExit()
+      .then(() => {
+        logger.debug("Cleanup completed successfully");
+      })
+      .catch((err) => {
+        logger.error("Error during SIGTERM cleanup:", err.message);
+      })
+      .finally(() => {
+        logger.debug(`Exiting with code ${exitCode}`);
+        process.exit = originalExit;
+        process.exit(exitCode);
+      });
   });
   
 }
@@ -512,9 +582,9 @@ class TestDriverReporter {
   }
 
   async initializeTestRun() {
-    logger.debug("Initializing test run...");
-    logger.debug("Current API key in pluginState:", !!pluginState.apiKey);
-    logger.debug("Current API root in pluginState:", pluginState.apiRoot);
+    logger.debug("initializeTestRun called");
+    logger.debug("API key present:", !!pluginState.apiKey);
+    logger.debug("API root:", pluginState.apiRoot);
 
     // Check if we should enable the reporter
     if (!pluginState.apiKey) {
@@ -552,9 +622,9 @@ class TestDriverReporter {
       // Default to linux if no tests write platform info
       testRunData.platform = "linux";
 
-      logger.debug("Creating test run with data:", testRunData);
+      logger.debug("Creating test run with data:", JSON.stringify(testRunData));
       pluginState.testRun = await createTestRun(testRunData);
-      logger.debug("Test run created successfully:", pluginState.testRun);
+      logger.debug("Test run created:", JSON.stringify(pluginState.testRun));
 
       // Store in environment variables for worker processes to access
       process.env.TD_TEST_RUN_ID = pluginState.testRunId;
@@ -571,7 +641,7 @@ class TestDriverReporter {
         startTime: pluginState.startTime,
       });
 
-      logger.debug(`Test run created: ${pluginState.testRunId}`);
+      logger.info(`Test run created: ${pluginState.testRunId}`);
     } catch (error) {
       logger.error("Failed to initialize:", error.message);
       pluginState.apiKey = null;
@@ -580,8 +650,24 @@ class TestDriverReporter {
   }
 
   async onTestRunEnd(testModules, unhandledErrors, reason) {
-    logger.debug("Test run ending with reason:", reason);
-    logger.debug("Plugin state - API key present:", !!pluginState.apiKey, "Test run present:", !!pluginState.testRun);
+    logger.debug("onTestRunEnd called with reason:", reason);
+    logger.debug("API key present:", !!pluginState.apiKey);
+    logger.debug("Test run present:", !!pluginState.testRun);
+    logger.debug("Test run ID:", pluginState.testRunId);
+    logger.debug("isCancelling:", isCancelling);
+    logger.debug("testRunCompleted:", pluginState.testRunCompleted);
+
+    // If we're cancelling due to SIGINT/SIGTERM, skip - handleProcessExit will handle it
+    if (isCancelling) {
+      logger.debug("Cancellation in progress via signal handler, skipping onTestRunEnd");
+      return;
+    }
+
+    // If already completed (by handleProcessExit), skip
+    if (pluginState.testRunCompleted) {
+      logger.debug("Test run already completed, skipping");
+      return;
+    }
 
     if (!pluginState.apiKey) {
       logger.warn("Skipping completion - no API key (was it cleared after init failure?)");
@@ -644,10 +730,11 @@ class TestDriverReporter {
       }
 
       // Test cases are reported directly from teardownTest
-      logger.debug("All test cases reported from teardown");
+      logger.debug("Calling completeTestRun API...");
+      logger.debug("Complete data:", JSON.stringify(completeData));
 
       const completeResponse = await completeTestRun(completeData);
-      logger.debug("Test run completion API response:", completeResponse);
+      logger.debug("API response:", JSON.stringify(completeResponse));
 
       // Mark test run as completed to prevent duplicate completion
       pluginState.testRunCompleted = true;
@@ -657,7 +744,7 @@ class TestDriverReporter {
       const consoleUrl = getConsoleUrl(pluginState.apiRoot);
       if (testRunDbId) {
         const testRunUrl = `${consoleUrl}/runs/${testRunDbId}`;
-        logger.debug(`🔗 View test run: ${testRunUrl}`);
+        logger.info(`View test run: ${testRunUrl}`);
         // Output in a parseable format for CI
         console.log(`TESTDRIVER_RUN_URL=${testRunUrl}`);
         
@@ -665,7 +752,7 @@ class TestDriverReporter {
         await postGitHubCommentIfEnabled(testRunUrl, stats, completeData);
       }
 
-      logger.debug(`✅ Test run completed: ${stats.passedTests}/${stats.totalTests} passed`);
+      logger.info(`Test run completed: ${stats.passedTests}/${stats.totalTests} passed`);
     } catch (error) {
       logger.error("Failed to complete test run:", error.message);
       logger.debug("Error stack:", error.stack);
@@ -1134,27 +1221,38 @@ async function createTestRun(data) {
 
 async function completeTestRun(data) {
   const url = `${pluginState.apiRoot}/api/v1/testdriver/test-run-complete`;
-  const response = await withTimeout(
-    fetch(url, {
-      method: "POST",
-      headers: {
-        "Content-Type": "application/json",
-        Authorization: `Bearer ${pluginState.token}`,
-      },
-      body: JSON.stringify(data),
-    }),
-    10000,
-    "Internal Complete Test Run",
-  );
-
-  if (!response.ok) {
-    const errorText = await response.text();
-    throw new Error(
-      `API error: ${response.status} ${response.statusText} - ${errorText}`,
+  logger.debug(`completeTestRun: POSTing to ${url}`);
+  
+  try {
+    const response = await withTimeout(
+      fetch(url, {
+        method: "POST",
+        headers: {
+          "Content-Type": "application/json",
+          Authorization: `Bearer ${pluginState.token}`,
+        },
+        body: JSON.stringify(data),
+      }),
+      10000,
+      "Internal Complete Test Run",
     );
-  }
 
-  return await response.json();
+    logger.debug(`completeTestRun: Response status ${response.status}`);
+
+    if (!response.ok) {
+      const errorText = await response.text();
+      throw new Error(
+        `API error: ${response.status} ${response.statusText} - ${errorText}`,
+      );
+    }
+
+    const result = await response.json();
+    logger.debug(`completeTestRun: Success`);
+    return result;
+  } catch (error) {
+    logger.error(`completeTestRun: Error - ${error.message}`);
+    throw error;
+  }
 }
 
 // Global state setup moved to setup file (vitestSetup.mjs)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "testdriverai",
-  "version": "7.2.35",
+  "version": "7.2.37",
   "description": "Next generation autonomous AI agent for end-to-end testing of web & desktop",
   "main": "sdk.js",
   "exports": {

package/setup/aws/cloudformation.yaml

@@ -81,8 +81,8 @@ Parameters:
 
   CreateKeyPair:
     Type: String
-    AllowedValues: [yes, no]
-    Default: yes
+    AllowedValues: ["yes", "no"]
+    Default: "yes"
     Description: Create a new key pair for instance access? (If 'no', you must provide an existing key name)
   ExistingKeyName:
     Type: String
@@ -168,15 +168,15 @@ Resources:
             IpProtocol: tcp,
             FromPort: 8765,
             ToPort: 8765,
-            CidrIp: 74.220.58.0/24,
-            Description: "pyautogui-cli WebSockets - Static IP 1",
+            CidrIp: 0.0.0.0/0,
+            Description: "pyautogui-cli WebSockets - Open for testing",
           }
         - {
             IpProtocol: tcp,
-            FromPort: 8765,
-            ToPort: 8765,
-            CidrIp: 74.220.50.0/24,
-            Description: "pyautogui-cli WebSockets - Static IP 2",
+            FromPort: 5800,
+            ToPort: 5800,
+            CidrIp: !Ref AllowedIngressCidr,
+            Description: "WebVNC 5800",
           }
         - {
             IpProtocol: tcp,