@checksum-ai/runtime 1.0.29 → 1.0.31

package/README.md

@@ -1 +1,55 @@
-Checksum npm package readme
+# Checksum.ai Tests
+
+## Quick Start
+
+1. Run `npm i @checksum-ai/runtime`
+2. Navigate to the directory you want to add Checksum tests and run `npm run checksum init`
+3. Run `npx playwright install --with-deps` to install Playwright dependencies.
+4. Update `checksum.config.ts`. At the very least you need to add:
+   1. apiKey
+   2. baseURL
+   3. username
+   4. password
+5. Update `login.ts` with your login function using Playwright (see bellow)
+6. Run `npm run checksum test` to run the example test and make sure login is successful
+7. If you haven't already done so, go to [app.checksum.ai](https://app.checksum.ai) finish the configuration and generate a test. Then wait for the PR to be created and approve it.
+
+## Login Function
+
+1. This function will be run at the beginning of each test.
+2. We recommend using a consistent seeded user for each test. For example, before each test, call a webhook that creates a user, seeds it with data and returns the username and password. Doing so will keep tests reliable and allow running tests in parallel. If you do use a webhook, make sure to configure it [in your project](https://app.checksum.ai/#/settings/wizard) as well so test generation runs in the same context.
+3. After login-in, assert that the login was successful. Playwright waits for assertions to be correct, so adding an assertion assures that the page is ready fo interaction before returning.
+4. If you'd like to reuse authentication state between tests, follow Playwright guide https://playwright.dev/docs/auth. Then, check at the beginning of the login function if user is already authenticated and if so return.
+
+## Checksum AI Magic
+
+The tests Checksum generates are Playwright tests. However, when executed using Checksum CLI with an API key, Checksum extends Playwright functionality to improve test reliability and automatically maintain tests.
+
+### Autonomous Test Agent
+
+Checksum runs your Playwright tests regularly, but we added a few extra features to make tests more reliable. All of the features can be turned on/off through `checksum.config.ts`
+
+**Smart Selectors**
+when the test is generated, Checksum stores vast metadata for every action (see test-data folder). When a classic selector fails, we use the metadata to fix it. For example, if a test identifies an element by its ID, but the ID changed, Checksum looks at hundreds of other data points (eg element class, text, parents) to find the element. To connect an action to its metadata, we use the `checksumSelector("<id>")` method. Do not change the IDs.
+
+**Checksum AI**
+If Smart Selectors fail as well, Checksum can use our custom-trained model to completely regenerate the failed section. In that case, the model might add, remove of take different actions to complete the same goals. The model will not change the assertions and the assumption is that as long as the assertions pass, the model has fixed the test. `.checksumAI("<natural language description of the test>")` method is used to instruct the model on how to fix the test.
+
+You can edit the description as needed to help inform our model. You can also add steps with only ChecksumAI descriptions so our model will generate the Playwright code. For example, adding `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the Playwright code for this action. You can even author full tests this way.
+
+### Run Modes
+
+Checksum has three run modes:
+
+1. Normal - tests are run using the Autonomous Test Agent as defined in the config file.
+2. Heal - If the Autonomous Test Agent corrects a test, we create a new test file with the fix. By default the test file will be created locally, but you can also have the Agent open a PR to your github repo by setting `autoHealPRs` to true
+3. Refactor (wip) - Checksum Autonomous Test Agent will run the test and for each action, regenerate a regular Playwright selector, a Smart Selector and a Checksum AI description.
+
+### Mock Data
+
+When Checksum generates the test, we record all of the Backend responses so you can run the tests with exactly the same Backend context. Its useful when debugging a test, or when running it for the first time, especially if your testing DB/context is different then the one used for test generation. If your Backend response format, changes the Mocked data might not work as expected anymore.
+
+### CLI Commands (Needs to be updated)
+
+1. `init` - initialize Checksum directory and configs
+2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override the`checksum.config.ts` you can pass full or partial json as a string. E.g. `--checksum-config='{"baseURL" = "https://example.com"}'`

package/checksum-root/README.md

@@ -1 +1,55 @@
-Checksum readme
+# Checksum.ai Tests
+
+## Quick Start
+
+1. Run `npm i @checksum-ai/runtime`
+2. Navigate to the directory you want to add Checksum tests and run `npm run checksum init`
+3. Run `npx playwright install --with-deps` to install Playwright dependencies.
+4. Update `checksum.config.ts`. At the very least you need to add:
+   1. apiKey
+   2. baseURL
+   3. username
+   4. password
+5. Update `login.ts` with your login function using Playwright (see bellow)
+6. Run `npm run checksum test` to run the example test and make sure login is successful
+7. If you haven't already done so, go to [app.checksum.ai](https://app.checksum.ai) finish the configuration and generate a test. Then wait for the PR to be created and approve it.
+
+## Login Function
+
+1. This function will be run at the beginning of each test.
+2. We recommend using a consistent seeded user for each test. For example, before each test, call a webhook that creates a user, seeds it with data and returns the username and password. Doing so will keep tests reliable and allow running tests in parallel. If you do use a webhook, make sure to configure it [in your project](https://app.checksum.ai/#/settings/wizard) as well so test generation runs in the same context.
+3. After login-in, assert that the login was successful. Playwright waits for assertions to be correct, so adding an assertion assures that the page is ready fo interaction before returning.
+4. If you'd like to reuse authentication state between tests, follow Playwright guide https://playwright.dev/docs/auth. Then, check at the beginning of the login function if user is already authenticated and if so return.
+
+## Checksum AI Magic
+
+The tests Checksum generates are Playwright tests. However, when executed using Checksum CLI with an API key, Checksum extends Playwright functionality to improve test reliability and automatically maintain tests.
+
+### Autonomous Test Agent
+
+Checksum runs your Playwright tests regularly, but we added a few extra features to make tests more reliable. All of the features can be turned on/off through `checksum.config.ts`
+
+**Smart Selectors**
+when the test is generated, Checksum stores vast metadata for every action (see test-data folder). When a classic selector fails, we use the metadata to fix it. For example, if a test identifies an element by its ID, but the ID changed, Checksum looks at hundreds of other data points (eg element class, text, parents) to find the element. To connect an action to its metadata, we use the `checksumSelector("<id>")` method. Do not change the IDs.
+
+**Checksum AI**
+If Smart Selectors fail as well, Checksum can use our custom-trained model to completely regenerate the failed section. In that case, the model might add, remove of take different actions to complete the same goals. The model will not change the assertions and the assumption is that as long as the assertions pass, the model has fixed the test. `.checksumAI("<natural language description of the test>")` method is used to instruct the model on how to fix the test.
+
+You can edit the description as needed to help inform our model. You can also add steps with only ChecksumAI descriptions so our model will generate the Playwright code. For example, adding `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the Playwright code for this action. You can even author full tests this way.
+
+### Run Modes
+
+Checksum has three run modes:
+
+1. Normal - tests are run using the Autonomous Test Agent as defined in the config file.
+2. Heal - If the Autonomous Test Agent corrects a test, we create a new test file with the fix. By default the test file will be created locally, but you can also have the Agent open a PR to your github repo by setting `autoHealPRs` to true
+3. Refactor (wip) - Checksum Autonomous Test Agent will run the test and for each action, regenerate a regular Playwright selector, a Smart Selector and a Checksum AI description.
+
+### Mock Data
+
+When Checksum generates the test, we record all of the Backend responses so you can run the tests with exactly the same Backend context. Its useful when debugging a test, or when running it for the first time, especially if your testing DB/context is different then the one used for test generation. If your Backend response format, changes the Mocked data might not work as expected anymore.
+
+### CLI Commands (Needs to be updated)
+
+1. `init` - initialize Checksum directory and configs
+2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override the`checksum.config.ts` you can pass full or partial json as a string. E.g. `--checksum-config='{"baseURL" = "https://example.com"}'`

package/checksum-root/checksum-tests.example.yml

@@ -0,0 +1,55 @@
+#################################
+# This file has two example Github workflows to run Checksum tests
+# 1. Runs Checksum tests on every push or PR to main/master
+# 2. Runs the test using Docker container
+
+
+
+name: Checksum Tests
+on:
+  push:
+    branches: [ main, master ]
+  pull_request:
+  branches: [ main, master ]
+jobs:
+  test:
+    timeout-minutes: 120
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v3
+        with:
+          node-version: 18
+      # Installing deps, which should Include Checksum runtime
+      - name: Install dependencies
+        run: npm ci
+      - name: Install Playwright Browsers
+        run: npx playwright install --with-deps
+      # Run tests 
+      - name: Run Checksum tests
+        run: npm run checksum test
+
+
+name: Checksum Tests with Docker
+on:
+  push:
+    branches: [ main, master ]
+  pull_request:
+    branches: [ main, master ]
+jobs:
+  playwright:
+    name: 'Checksum Tests with Docker'
+    runs-on: ubuntu-latest
+    container:
+      image: mcr.microsoft.com/playwright:v1.40.0-jammy
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v3
+        with:
+          node-version: 18
+      - name: Install dependencies
+        run: npm ci
+      - name: Run Checksum tests
+        run: npm run checksum test
+        env:
+          HOME: /root

package/checksum-root/checksum.config.ts

@@ -2,55 +2,41 @@ import { RunMode, getChecksumConfig } from "@checksum-ai/runtime";
 
 export default getChecksumConfig({
   /**
-   * Checksum runtime running mode -
-   * normal -   tests run normally
-   * heal -     checksum will attempt to heal tests that failed using fallback
-   * refactor - checksum will attempt to refactor and improve your tests
+   * Checksum Run mode. See Readme for more info
    */
   runMode: RunMode.Normal,
 
   /**
-   * Insert here your Checksum API key
+   * Insert here your Checksum API key. You can find it in https://app.checksum.ai/#/settings/
    */
   apiKey: "<API key>",
 
   /**
-   * This is the base URL of the tested app
+   * This is the base URL of the tested app. E.g. https://example.com. URLs in the tests will be relative to the base URL.
    */
   baseURL: "<base URL>",
 
-  /**
-   * Insert the account's username that will be used
-   * to login into your testing environment
-   */
-  username: "<username>",
-
-  /**
-   * Insert the account's password that will be used
-   * to login into your testing environment
-   */
-  password: "<password>",
-
   options: {
     /**
-     * Whether to use Checksum Smart Selector when trying to locate elements
+     * Whether to use Checksum Smart Selector when an action fails (see README)
      */
     useChecksumSelectors: true,
-
     /**
-     * Whether to use Checksum AI when trying to locate elements
+     * Whether to use Checksum AI when an action fails (see README)
      */
     useChecksumAI: true,
-
     /**
-     * Whether to use mock API data when running your tests
+     * Whether to use mock API data when running your tests (see README)
      */
     useMockData: false,
-
     /**
-     * Print runtime logs.
-     * Use for debug only
+     * Whether to Upload HTML test reports to app.checksum.ai so they can be viewed through the UI. Only relevant if Playwright reporter config is set to HTML
+     * Reports will be saved locally either way (according to Playwright Configs) and can be viewed using the CLI command show-reports.
+     */
+    hostReports: !!process.env.CI,
+    /**
+     * Whether to create a PR with healed tests. Only relevant when in Heal mode.
      */
-    printLogs: false,
+    autoHealPRs: !!process.env.CI,
   },
 });

package/checksum-root/example.checksum.spec.ts

@@ -0,0 +1,18 @@
+/* Checksum.ai autogenerated test */
+import { test as base, expect } from "@playwright/test";
+import { init, IChecksumPage } from "@checksum-ai/runtime";
+
+const { test, defineChecksumTest, login } = init(base);
+
+test.describe("Taskboard", () => {
+  test.beforeEach(async ({ page }: { page: IChecksumPage }) => {
+    await login(page);
+  });
+
+  test(
+    defineChecksumTest("Navigate to home page", "GPzdp"),
+    async ({ page }) => {
+      await page.goto("/");
+    }
+  );
+});

package/checksum-root/login.ts

@@ -1,19 +1,32 @@
 import { ChecksumConfig, IChecksumPage } from "@checksum-ai/runtime";
+import { expect, request } from "@playwright/test";
 
-/**
- * Login method
- */
 export default async function login(
   page: IChecksumPage,
   config: ChecksumConfig
 ) {
   /**
-   * Update this function to login to your application
-   * i.e.:
-   *    await page.goto('/login');
-   *    await page.getByPlaceholder("Email...").fill(config.username);
-   *    await page.getByPlaceholder("Password...").fill(config.password);
-   *    await page.getByText("Continue").click();
-   *    await page.waitForURL("/main");
+   * This code provides examples of how to write functions for different login scenarios.
+   * See README for more details
+   *
+   * Example with Seed Function:
    */
+  const apiContext = await request.newContext();
+  const response = await apiContext.get("https://example.com/createseed");
+  const { username, password } = await response.json();
+  await page.goto("/login");
+  await page.getByPlaceholder("Email...").fill(process.env.username);
+  await page.getByPlaceholder("Password...").fill(process.env.password);
+  await page.getByText("Login").click();
+  await expect(page.getByText("Login Successful")).toBeVisible();
+
+  /**
+   * Example with Default Username and Password:
+   * This example demonstrates how to log in to a page using a predefined username and password from a config file.
+   */
+  await page.goto("/login");
+  await page.getByPlaceholder("Email...").fill(config.username);
+  await page.getByPlaceholder("Password...").fill(config.password);
+  await page.getByText("Login").click();
+  await expect(page.getByText("Login Successful")).toBeVisible();
 }

package/checksum-root/playwright.config.ts

@@ -1,27 +1,62 @@
 import { defineConfig, devices } from "@playwright/test";
 
+/**
+ * Read environment variables from file.
+ * https://github.com/motdotla/dotenv
+ */
+require("dotenv").config();
+
+/**
+ * See https://playwright.dev/docs/test-configuration.
+ */
 export default defineConfig({
-  timeout: 120000,
-  testMatch: [/.*.[.]checksum.spec.ts/],
   testDir: "..",
-  /* disable parallel test runs */
-  workers: 1,
+  /* Set test timeout to 10 minutes (relatively long) as Checksum implements its own timeout mechanism */
+  timeout: 1000 * 50 * 10,
   /* Run tests in files in parallel */
   fullyParallel: false,
+  /* Fail the build on CI if you accidentally left test.only in the source code. */
+  forbidOnly: !!process.env.CI,
+  /* Retry on CI only */
+  retries: process.env.CI ? 2 : 0,
+  /* Opt out of parallel tests on CI. */
+  workers: process.env.CI ? 1 : 1,
   /* Reporter to use. See https://playwright.dev/docs/test-reporters */
-  reporter: "html",
+  reporter: process.env.CI
+    ? [["html", { open: "never", outputFolder: "test-results" }], ["line"]]
+    : "html",
   /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
   use: {
+    /* Base URL to use in actions like `await page.goto('/')`. */
+    // baseURL: 'http://127.0.0.1:3000',
+
     /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
     trace: "on",
     video: "on",
+    screenshot: "on",
+    locale: "en-US",
+    timezoneId: "America/Los_Angeles",
+    permissions: ["clipboard-read"],
+    actionTimeout: undefined, // Keep action timeout undefined as Checksum implements its own timeout
+  },
+  expect: {
+    toHaveScreenshot: { maxDiffPixelRatio: 0.05, maxDiffPixels: 200 },
   },
 
   /* Configure projects for major browsers */
   projects: [
     {
       name: "chromium",
-      use: { ...devices["Desktop Chrome"] },
+      testMatch: /^(?!.*refactored).*spec.*/,
+      use: {
+        ...devices["Desktop Chrome"],
+      },
     },
   ],
+  /* Run your local dev server before starting the tests */
+  // webServer: {
+  //   command: 'npm run start',
+  //   url: 'http://127.0.0.1:3000',
+  //   reuseExistingServer: !process.env.CI,
+  // },
 });

package/cli.js

@@ -1,4 +1,4 @@
-var f=Object.create;var g=Object.defineProperty;var C=Object.getOwnPropertyDescriptor;var w=Object.getOwnPropertyNames;var y=Object.getPrototypeOf,P=Object.prototype.hasOwnProperty;var u=(n,t)=>g(n,"name",{value:t,configurable:!0});var k=(n,t,e,s)=>{if(t&&typeof t=="object"||typeof t=="function")for(let o of w(t))!P.call(n,o)&&o!==e&&g(n,o,{get:()=>t[o],enumerable:!(s=C(t,o))||s.enumerable});return n};var R=(n,t,e)=>(e=n!=null?f(y(n)):{},k(t||!n||!n.__esModule?g(e,"default",{value:n,enumerable:!0}):e,n));var i=require("fs"),l=R(require("child_process")),r=require("path");var d="checksum";var h=class{constructor(){this.TEST_RUN_MONITOR_PATH=(0,r.join)(__dirname,"test-run-monitor.js");this.CHECKSUM_API_URL="https://api.checksum.ai";this.CHECKSUM_APP_URL="https://app.checksum.ai";this.didFail=!1;this.mock=!1;this.isolatedMode=!1;this.completeIndicators={upload:!1,tests:!1,report:!1};this.guardReturn=async(t,e=1e3,s="action hang guard timed out")=>{let o="guard-timed-out",a=u(async()=>(await this.awaitSleep(e+1e3),o),"guard"),c=await Promise.race([t,a()]);if(typeof c=="string"&&c===o)throw new Error(s);return c};this.awaitSleep=t=>new Promise(e=>setTimeout(e,t))}async execute(){switch(process.argv.find(t=>t==="--help"||t==="-h")&&(await this.printHelp(process.argv[2]),process.exit(0)),process.argv[2]){case"init":this.install();break;case"test":await this.test(process.argv.slice(3));break;case"show-report":this.showReport(process.argv.slice(3));break;default:await this.printHelp()}process.exit(0)}async execCmd(t){let e=await l.spawn(t,{shell:!0,stdio:"inherit"});return new Promise((o,a)=>{e.on("exit",c=>{c===0?o(!0):a(new Error(`Checsum failed execution with code: ${c} `))})})}async getCmdOutput(t){return new Promise(function(e,s){l.exec(t,(o,a,c)=>{if(o){s(`Error executing command: ${o.message}`);return}e(a)})})}async printHelp(t){switch(t){default:console.log(`
+var f=Object.create;var g=Object.defineProperty;var C=Object.getOwnPropertyDescriptor;var w=Object.getOwnPropertyNames;var y=Object.getPrototypeOf,P=Object.prototype.hasOwnProperty;var u=(n,t)=>g(n,"name",{value:t,configurable:!0});var k=(n,t,e,s)=>{if(t&&typeof t=="object"||typeof t=="function")for(let o of w(t))!P.call(n,o)&&o!==e&&g(n,o,{get:()=>t[o],enumerable:!(s=C(t,o))||s.enumerable});return n};var R=(n,t,e)=>(e=n!=null?f(y(n)):{},k(t||!n||!n.__esModule?g(e,"default",{value:n,enumerable:!0}):e,n));var i=require("fs"),l=R(require("child_process")),r=require("path");var m="checksum";var h=class{constructor(){this.TEST_RUN_MONITOR_PATH=(0,r.join)(__dirname,"test-run-monitor.js");this.CHECKSUM_API_URL="https://api.checksum.ai";this.CHECKSUM_APP_URL="https://app.checksum.ai";this.didFail=!1;this.isolatedMode=!1;this.completeIndicators={upload:!1,tests:!1,report:!1};this.guardReturn=async(t,e=1e3,s="action hang guard timed out")=>{let o="guard-timed-out",a=u(async()=>(await this.awaitSleep(e+1e3),o),"guard"),c=await Promise.race([t,a()]);if(typeof c=="string"&&c===o)throw new Error(s);return c};this.awaitSleep=t=>new Promise(e=>setTimeout(e,t))}async execute(){switch(process.argv.find(t=>t==="--help"||t==="-h")&&(await this.printHelp(process.argv[2]),process.exit(0)),process.argv[2]){case"init":this.install();break;case"test":await this.test(process.argv.slice(3));break;case"show-report":this.showReport(process.argv.slice(3));break;default:await this.printHelp()}process.exit(0)}async execCmd(t){let e=await l.spawn(t,{shell:!0,stdio:"inherit"});return new Promise((o,a)=>{e.on("exit",c=>{c===0?o(!0):a(new Error(`Checsum failed execution with code: ${c} `))})})}async getCmdOutput(t){return new Promise(function(e,s){l.exec(t,(o,a,c)=>{if(o){s(`Error executing command: ${o.message}`);return}e(a)})})}async printHelp(t){switch(t){default:console.log(`
 Checksum CLI
 Usage: checksum [command] [options]
 
@@ -10,10 +10,10 @@ show-report [options] [report]  show HTML report
 `);break;case"test":try{let e="npx playwright test --help",s=(await this.getCmdOutput(e)).replace(/npx playwright/g,"yarn checksum").split(`
 `);s.splice(5,0,"  --checksum-config=<config>   Checksum configuration in JSON format").join(`
 `),console.log(s.join(`
-`))}catch(e){console.log("Error",e.message)}break;case"show-report":try{let e="npx playwright show-report --help",s=(await this.getCmdOutput(e)).replace(/npx playwright/g,"yarn checksum");console.log(s)}catch(e){console.log("Error",e.message)}break}}async showReport(t){let e=`npx playwright show-report ${t.join(" ")}`;try{await this.execCmd(e)}catch(s){console.log("Error showing report",s.message)}}async test(t){this.processChecksumArguments(t),this.setChecksumConfig(),await this.getSession();let e;try{e=await this.guardReturn(this.startTestRunMonitor(this.testSession),1e4,"test run monitor timeout")}catch{console.log("Error starting test run monitor. Test results will not be available on checksum.")}this.buildVolatileConfig();let s=`${e?`CHECKSUM_UPLOAD_AGENT_PORT=${e} `:""} npx playwright test --config ${this.getPlaywrightConfigFile()} ${t.join(" ")}`;await this.patchPlaywright();try{await this.execCmd(s),console.log("Tests execution finished")}catch(o){this.didFail=!0,console.log("Error during test",o.message)}finally{let o=this.getPlaywrightReportPath();(0,i.existsSync)(o)?this.testRunMonitorProcess.stdin.write(`cli:report=${o}`):console.log(`Could not find report file at ${o}`),await this.patchPlaywright(!0),this.completeIndicators.tests=!0,await this.handleCompleteMessage()}}async patchPlaywright(t=!1){let e=`bash ${(0,r.join)(__dirname,"scripts/patch.sh")}${t?" off":""}`;try{await this.execCmd(e)}catch(s){console.log("Error patching playwright",s.message)}}getPlaywrightReportPath(){var o,a;let t=(0,r.join)(process.cwd(),"playwright-report"),e=require(this.getPlaywrightConfigFile()),{reporter:s}=e;return s instanceof Array&&s.length>1&&((o=s[1])!=null&&o.outputFolder)&&(t=(a=s[1])==null?void 0:a.outputFolder),process.env.PLAYWRIGHT_HTML_REPORT&&(t=process.env.PLAYWRIGHT_HTML_REPORT),(0,r.join)(t,"index.html")}getPlaywrightConfigFile(){return(0,r.join)(this.getRootDirPath(),"playwright.config.ts")}startTestRunMonitor(t){return new Promise(e=>{console.log("Starting test run monitor"),this.testRunMonitorProcess=l.spawn("node",[this.TEST_RUN_MONITOR_PATH,JSON.stringify({sessionId:t,checksumApiURL:this.CHECKSUM_API_URL,apiKey:this.config.apiKey}),...this.mock?["mock"]:[]]),this.testRunMonitorProcess.stdout.on("data",s=>{var p,m;let o=s.toString().trim();if(!o.startsWith("monitor")){(m=(p=this.config)==null?void 0:p.options)!=null&&m.printLogs&&console.log(`Message from test run monitor: ${o}`);return}let[a,c]=o.substring(o.indexOf(":")+1).split("=");a==="port"?e(c):this.handleTestRunMonitorMessage(a,c)}),this.testRunMonitorProcess.on("exit",(s,o)=>{console.log(`test run monitor process exited with code ${s} and signal ${o}`)}),this.testRunMonitorProcess.on("error",s=>{console.error(`Error starting test run monitor: ${s.message}`)})})}async handleTestRunMonitorMessage(t,e){switch(t){case"complete":this.isolatedMode||console.log("Test artifacts uploaded successfully"),this.sendUploadsComplete().then(()=>{this.completeIndicators.upload=!0});break;case"report-uploaded":{this.isolatedMode||console.log("Report generated and uploaded to checksum");let s={};try{s=JSON.parse(e)}catch(o){console.log("Error parsing stats",o.message)}await this.sendTestRunEnd(s),this.completeIndicators.report=!0,this.isolatedMode||console.log(`*******************
+`))}catch(e){console.log("Error",e.message)}break;case"show-report":try{let e="npx playwright show-report --help",s=(await this.getCmdOutput(e)).replace(/npx playwright/g,"yarn checksum");console.log(s)}catch(e){console.log("Error",e.message)}break}}async showReport(t){let e=`npx playwright show-report ${t.join(" ")}`;try{await this.execCmd(e)}catch(s){console.log("Error showing report",s.message)}}async test(t){this.processChecksumArguments(t),this.setChecksumConfig(),await this.getSession();let e;try{e=await this.guardReturn(this.startTestRunMonitor(this.testSession),1e4,"test run monitor timeout")}catch{console.log("Error starting test run monitor. Test results will not be available on checksum.")}this.buildVolatileConfig();let s=`${e?`CHECKSUM_UPLOAD_AGENT_PORT=${e} `:""}${this.config.options.hostReports&&!this.isolatedMode?" PW_TEST_HTML_REPORT_OPEN=never":""} npx playwright test --config ${this.getPlaywrightConfigFile()} ${t.join(" ")}`;await this.patchPlaywright();try{await this.execCmd(s),console.log("Tests execution finished")}catch(o){this.didFail=!0,console.log("Error during test",o.message)}finally{let o=this.getPlaywrightReportPath();(0,i.existsSync)(o)?this.testRunMonitorProcess.stdin.write(`cli:report=${o}`):console.log(`Could not find report file at ${o}`),await this.patchPlaywright(!0),this.completeIndicators.tests=!0,await this.handleCompleteMessage()}}async patchPlaywright(t=!1){let e=`bash ${(0,r.join)(__dirname,"scripts/patch.sh")}${t?" off":""}`;try{await this.execCmd(e)}catch(s){console.log("Error patching playwright",s.message)}}getPlaywrightReportPath(){var o,a;let t=(0,r.join)(process.cwd(),"playwright-report"),e=require(this.getPlaywrightConfigFile()),{reporter:s}=e;return s instanceof Array&&s.length>1&&((o=s[1])!=null&&o.outputFolder)&&(t=(a=s[1])==null?void 0:a.outputFolder),process.env.PLAYWRIGHT_HTML_REPORT&&(t=process.env.PLAYWRIGHT_HTML_REPORT),(0,r.join)(t,"index.html")}getPlaywrightConfigFile(){return(0,r.join)(this.getRootDirPath(),"playwright.config.ts")}startTestRunMonitor(t){return new Promise(e=>{console.log("Starting test run monitor"),this.testRunMonitorProcess=l.spawn("node",[this.TEST_RUN_MONITOR_PATH,JSON.stringify({sessionId:t,checksumApiURL:this.CHECKSUM_API_URL,apiKey:this.config.apiKey}),...this.isolatedMode?["isolated"]:[]]),this.testRunMonitorProcess.stdout.on("data",s=>{var p,d;let o=s.toString().trim();if(!o.startsWith("monitor")){(d=(p=this.config)==null?void 0:p.options)!=null&&d.printLogs&&console.log(`Message from test run monitor: ${o}`);return}let[a,c]=o.substring(o.indexOf(":")+1).split("=");a==="port"?e(c):this.handleTestRunMonitorMessage(a,c)}),this.testRunMonitorProcess.on("exit",(s,o)=>{console.log(`test run monitor process exited with code ${s} and signal ${o}`)}),this.testRunMonitorProcess.on("error",s=>{console.error(`Error starting test run monitor: ${s.message}`)})})}async handleTestRunMonitorMessage(t,e){switch(t){case"complete":this.isolatedMode||console.log("Test artifacts uploaded successfully"),this.sendUploadsComplete().then(()=>{this.completeIndicators.upload=!0});break;case"report-uploaded":{if(this.isolatedMode){this.completeIndicators.report=!0;break}console.log("Report generated and uploaded to checksum");let s={};try{s=JSON.parse(e)}catch(o){console.log("Error parsing stats",o.message)}await this.sendTestRunEnd(s),this.completeIndicators.report=!0,console.log(`*******************
 * Checksum report URL: ${this.CHECKSUM_APP_URL}/#/test-runs/${this.testSession}
 *******************`);break}default:console.warn(`Unhandled test run monitor message: ${t}=${e}`)}}async handleCompleteMessage(){for(;;)Object.keys(this.completeIndicators).find(t=>!this.completeIndicators[t])?await this.awaitSleep(1e3):(console.log("Checksum test run complete"),this.shutdown(this.didFail?1:0))}shutdown(t=0){this.cleanup(),process.exit(t)}buildVolatileConfig(){if(!this.volatileChecksumConfig)return;let t=this.getVolatileConfigPath(),e=`
     import { RunMode, getChecksumConfig } from "@checksum-ai/runtime";
     
     export default getChecksumConfig(${JSON.stringify(this.config,null,2)});
-    `;(0,i.writeFileSync)(t,e)}cleanup(){this.deleteVolatileConfig(),this.testRunMonitorProcess.stdin.write("cli:shutdown"),this.testRunMonitorProcess.kill()}async getSession(){try{if(this.mock){this.testSession="mock-session";return}let t=this.config.apiKey;(!t||t==="<API key>")&&(console.error("No API key found in checksum config"),this.shutdown(1));let e=JSON.stringify(await this.getEnvInfo()),s=await fetch(`${this.CHECKSUM_API_URL}/client-api/test-runs`,{method:"POST",headers:{Accept:"application/json","Content-Type":"application/json",ChecksumAppCode:t},body:e});this.testSession=(await s.json()).uuid}catch{console.log("Error getting checksum test session, will run in isolation mode"),this.isolatedMode=!0,this.mock=!0,this.testSession="isolated-session"}}async sendTestRunEnd(t){try{if(this.mock)return"https://mock.report.url";let e=JSON.stringify({...t,endedAt:Date.now()});await this.updateTestRun(`${this.CHECKSUM_API_URL}/client-api/test-runs/${this.testSession}`,"PATCH",e)}catch(e){return console.log("Error sending test run end",e.message),null}}async sendUploadsComplete(){if(!this.mock)try{await this.updateTestRun(`${this.CHECKSUM_API_URL}/client-api/test-runs/${this.testSession}/uploads-completed`,"PATCH")}catch(t){console.log("Error sending test run uploads complete",t.message)}}async updateTestRun(t,e,s=void 0){return(await fetch(t,{method:e,headers:{Accept:"application/json","Content-Type":"application/json",ChecksumAppCode:this.config.apiKey},body:s})).json()}async getEnvInfo(){let t={commitHash:"",branch:"branch",environment:process.env.CI?"CI":"local",name:"name",startedAt:Date.now()};try{t.commitHash=(await this.getCmdOutput("git rev-parse HEAD")).toString().trim()}catch(e){console.log("Error getting git hash",e.message)}try{t.branch=(await this.getCmdOutput("git rev-parse --abbrev-ref HEAD")).toString().trim()}catch(e){console.log("Error getting branch name",e.message)}return t}getVolatileConfigPath(){return(0,r.join)(this.getRootDirPath(),"checksum.config.tmp.ts")}deleteVolatileConfig(){let t=this.getVolatileConfigPath();(0,i.existsSync)(t)&&(0,i.rmSync)(t)}setChecksumConfig(){this.config={...require((0,r.join)(this.getRootDirPath(),"checksum.config.ts")).default||{},...this.volatileChecksumConfig||{}},this.config.apiURL&&(this.CHECKSUM_API_URL=this.config.apiURL)}processChecksumArguments(t){this.deleteVolatileConfig();for(let e of t)if(e.startsWith("--checksum-config"))try{this.volatileChecksumConfig=JSON.parse(e.split("=")[1]),t=t.filter(s=>s!==e)}catch(s){console.log("Error parsing checksum config",s.message),this.volatileChecksumConfig=void 0}}install(){console.log("Creating Checksum directory and necessary files to run your tests");let t=this.getRootDirPath();if((0,i.existsSync)(this.getRootDirPath())||(0,i.mkdirSync)(t),!(0,i.existsSync)(this.getChecksumRootOrigin()))throw new Error("Could not find checksum root directory, please install @checksum-ai/runtime package");["checksum.config.ts","playwright.config.ts","login.ts","README.md"].forEach(e=>{(0,i.copyFileSync)((0,r.join)(this.getChecksumRootOrigin(),e),(0,r.join)(t,e))}),(0,i.mkdirSync)((0,r.join)(t,"tests"),{recursive:!0}),["esra","har","trace","log"].forEach(e=>{(0,i.mkdirSync)((0,r.join)(t,"test-data",e),{recursive:!0})})}getRootDirPath(){return(0,r.join)(process.cwd(),d)}getChecksumRootOrigin(){return(0,r.join)(process.cwd(),"node_modules","@checksum-ai","runtime","checksum-root")}};u(h,"ChecksumCLI");(async()=>await new h().execute())();
+    `;(0,i.writeFileSync)(t,e)}cleanup(){this.deleteVolatileConfig(),this.testRunMonitorProcess.stdin.write("cli:shutdown"),this.testRunMonitorProcess.kill()}async getSession(){try{if(!this.config.options.hostReports){this.setIsolatedMode();return}let t=this.config.apiKey;(!t||t==="<API key>")&&(console.error("No API key found in checksum config"),this.shutdown(1));let e=JSON.stringify(await this.getEnvInfo()),s=await fetch(`${this.CHECKSUM_API_URL}/client-api/test-runs`,{method:"POST",headers:{Accept:"application/json","Content-Type":"application/json",ChecksumAppCode:t},body:e});this.testSession=(await s.json()).uuid}catch{console.log("Error getting checksum test session, will run in isolation mode"),this.setIsolatedMode()}}setIsolatedMode(){this.isolatedMode=!0,this.testSession="isolated-session"}async sendTestRunEnd(t){if(!this.isolatedMode)try{let e=JSON.stringify({...t,endedAt:Date.now()});await this.updateTestRun(`${this.CHECKSUM_API_URL}/client-api/test-runs/${this.testSession}`,"PATCH",e)}catch(e){return console.log("Error sending test run end",e.message),null}}async sendUploadsComplete(){if(!this.isolatedMode)try{await this.updateTestRun(`${this.CHECKSUM_API_URL}/client-api/test-runs/${this.testSession}/uploads-completed`,"PATCH")}catch(t){console.log("Error sending test run uploads complete",t.message)}}async updateTestRun(t,e,s=void 0){return(await fetch(t,{method:e,headers:{Accept:"application/json","Content-Type":"application/json",ChecksumAppCode:this.config.apiKey},body:s})).json()}async getEnvInfo(){let t={commitHash:"",branch:"branch",environment:process.env.CI?"CI":"local",name:"name",startedAt:Date.now()};try{t.commitHash=(await this.getCmdOutput("git rev-parse HEAD")).toString().trim()}catch(e){console.log("Error getting git hash",e.message)}try{t.branch=(await this.getCmdOutput("git rev-parse --abbrev-ref HEAD")).toString().trim()}catch(e){console.log("Error getting branch name",e.message)}return t}getVolatileConfigPath(){return(0,r.join)(this.getRootDirPath(),"checksum.config.tmp.ts")}deleteVolatileConfig(){let t=this.getVolatileConfigPath();(0,i.existsSync)(t)&&(0,i.rmSync)(t)}setChecksumConfig(){this.config={...require((0,r.join)(this.getRootDirPath(),"checksum.config.ts")).default||{},...this.volatileChecksumConfig||{}},this.config.apiURL&&(this.CHECKSUM_API_URL=this.config.apiURL)}processChecksumArguments(t){this.deleteVolatileConfig();for(let e of t)if(e.startsWith("--checksum-config"))try{this.volatileChecksumConfig=JSON.parse(e.split("=")[1]),t=t.filter(s=>s!==e)}catch(s){console.log("Error parsing checksum config",s.message),this.volatileChecksumConfig=void 0}}install(){console.log("Creating Checksum directory and necessary files to run your tests");let t=this.getRootDirPath();if((0,i.existsSync)(this.getRootDirPath())||(0,i.mkdirSync)(t),!(0,i.existsSync)(this.getChecksumRootOrigin()))throw new Error("Could not find checksum root directory, please install @checksum-ai/runtime package");["checksum.config.ts","playwright.config.ts","login.ts","README.md"].forEach(e=>{(0,i.copyFileSync)((0,r.join)(this.getChecksumRootOrigin(),e),(0,r.join)(t,e))}),(0,i.mkdirSync)((0,r.join)(t,"tests"),{recursive:!0}),["esra","har","trace","log"].forEach(e=>{(0,i.mkdirSync)((0,r.join)(t,"test-data",e),{recursive:!0})})}getRootDirPath(){return(0,r.join)(process.cwd(),m)}getChecksumRootOrigin(){return(0,r.join)(process.cwd(),"node_modules","@checksum-ai","runtime","checksum-root")}};u(h,"ChecksumCLI");(async()=>await new h().execute())();

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checksum-ai/runtime",
-  "version": "1.0.29",
+  "version": "1.0.31",
   "description": "Checksum.ai test runtime",
   "main": "index.js",
   "scripts": {