@checksum-ai/runtime 1.1.14 → 1.1.16

package/README.md

@@ -1,20 +1,17 @@
-# Checksum.ai Tests
+# Checksum.ai Runtime
 
-## Quick Start
+### Quick Start
 
-1. Run `npm install -D checksumai`.
-2. Navigate to the directory where you want to add Checksum tests and run `npx checksumai init`.
-3. Run `npx playwright install --with-deps` to install Playwright dependencies.
-4. Edit `checksum.config.ts` to include necessary configurations such as:
-   - `apiKey`
-   - `baseURL`
-   - `username`
-   - `password`
-5. Update `login.ts` with your login function using Playwright. See the Login Function section below for guidance.
-6. Run `npx checksumai test` to execute the example test and verify successful login.
-7. If you haven't already, visit [app.checksum.ai](https://app.checksum.ai) to complete the configuration and generate a test. Then, wait for the pull request (PR) to be created and approve it.
+1. Install the package using `npm install -D @checksum-ai/runtime` or `yarn add @checksum-ai/runtime -D`.
+2. Navigate to the directory where you want to initialize the Checksum tests folder and run `npx checksumai init`.
+3. In the newly created "checksum" folder
+   1. Edit `checksum.config.ts` and add the necessary configurations, including your apiKey, application baseURL, environment info, etc.
+   2. Update `login.ts` with your login function using Playwright. See the Login Function section below for guidance.
+4. Run `npx playwright install --with-deps` to install Playwright dependencies.
+5. Run `npx checksumai test` to execute the example test and verify successful login.
+6. If you haven't already, visit [app.checksum.ai](https://app.checksum.ai) to complete the configuration and generate a test. Then, wait for the pull request (PR) to be created and approve it.
 
-## Login Function
+### Login Function
 
 1. This function is executed at the start of each test.
 2. We recommend using a consistent seeded user for every test. For example, before each test, call a webhook that creates a user, seeds it with data, and returns the username and password. This approach ensures test reliability and allows parallel test execution. Configure this webhook [in your project](https://app.checksum.ai/#/settings/wizard) for consistent test generation.
@@ -23,36 +20,183 @@
 
 ## Checksum AI Magic
 
-The tests generated by Checksum are based on Playwright. When executed using the Checksum CLI with an API key, Checksum enhances Playwright's functionality, improving test reliability and automatically maintaining tests.
+The tests generated by Checksum are based on Playwright. When executed using the Checksum runtime CLI with an API key, Checksum enhances Playwright's functionality, improving test reliability and automatically maintains tests. Each enhancement is enabled/disabled using the `checksum.config.ts` file.
 
-### Autonomous Test Agent
+#### Smart Selectors
+When generating tests, Checksum stores extensive metadata for each action (stored in the `checksum/test-data` folder). If a traditional selector fails, this metadata is used to locate the target element. For example, if a test identifies an element by its ID but the ID changes, Checksum utilizes other data points (e.g., element class, text, parents) to locate the element. Use the `checksumSelector("<id>")` method to link an action to its metadata. Do not alter the IDs.
 
-Checksum runs your Playwright tests regularly, but we added a few extra features to make tests more reliable. All of the features can be turned on/off through `checksum.config.ts`
 
-**Smart Selectors**
-When generating tests, Checksum stores extensive metadata for each action (see the `test-data` folder). If a traditional selector fails, this metadata is used for correction. For example, if a test identifies an element by its ID but the ID changes, Checksum utilizes other data points (e.g., element class, text, parents) to locate the element. Use the `checksumSelector("<id>")` method to link an action to its metadata. Do not alter the IDs.
+#### Checksum AI
+If Smart Selectors also fail, Checksum's custom-trained model can regenerate the failed section of the test. In such cases, the model might add, remove, or alter actions to achieve the same objectives, without changing the assertions. The assumption is that as long as the assertions pass, the model has successfully fixed the test. Use the `checksumAI` method to wrap each step in order to achieve this behavior.
 
-**Checksum AI**
-If Smart Selectors also fail, Checksum's custom-trained model can regenerate the failed section of the test. In such cases, the model might add, remove, or alter actions to achieve the same objectives, without changing the assertions. The assumption is that as long as the assertions pass, the model has successfully fixed the test. Use the `.checksumAI("<natural language description of the test>")` method to guide the model in fixing the test.
+Example using both `checksumSelector` and `checksumAI`:
+```js
+ await checksumAI("Save the form and continue", () =>
+      page
+        .checksumSelector("gIexv")
+        .getByRole("button", { name: "Save" })
+        .click()
+    );
+```
 
 You can modify the description as needed for our model. Additionally, you can include steps with only ChecksumAI descriptions, prompting our model to generate the Playwright code. For example, `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the necessary Playwright code. You can even author entire tests in this manner.
 
-### Run Modes
-
-Checksum offers three run modes:
-
-1. **Normal** - Tests are executed using the Autonomous Test Agent as defined in the config file.
-2. **Heal** - If the Autonomous Test Agent corrects a test, a new test file with the fix is created. By default, this file is created locally, but you can also configure the Agent to open a PR to your GitHub repository by setting `autoHealPRs` to true.
-3. **Refactor (Work in Progress)** - The Autonomous Test Agent runs the test and, for each action, regenerates a regular Playwright selector, a Smart Selector, and a Checksum AI description.
 
 ### Mock Data
 
-When generating tests, Checksum records all backend responses, allowing tests to run in the same backend context. This is particularly useful for debugging or initial test runs, especially if your testing database/context differs from that used for test generation. Note that if your backend response format changes, the mocked data may not work as expected.
-
-### CLI Commands
+When generating tests, Checksum records all network responses, allowing tests to run in the same context. These recordings are stored as HAR files, stored in the  `checksum/test-data/har` folder. Using this method is particularly useful for debugging newly generated tests, especially if your testing database/context differs from the one used for test generation. Note that if your network responses' format changes, the mocked data may not work as expected. 
+
+## Checksum Config
+Alongside standard test run configurations found in `playwright.config.ts`, use the `checksum.config.ts` file to configure your test runs.
+
+```js
+{
+  /**
+   * Checksum Run mode.
+   * Normal is currently the only supported mode.
+   */
+  runMode: RunMode.Normal,
+
+  /**
+   * Insert here your Checksum API key.
+   * You can find it in https://app.checksum.ai/#/settings/
+   */
+  apiKey: "<API key>",
+
+     /**
+   * Define your test run environments and test users within each environment.
+   * The environments must be aligned with those set here:
+   * https://app.checksum.ai/#/settings/
+   */
+  environments: [{
+      name: 'staging';
+      users: [
+         {
+            role: "host",
+            username: "<host username>",
+            password: "<host password>",
+         },
+         {
+            role: "guest",
+            username: "<guest username>",
+            password: "<guest password>",
+         },
+      ],
+      baseURL: 'staging.mydomain.com';
+      loginURL: 'staging.mydomain.com/login';
+      default: true;
+  }]
+
+  options: {
+    /**
+     * Whether to use Checksum Smart Selector in order to
+     * recover from failing to locate an element for an action
+     */
+    useChecksumSelectors: true,
+    /**
+     * Whether to use Checksum AI in order to 
+     * recover from a failed action or assertion
+     */
+    useChecksumAI: { actions: true, assertions: false },
+    /**
+     * Whether to use recorded network responses when running your tests
+     */
+    useMockData: false,
+    /**
+     * Whether to upload the test reports to Checksum cloud 
+     * allowing you to view them at https://app.checksum.ai/#/test-runs.
+     * Note: Only relevant if Playwright reporter config is set to HTML
+     * Note: Reports will be saved locally either way (according to Playwright configs)
+     * and can be viewed using the CLI command show-reports.
+     */
+    hostReports: !!process.env.CI,
+  }
+}
+```
+
+## Checksum Helpers API
+Helpers are deconstructed from the result of the initial call to the imported @checksum-ai/runtime `init` method, as following:
+```js
+import { test as base } from "@playwright/test";
+import { init, IChecksumPage } from "@checksum-ai/runtime";
+const { test, expect, defineChecksumTest, login, checksumAI } = init(base);
+```
+
+```js
+function init(base: ChecksumTestType<PlaywrightTestArgs>): {
+   /**
+    * The Playwright test instance enhanced by Checksum
+    */ 
+  test: ChecksumTestType<ChecksumPlaywrightTestArgs>;
+  /**
+   * The login method which calls the user defined login function
+   */
+  login: ReturnType<typeof getLogin>;
+  /**
+   * The test title definition along with the test id
+   * used by Checksum to identify the test
+   */
+  defineChecksumTest: (title: string, testId: string) => string;
+     /**
+    * The Playwright expect instance enhanced by Checksum
+    */ 
+  expect: ChecksumExpect;
+  /**
+   * Wrapping each test action with this method allows
+   * Checksum to recover from failure using our test recover AI agent
+   */
+  checksumAI: (description: string, testFunction: Function) => Promise<any>;
+  /**
+   * Return environment data as defined in the checksum configuration 
+   * given the environment name and userRole, along with a login function
+   * for this environment and user combination
+   */
+  getEnvironment: ({
+    name,
+    userRole,
+  }: {
+    name?: string;
+    userRole?: string;
+  }) => {
+    environment: ChecksumConfigEnvironment;
+    user: EnvironmentUser;
+    login: ReturnType<typeof getLogin>;
+  };
+};
+```
+## Checksum Page API
+
+```js
+interface ChecksumPage extends Page {
+   /**
+    * Affiliates the test step with selection data calculated during test generation and used as part of the recovery mechanism in case Playwright fails to locate the element
+    */
+  checksumSelector: (id: string) => ChecksumPage;
+
+   /**
+    * For tests requiring file uploads, this method resolves to the configured assets folder storing all relevant files
+    */ 
+  resolveAssetsFolder: (assets: string[]) => string[];
+
+  /**
+   * For multi-tab test flows, this method gets the ChecksumPage in order of appearance, when 0 is the initially opened tab.
+   */
+  getPage(index: number): Promise<ChecksumPage>;
+
+  /**
+   * When required to login mid-test, use reauthenticate with the user's role. Note: It is not possible to change environments during a single test run.
+   */
+  reauthenticate: (role: string) => Promise<void>;
+}
+
+```
+
+
+## CLI Commands
 
 1. `init` - Initialize the Checksum directory and configurations.
 2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override `checksum.config.ts`, pass full or partial JSON as a string, e.g., `--checksum-config='{"baseURL": "https://example.com"}'`.
+3. `show-report` - Locally shows the latest test run HTML report. Only applicable following completion of a test run configured to output an HTML report.
 
 ## Running with GitHub Actions