@checksum-ai/runtime 1.0.39 → 1.0.41

package/LICENSE

@@ -1 +1,3 @@
-Checksum license
+The code provided in this package is a proprietary code owned by Checksum AI INC. Checksum retains all right, title and interest in to the code. The Customer shall maintain the confidentiality of the Software and such related information, technical data, and know-how.
+
+Customer will not copy, merge, publish, distribute, publicly display, transfer, sublicense, and/or sell the Software and will not, directly or indirectly: reverse engineer, decompile, disassemble or otherwise attempt to discover the source code, object code or underlying structure, ideas, know-how or algorithms relevant to the Services or any software, documentation or data related to the Services (“Software”); modify, translate, or create derivative works based on the Services or any Software (except to the extent expressly permitted by Company or authorized within the Services); use the Services or any Software for timesharing or service bureau purposes or otherwise for the benefit of a third; or remove any proprietary notices or labels. Checksum hereby grants Customer a non-exclusive, non-transferable, non-sublicensable license to use such Software during the Term only in connection with the Services.

package/README.md

@@ -2,54 +2,64 @@
 
 ## Quick Start
 
-1. Run `npm i @checksum-ai/runtime`
-2. Navigate to the directory you want to add Checksum tests and run `npm run checksum init`
+1. Run `npm install -D checksumai`.
+2. Navigate to the directory where you want to add Checksum tests and run `npx checksumai init`.
 3. Run `npx playwright install --with-deps` to install Playwright dependencies.
-4. Update `checksum.config.ts`. At the very least you need to add:
-   1. apiKey
-   2. baseURL
-   3. username
-   4. password
-5. Update `login.ts` with your login function using Playwright (see bellow)
-6. Run `npm run checksum test` to run the example test and make sure login is successful
-7. If you haven't already done so, go to [app.checksum.ai](https://app.checksum.ai) finish the configuration and generate a test. Then wait for the PR to be created and approve it.
+4. Edit `checksum.config.ts` to include necessary configurations such as:
+   - `apiKey`
+   - `baseURL`
+   - `username`
+   - `password`
+5. Update `login.ts` with your login function using Playwright. See the Login Function section below for guidance.
+6. Run `npx checksumai test` to execute the example test and verify successful login.
+7. If you haven't already, visit [app.checksum.ai](https://app.checksum.ai) to complete the configuration and generate a test. Then, wait for the pull request (PR) to be created and approve it.
 
 ## Login Function
 
-1. This function will be run at the beginning of each test.
-2. We recommend using a consistent seeded user for each test. For example, before each test, call a webhook that creates a user, seeds it with data and returns the username and password. Doing so will keep tests reliable and allow running tests in parallel. If you do use a webhook, make sure to configure it [in your project](https://app.checksum.ai/#/settings/wizard) as well so test generation runs in the same context.
-3. After login-in, assert that the login was successful. Playwright waits for assertions to be correct, so adding an assertion assures that the page is ready fo interaction before returning.
-4. If you'd like to reuse authentication state between tests, follow Playwright guide https://playwright.dev/docs/auth. Then, check at the beginning of the login function if user is already authenticated and if so return.
+1. This function is executed at the start of each test.
+2. We recommend using a consistent seeded user for every test. For example, before each test, call a webhook that creates a user, seeds it with data, and returns the username and password. This approach ensures test reliability and allows parallel test execution. Configure this webhook [in your project](https://app.checksum.ai/#/settings/wizard) for consistent test generation.
+3. After logging in, assert that the login was successful. Playwright waits for assertions to be correct, ensuring that the page is ready for interaction before proceeding.
+4. To reuse authentication states between tests, refer to the Playwright guide on [authentication](https://playwright.dev/docs/auth). At the start of the login function, check if the user is already authenticated and return if so.
 
 ## Checksum AI Magic
 
-The tests Checksum generates are Playwright tests. However, when executed using Checksum CLI with an API key, Checksum extends Playwright functionality to improve test reliability and automatically maintain tests.
+The tests generated by Checksum are based on Playwright. When executed using the Checksum CLI with an API key, Checksum enhances Playwright's functionality, improving test reliability and automatically maintaining tests.
 
 ### Autonomous Test Agent
 
 Checksum runs your Playwright tests regularly, but we added a few extra features to make tests more reliable. All of the features can be turned on/off through `checksum.config.ts`
 
 **Smart Selectors**
-when the test is generated, Checksum stores vast metadata for every action (see test-data folder). When a classic selector fails, we use the metadata to fix it. For example, if a test identifies an element by its ID, but the ID changed, Checksum looks at hundreds of other data points (eg element class, text, parents) to find the element. To connect an action to its metadata, we use the `checksumSelector("<id>")` method. Do not change the IDs.
+When generating tests, Checksum stores extensive metadata for each action (see the `test-data` folder). If a traditional selector fails, this metadata is used for correction. For example, if a test identifies an element by its ID but the ID changes, Checksum utilizes other data points (e.g., element class, text, parents) to locate the element. Use the `checksumSelector("<id>")` method to link an action to its metadata. Do not alter the IDs.
 
 **Checksum AI**
-If Smart Selectors fail as well, Checksum can use our custom-trained model to completely regenerate the failed section. In that case, the model might add, remove of take different actions to complete the same goals. The model will not change the assertions and the assumption is that as long as the assertions pass, the model has fixed the test. `.checksumAI("<natural language description of the test>")` method is used to instruct the model on how to fix the test.
+If Smart Selectors also fail, Checksum's custom-trained model can regenerate the failed section of the test. In such cases, the model might add, remove, or alter actions to achieve the same objectives, without changing the assertions. The assumption is that as long as the assertions pass, the model has successfully fixed the test. Use the `.checksumAI("<natural language description of the test>")` method to guide the model in fixing the test.
 
-You can edit the description as needed to help inform our model. You can also add steps with only ChecksumAI descriptions so our model will generate the Playwright code. For example, adding `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the Playwright code for this action. You can even author full tests this way.
+You can modify the description as needed for our model. Additionally, you can include steps with only ChecksumAI descriptions, prompting our model to generate the Playwright code. For example, `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the necessary Playwright code. You can even author entire tests in this manner.
 
 ### Run Modes
 
-Checksum has three run modes:
+Checksum offers three run modes:
 
-1. Normal - tests are run using the Autonomous Test Agent as defined in the config file.
-2. Heal - If the Autonomous Test Agent corrects a test, we create a new test file with the fix. By default the test file will be created locally, but you can also have the Agent open a PR to your github repo by setting `autoHealPRs` to true
-3. Refactor (wip) - Checksum Autonomous Test Agent will run the test and for each action, regenerate a regular Playwright selector, a Smart Selector and a Checksum AI description.
+1. **Normal** - Tests are executed using the Autonomous Test Agent as defined in the config file.
+2. **Heal** - If the Autonomous Test Agent corrects a test, a new test file with the fix is created. By default, this file is created locally, but you can also configure the Agent to open a PR to your GitHub repository by setting `autoHealPRs` to true.
+3. **Refactor (Work in Progress)** - The Autonomous Test Agent runs the test and, for each action, regenerates a regular Playwright selector, a Smart Selector, and a Checksum AI description.
 
 ### Mock Data
 
-When Checksum generates the test, we record all of the Backend responses so you can run the tests with exactly the same Backend context. Its useful when debugging a test, or when running it for the first time, especially if your testing DB/context is different then the one used for test generation. If your Backend response format, changes the Mocked data might not work as expected anymore.
+When generating tests, Checksum records all backend responses, allowing tests to run in the same backend context. This is particularly useful for debugging or initial test runs, especially if your testing database/context differs from that used for test generation. Note that if your backend response format changes, the mocked data may not work as expected.
 
-### CLI Commands (Needs to be updated)
+### CLI Commands
 
-1. `init` - initialize Checksum directory and configs
-2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override the`checksum.config.ts` you can pass full or partial json as a string. E.g. `--checksum-config='{"baseURL" = "https://example.com"}'`
+1. `init` - Initialize the Checksum directory and configurations.
+2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override `checksum.config.ts`, pass full or partial JSON as a string, e.g., `--checksum-config='{"baseURL": "https://example.com"}'`.
+
+## Running with GitHub Actions
+
+See the example file `github-actions.example.yml`.
+
+## Troubleshooting
+
+**Q: I'm seeing various exceptions when I run "npx checksumai test", even before the test starts.**
+
+A: If you had a pre-installed version of Playwright, it might not be compatible with Checksum. Remove both Playwright and Checksum libraries, delete the relevant folder from `node_modules`, and run `npm install -D checksumai`.

package/checksum-root/README.md

@@ -2,54 +2,64 @@
 
 ## Quick Start
 
-1. Run `npm i @checksum-ai/runtime`
-2. Navigate to the directory you want to add Checksum tests and run `npm run checksum init`
+1. Run `npm install -D checksumai`.
+2. Navigate to the directory where you want to add Checksum tests and run `npx checksumai init`.
 3. Run `npx playwright install --with-deps` to install Playwright dependencies.
-4. Update `checksum.config.ts`. At the very least you need to add:
-   1. apiKey
-   2. baseURL
-   3. username
-   4. password
-5. Update `login.ts` with your login function using Playwright (see bellow)
-6. Run `npm run checksum test` to run the example test and make sure login is successful
-7. If you haven't already done so, go to [app.checksum.ai](https://app.checksum.ai) finish the configuration and generate a test. Then wait for the PR to be created and approve it.
+4. Edit `checksum.config.ts` to include necessary configurations such as:
+   - `apiKey`
+   - `baseURL`
+   - `username`
+   - `password`
+5. Update `login.ts` with your login function using Playwright. See the Login Function section below for guidance.
+6. Run `npx checksumai test` to execute the example test and verify successful login.
+7. If you haven't already, visit [app.checksum.ai](https://app.checksum.ai) to complete the configuration and generate a test. Then, wait for the pull request (PR) to be created and approve it.
 
 ## Login Function
 
-1. This function will be run at the beginning of each test.
-2. We recommend using a consistent seeded user for each test. For example, before each test, call a webhook that creates a user, seeds it with data and returns the username and password. Doing so will keep tests reliable and allow running tests in parallel. If you do use a webhook, make sure to configure it [in your project](https://app.checksum.ai/#/settings/wizard) as well so test generation runs in the same context.
-3. After login-in, assert that the login was successful. Playwright waits for assertions to be correct, so adding an assertion assures that the page is ready fo interaction before returning.
-4. If you'd like to reuse authentication state between tests, follow Playwright guide https://playwright.dev/docs/auth. Then, check at the beginning of the login function if user is already authenticated and if so return.
+1. This function is executed at the start of each test.
+2. We recommend using a consistent seeded user for every test. For example, before each test, call a webhook that creates a user, seeds it with data, and returns the username and password. This approach ensures test reliability and allows parallel test execution. Configure this webhook [in your project](https://app.checksum.ai/#/settings/wizard) for consistent test generation.
+3. After logging in, assert that the login was successful. Playwright waits for assertions to be correct, ensuring that the page is ready for interaction before proceeding.
+4. To reuse authentication states between tests, refer to the Playwright guide on [authentication](https://playwright.dev/docs/auth). At the start of the login function, check if the user is already authenticated and return if so.
 
 ## Checksum AI Magic
 
-The tests Checksum generates are Playwright tests. However, when executed using Checksum CLI with an API key, Checksum extends Playwright functionality to improve test reliability and automatically maintain tests.
+The tests generated by Checksum are based on Playwright. When executed using the Checksum CLI with an API key, Checksum enhances Playwright's functionality, improving test reliability and automatically maintaining tests.
 
 ### Autonomous Test Agent
 
 Checksum runs your Playwright tests regularly, but we added a few extra features to make tests more reliable. All of the features can be turned on/off through `checksum.config.ts`
 
 **Smart Selectors**
-when the test is generated, Checksum stores vast metadata for every action (see test-data folder). When a classic selector fails, we use the metadata to fix it. For example, if a test identifies an element by its ID, but the ID changed, Checksum looks at hundreds of other data points (eg element class, text, parents) to find the element. To connect an action to its metadata, we use the `checksumSelector("<id>")` method. Do not change the IDs.
+When generating tests, Checksum stores extensive metadata for each action (see the `test-data` folder). If a traditional selector fails, this metadata is used for correction. For example, if a test identifies an element by its ID but the ID changes, Checksum utilizes other data points (e.g., element class, text, parents) to locate the element. Use the `checksumSelector("<id>")` method to link an action to its metadata. Do not alter the IDs.
 
 **Checksum AI**
-If Smart Selectors fail as well, Checksum can use our custom-trained model to completely regenerate the failed section. In that case, the model might add, remove of take different actions to complete the same goals. The model will not change the assertions and the assumption is that as long as the assertions pass, the model has fixed the test. `.checksumAI("<natural language description of the test>")` method is used to instruct the model on how to fix the test.
+If Smart Selectors also fail, Checksum's custom-trained model can regenerate the failed section of the test. In such cases, the model might add, remove, or alter actions to achieve the same objectives, without changing the assertions. The assumption is that as long as the assertions pass, the model has successfully fixed the test. Use the `.checksumAI("<natural language description of the test>")` method to guide the model in fixing the test.
 
-You can edit the description as needed to help inform our model. You can also add steps with only ChecksumAI descriptions so our model will generate the Playwright code. For example, adding `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the Playwright code for this action. You can even author full tests this way.
+You can modify the description as needed for our model. Additionally, you can include steps with only ChecksumAI descriptions, prompting our model to generate the Playwright code. For example, `await page.checksumAI("Click on 'New Task' button")` without the actual action will have our model generate the necessary Playwright code. You can even author entire tests in this manner.
 
 ### Run Modes
 
-Checksum has three run modes:
+Checksum offers three run modes:
 
-1. Normal - tests are run using the Autonomous Test Agent as defined in the config file.
-2. Heal - If the Autonomous Test Agent corrects a test, we create a new test file with the fix. By default the test file will be created locally, but you can also have the Agent open a PR to your github repo by setting `autoHealPRs` to true
-3. Refactor (wip) - Checksum Autonomous Test Agent will run the test and for each action, regenerate a regular Playwright selector, a Smart Selector and a Checksum AI description.
+1. **Normal** - Tests are executed using the Autonomous Test Agent as defined in the config file.
+2. **Heal** - If the Autonomous Test Agent corrects a test, a new test file with the fix is created. By default, this file is created locally, but you can also configure the Agent to open a PR to your GitHub repository by setting `autoHealPRs` to true.
+3. **Refactor (Work in Progress)** - The Autonomous Test Agent runs the test and, for each action, regenerates a regular Playwright selector, a Smart Selector, and a Checksum AI description.
 
 ### Mock Data
 
-When Checksum generates the test, we record all of the Backend responses so you can run the tests with exactly the same Backend context. Its useful when debugging a test, or when running it for the first time, especially if your testing DB/context is different then the one used for test generation. If your Backend response format, changes the Mocked data might not work as expected anymore.
+When generating tests, Checksum records all backend responses, allowing tests to run in the same backend context. This is particularly useful for debugging or initial test runs, especially if your testing database/context differs from that used for test generation. Note that if your backend response format changes, the mocked data may not work as expected.
 
-### CLI Commands (Needs to be updated)
+### CLI Commands
 
-1. `init` - initialize Checksum directory and configs
-2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override the`checksum.config.ts` you can pass full or partial json as a string. E.g. `--checksum-config='{"baseURL" = "https://example.com"}'`
+1. `init` - Initialize the Checksum directory and configurations.
+2. `test` - Run Checksum tests. Accepts all [Playwright command line flags](https://playwright.dev/docs/test-cli). To override `checksum.config.ts`, pass full or partial JSON as a string, e.g., `--checksum-config='{"baseURL": "https://example.com"}'`.
+
+## Running with GitHub Actions
+
+See the example file `github-actions.example.yml`.
+
+## Troubleshooting
+
+**Q: I'm seeing various exceptions when I run "npx checksumai test", even before the test starts.**
+
+A: If you had a pre-installed version of Playwright, it might not be compatible with Checksum. Uninstall the Playwright and Checksum libraries, delete the relevant folder from `node_modules`, and run `npm install -D checksumai`.

package/checksum-root/checksum.config.ts

@@ -16,6 +16,18 @@ export default getChecksumConfig({
    */
   baseURL: "<base URL>",
 
+  /**
+   * Insert the account's username that will be used
+   * to login into your testing environment
+   */
+    username: "<username>",
+
+    /**
+     * Insert the account's password that will be used
+     * to login into your testing environment
+     */
+    password: "<password>",
+
   options: {
     /**
      * Whether to use Checksum Smart Selector when an action fails (see README)

package/checksum-root/{checksum-tests.example.yml → github-actions.example.yml}

@@ -1,5 +1,5 @@
 #################################
-# This file has two example Github workflows to run Checksum tests
+# This file has two example Github Action workflows to run Checksum tests
 # 1. Runs Checksum tests on every push or PR to main/master
 # 2. Runs the test using Docker container
 

package/checksum-root/login.ts

@@ -11,22 +11,22 @@ export default async function login(
    *
    * Example with Seed Function:
    */
-  const apiContext = await request.newContext();
-  const response = await apiContext.get("https://example.com/createseed");
-  const { username, password } = await response.json();
-  await page.goto("/login");
-  await page.getByPlaceholder("Email...").fill(process.env.username);
-  await page.getByPlaceholder("Password...").fill(process.env.password);
-  await page.getByText("Login").click();
-  await expect(page.getByText("Login Successful")).toBeVisible();
+  // const apiContext = await request.newContext();
+  // const response = await apiContext.get("https://example.com/createseed");
+  // const { username, password } = await response.json();
+  // await page.goto("/login");
+  // await page.getByPlaceholder("Email...").fill(process.env.username);
+  // await page.getByPlaceholder("Password...").fill(process.env.password);
+  // await page.getByText("Login").click();
+  // await expect(page.getByText("Login Successful")).toBeVisible();
 
   /**
    * Example with Default Username and Password:
    * This example demonstrates how to log in to a page using a predefined username and password from a config file.
    */
-  await page.goto("/login");
-  await page.getByPlaceholder("Email...").fill(config.username);
-  await page.getByPlaceholder("Password...").fill(config.password);
-  await page.getByText("Login").click();
-  await expect(page.getByText("Login Successful")).toBeVisible();
+  // await page.goto("/login");
+  // await page.getByPlaceholder("Email...").fill(config.username);
+  // await page.getByPlaceholder("Password...").fill(config.password);
+  // await page.getByText("Login").click();
+  // await expect(page.getByText("Login Successful")).toBeVisible();
 }

package/checksum-root/playwright.config.ts

@@ -47,7 +47,7 @@ export default defineConfig({
   projects: [
     {
       name: "chromium",
-      testMatch: /^(?!.*refactored).*spec.*/,
+      testMatch: /checksum.spec/,
       use: {
         ...devices["Desktop Chrome"],
       },

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checksum-ai/runtime",
-  "version": "1.0.39",
+  "version": "1.0.41",
   "description": "Checksum.ai test runtime",
   "main": "index.js",
   "scripts": {

package/postinstall.js

@@ -45,7 +45,7 @@ async function installPlaywright() {
     await run();
     console.log("Playwright installed successfully");
   } catch (e) {
-    console.error("Playwright installation failed", error);
+    console.error("Playwright installation failed", e);
   }
 }