@xoxoai/checkmate 0.4.0

package/docs/GUIDE.md

@@ -0,0 +1,467 @@
+# **_checkmate_** docs
+
+Technical documentation for **_checkmate_** - AI test automation with Playwright.
+
+## Table of Contents
+
+- [Core Concepts](#core-concepts)
+- [Configuration Reference](#configuration-reference)
+- [Writing Effective Tests](#writing-effective-tests)
+- [Cost Management](#cost-management)
+- [Web Extension](#web-extension)
+- [Salesforce Extension](#salesforce-extension)
+- [Test Reports](#test-reports)
+- [Troubleshooting](#troubleshooting)
+- [Architecture](#architecture)
+- [Advanced Topics](#advanced-topics)
+
+## Core Concepts
+
+**_checkmate_** is an AI-driven test runner. You describe a step in natural language, **_checkmate_** runs a tool loop, and the step passes or fails based on the observed result.
+
+Main building blocks:
+
+- **Runner**: The object that executes steps. The main programmatic entry point is `createRunner()` from `@xoxoai/checkmate/core`.
+- **Step**: A plain object with `action` and `expect`. This is the main unit of execution.
+- **Extensions**: Composable modules that add tools and runtime behavior. Built-ins include `web()` and `salesforce()`.
+- **Fixtures**: Convenience Playwright entry points that provide an `ai` runner in tests.
+
+Published entry points:
+
+- `@xoxoai/checkmate/core`: Build your own runner with extensions.
+- `@xoxoai/checkmate/playwright`: Use the built-in web extension with Playwright `test` and `expect`.
+- `@xoxoai/checkmate/salesforce`: Use the built-in web + Salesforce extensions with the same `ai` fixture shape.
+
+Most users start here:
+
+```typescript
+import { test } from '@xoxoai/checkmate/playwright'
+
+test('search flow', async ({ ai }) => {
+	await ai.run({
+		action: 'Search for playwright documentation',
+		expect: 'Search results are displayed',
+	})
+})
+```
+
+## Configuration Reference
+
+### AI API Settings
+
+| Variable                                | Default      | Description                                                                                               |
+| --------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------- |
+| `OPENAI_API_KEY`                        | -            | **Required** - Your OpenAI API key (or compatible provider)                                               |
+| `OPENAI_BASE_URL`                       | -            | Optional - Override for compatible providers (Claude, Gemini, local LLMs)                                 |
+| `OPENAI_MODEL`                          | `gpt-5-mini` | Model: gpt-5, gemini-2.5-flash, claude-4-5-sonnet etc.                                                    |
+| `OPENAI_TEMPERATURE`                    | `1.0`        | Creativity (below 0.5 = deterministic, above 0.5 = creative)                                              |
+| `OPENAI_REASONING_EFFORT`               | -            | Optional - Reasoning effort for models: low, medium, high                                                 |
+| `OPENAI_TIMEOUT_SECONDS`                | `60`         | API request timeout in seconds                                                                            |
+| `OPENAI_API_RATE_LIMIT_DELAY_SECONDS`   | `0`          | Optional fixed delay before each API call, useful when your provider is sensitive to burst traffic        |
+| `OPENAI_RETRY_MAX_ATTEMPTS`             | `3`          | Max retries with backoff (1s, 10s, 60s) for rate limits and server errors                                 |
+| `OPENAI_TOOL_CHOICE`                    | `required`   | Tool choice: auto, required, none                                                                         |
+| `OPENAI_ALLOWED_TOOLS`                  | -            | Comma-separated list of allowed tools (if not set, all tools available)                                   |
+| `OPENAI_INCLUDE_SCREENSHOT_IN_SNAPSHOT` | `false`      | Include compressed screenshots in snapshot responses                                                      |
+| `OPENAI_API_TOKEN_BUDGET_USD`           | -            | Optional - USD budget for total OpenAI API spend per test run. Only positive decimal values are enforced. |
+| `OPENAI_API_TOKEN_BUDGET_COUNT`         | -            | Optional - Token count limit for total tokens per test run. Only positive integers are enforced.          |
+| `OPENAI_LOOP_MAX_REPETITIONS`           | `5`          | Number of repetitive tool call patterns to detect before triggering loop recovery with random temperature |
+| `CHECKMATE_LOG_LEVEL`                   | `off`        | Logging verbosity: debug, info, warn, error, off                                                          |
+| `CHECKMATE_SNAPSHOT_FILTERING`          | `false`      | Enable semantic page snapshot filtering before requests are sent to the model                             |
+
+### Playwright Configuration
+
+Browser settings (viewport, headless mode, video recording, timeouts, etc.) are configured in [playwright.config.ts](../playwright.config.ts) using Playwright's [standard](https://playwright.dev/docs/test-configuration) configuration mechanism.
+
+## Writing Effective Tests
+
+### Best Practices
+
+1. **Be Specific** - Clear expectations help the AI validate success
+2. **One Action Per Step** - Break complex flows into discrete steps
+3. **Include Context** - Mention relevant UI elements and expected behavior
+4. **Add Timing Hints** - For slow operations, mention expected wait times
+5. **Handle Popups** - Explicitly mention consent dialogs or modals
+
+### Basic Example
+
+```typescript
+import { expect, test } from '@xoxoai/checkmate/playwright'
+
+test('search for playwright documentation', async ({ page, ai }) => {
+	await test.step('Navigate to Google', async () => {
+		await ai.run({
+			action: `Open the browser and navigate to google.com`,
+			expect: `google.com is loaded and the search bar is visible`,
+		})
+	})
+
+	await test.step('Search for Playwright', async () => {
+		await ai.run({
+			action: `Type 'playwright test automation' in the search bar and press Enter`,
+			expect: `Search results contain the playwright.dev link`,
+		})
+	})
+
+	await expect(page.getByRole('link', { name: /playwright/i }).first()).toBeVisible()
+})
+```
+
+### Complex Interactions
+
+```typescript
+await test.step('Fill form and submit', async () => {
+	await ai.run({
+		action: `
+            Wait for the newsletter popup (takes ~30 seconds), 
+            then close it by clicking the X button.
+            Scroll to the comment section and click to activate it.
+            Type 'Great article!' into the comment textarea.
+            Click the Submit button.
+        `,
+		expect: `
+            The comment is submitted, 
+            and either a success message appears 
+            or a login form is displayed if not authenticated.
+        `,
+	})
+})
+```
+
+### Programmatic Composition
+
+Use `@xoxoai/checkmate/core` when you want to build your own runner explicitly:
+
+```typescript
+import { createRunner } from '@xoxoai/checkmate/core'
+import { web } from '@xoxoai/checkmate/playwright'
+import { jira, notion, database } from 'your-own-extension-examples'
+
+const ai = createRunner({
+	extensions: [web({ page }), jira(), notion(), database()],
+})
+```
+
+## Cost Management
+
+**_checkmate_** includes built-in token usage monitoring:
+
+```json
+{
+	"response input": "2543 @ $0.00$",
+	"response output": "456 @ $0.00$",
+	"history (estimated)": 45234,
+	"step input": "5123 @ $0.00$",
+	"step output": "892 @ $0.00$",
+	"test input": "25678 @ $0.01$",
+	"test output": "4521 @ $0.01$"
+}
+```
+
+### Cost Optimization Features
+
+1. **Smart Snapshots** - Instead of full HTML, only the ARIA accessibility tree is sent to the AI
+2. **History Filtering** - Continuously filters old page snapshots (reduces token usage by up to 50%)
+3. **Snapshot Minification** - Removes unnecessary whitespace and quotes from ARIA snapshots
+4. **Snapshot Filtering** - Local semantic filtering of page snapshots using the current step description (reduces token usage by up to 90%)
+5. **Screenshots** - Normalized and compressed locally, helps vision models understand UI better
+6. **Chat Recycling** - New session per step to prevent context bloat and isolation
+7. **Token Counting** - Real-time usage tracking per step and test with budgets
+8. **Loop Detection** - Detects and mitigates repetitive tool call patterns, preventing AI runaway costs
+
+### Budgeting & Cost Limits
+
+You can set one or both token budget environment variables to enforce limits during a single test run.
+
+- `OPENAI_API_TOKEN_BUDGET_USD` — Sets a USD budget (e.g. 0.50) per test execution. The framework checks the current estimated cost (input+output tokens) and throws an error if the budget is exceeded.
+- `OPENAI_API_TOKEN_BUDGET_COUNT` — Sets a token limit (e.g. 100000). The framework tracks input and output tokens across the test and throws an error when the total exceeds this limit.
+
+Notes:
+
+- Only positive numbers are enforced; `0` or non-positive values are effectively treated as disabled.
+- If the env var is unset or invalid (non-number), it is ignored.
+
+### Using Snapshot Filtering for Token Optimization
+
+When snapshot filtering is enabled, **_checkmate_** scores the page snapshot locally with a semantic embedding model and keeps the most relevant branches of the accessibility tree.
+
+Default behavior:
+
+- Build one query from `action + expect`
+- Score snapshot keys and string leaves against that query
+- If `search` is provided on the step, use those keywords instead of semantic `action + expect`
+- Keep the top `10%` of scored elements by default
+- If top-percent selection yields nothing, fall back to hard threshold `0.3`
+
+**This feature significantly reduces the payload size, minimizing costs while improving AI determinism, reliability and speed.**
+
+```typescript
+await ai.run({
+	action: `Click on the link that leads to playwright.dev`,
+	expect: `The playwright.dev homepage is displayed`,
+
+	// optional snapshot filtering override
+	topPercent: 20,
+})
+```
+
+```
+debug: Scored 107 elements
+debug: Filtered to 21 elements from top 20%
+debug: Reduced snapshot from 4283 to 326 chars (92% reduction)
+```
+
+Feature is controlled by the `CHECKMATE_SNAPSHOT_FILTERING` environment variable (default: `false`). Set it explicitly to `true` to enable filtering. `search` is now an explicit keyword query override, and `topPercent` lets you tune how much of the scored snapshot should be kept for a specific step.
+
+The model can still request a full snapshot with the browser snapshot tool if the filtered tree is insufficient, so steps should not fail just because the initial snapshot was compact.
+
+For optimal results, write concrete `action` and `expect` text. Use `topPercent` as a real percentage from `1` to `100` when you need to keep more or less of the scored snapshot. Optional `search` terms still help when you want direct keyword control.
+
+**Tips for effective step text:**
+
+- Include relevant UI element types (button, input, link, checkbox, etc.)
+- Include key text that appears on the page
+- Include action-related terms (search, filter, submit, etc.)
+- Keep the step focused on one user intent
+- Use `topPercent` only when you need to tune how aggressively snapshot content is pruned
+
+### Estimated Costs
+
+**Gemini-2.5-flash / GPT-5-mini**:
+
+- Simple test (~5 steps): ~$0.01 - $0.05
+- Complex test (~20 steps): ~$0.10 - $0.40
+- Full E2E suite (~50 complex tests): ~$5.00 - $20.00
+
+**GPT-OSS-20B via groq**:
+
+- Simple test (~5 steps): ~$0.001 - $0.01
+- Complex test (~20 steps): ~$0.01 - $0.05
+- Full E2E suite (~50 complex tests): ~$1.00 - $2.00
+
+_Costs vary based on model, screenshot size and count, and page complexity_
+
+## Web Extension
+
+`@xoxoai/checkmate/playwright` is the pre-built web entry point. It composes the core runner with the built-in `web()` extension and exposes a Playwright-friendly `ai` fixture.
+
+What it adds:
+
+- browser tools for navigation and interaction
+- initial page snapshots and optional screenshots
+- `test`, `expect`, `web()`, and `createPlaywrightRunner(page)` exports
+
+```typescript
+import { test } from '@xoxoai/checkmate/playwright'
+
+test('search flow', async ({ ai }) => {
+	await ai.run({
+		action: 'Search for playwright documentation',
+		expect: 'Search results are displayed',
+	})
+})
+```
+
+## Salesforce Extension
+
+`@xoxoai/checkmate/salesforce` builds on the web extension. It adds Salesforce-specific tools and keeps the same `ai` fixture shape as the Playwright entry point.
+
+What it adds:
+
+- the built-in `salesforce()` extension
+- `test`, `expect`, and `createSalesforceRunner(page)` exports
+- the `login_to_salesforce_org` tool backed by the Salesforce CLI
+
+Prerequisites:
+
+```bash
+# Install Salesforce CLI
+npm install -g @salesforce/cli
+
+# Authenticate to your org and set is as default
+sf org login web --alias my-checkmate-org --set-default
+```
+
+```typescript
+import { test } from '@xoxoai/checkmate/salesforce'
+
+test('create and configure itinerary', async ({ ai }) => {
+	await test.step('Login to Salesforce', async () => {
+		await ai.run({
+			action: 'Login to Salesforce org and open Test QA Application',
+			expect: 'Test QA homepage is displayed',
+		})
+	})
+})
+```
+
+The `login_to_salesforce_org` tool handles the authentication flow by retrieving a front-door URL from the authenticated SF CLI session and navigating the browser for you.
+
+## Test Reports
+
+Multiple report formats are generated after each run:
+
+- **HTML Report**: `test-reports/html/index.html` (interactive - no screenshots/video yet though)
+- **JUnit XML**: `test-reports/junit/results.xml` (CI/CD integration)
+- **Console Output**: Real-time step results and token usage
+
+```bash
+# Open HTML report in browser
+npx playwright show-report test-reports/html
+```
+
+## Troubleshooting
+
+### AI makes incorrect decisions
+
+**Symptoms**: The AI clicks wrong elements, misinterprets the page, or fails to complete actions correctly.
+
+**Solutions**:
+
+- Provide more precise descriptions in `action` and more focused assertions in `expect`
+- Reference specific element identifiers and roles (for example: text, label, button, list)
+- Break complex workflows into single-action steps; use a step-by-step approach
+
+### Tests loop during step execution
+
+**Symptoms**: The AI repeats the same actions or gets stuck in a loop, consuming tokens unnecessarily.
+
+**Solutions**:
+
+- Increase `OPENAI_TEMPERATURE` to encourage exploration
+- Use a reasoning/thinking model (if available) to improve planning and avoid repetitive loops
+
+### High token costs
+
+**Symptoms**: Tests consume more tokens than expected, leading to high API costs.
+
+**Solutions**:
+
+- Set a lower reasoning effort: `OPENAI_REASONING_EFFORT`
+- Consider disabling `OPENAI_INCLUDE_SCREENSHOT_IN_SNAPSHOT`
+- Use a cheaper model, lower-end models often perform well (e.g., `gemini-2.5-flash-lite` or `gpt-5-nano`)
+
+### Rate limiting errors
+
+**Symptoms**: API calls fail with 429 errors or rate limit messages.
+
+**Solutions**:
+
+- The framework automatically retries with backoff (1s, 10s, 60s)
+- Upgrade your API plan with your provider
+- Reduce concurrent test execution
+- Increase `OPENAI_TIMEOUT_SECONDS` if needed
+
+### Timeout errors
+
+**Symptoms**: Tests fail with timeout errors before completing actions.
+
+**Solutions**:
+
+- Increase `OPENAI_TIMEOUT_SECONDS` in your `.env` file
+- Mention expected wait times in your action descriptions
+- Break long-running actions into smaller steps
+
+## Architecture
+
+**_checkmate_** combines multiple components to enable AI-driven test automation:
+
+```
+@xoxoai/checkmate/core
+│
+├── createRunner({ extensions })
+├── runtime/
+│   ├── CheckmateRunner
+│   ├── StepExecution
+│   └── ExtensionHost
+│
+├── ai/
+│   ├── AiClient
+│   ├── ResponseProcessor
+│   ├── MessageHistory
+│   └── TokenTracker
+│
+├── tools/
+│   └── step/
+│       └── StepResultTools
+│
+├── @xoxoai/checkmate/playwright
+│   └── web()
+│       ├── BrowserToolRuntime
+│       ├── SnapshotService
+│       └── Browser tools
+│
+└── @xoxoai/checkmate/salesforce
+    └── salesforce()
+        ├── SalesforceTools
+        └── Salesforce CLI integration
+```
+
+### Key Components
+
+**Test Layer**
+
+- Playwright Test framework manages test execution, reporting, and fixtures
+- Tests written in natural language via `ai.run()` fixtures
+
+**Core Engine**
+
+- **createRunner**: Public composition entry point for building runners from extensions
+- **CheckmateRunner**: Runtime instance returned by `createRunner`
+- **AiClient**: Manages model interactions, retries, and tool-calling requests
+- **Response Processor**: Handles tool responses, append-only history, and retries through the step loop
+- **ExtensionHost**: Registers tools, instructions, step context builders, and post-tool hooks from extensions
+- **Tool Registry**: Owns Zod-defined tool declarations and explicit tool resolution
+
+**Tools**
+
+- **Core Tools**: Step control (pass/fail step assertions)
+- **Web Extension**: Playwright-powered browser tools, snapshots, and screenshots
+- **Salesforce Extension**: SF CLI login flow layered on top of the web extension
+
+**Cost Optimization**
+
+- Token tracking with budget enforcement
+- History filtering (removes old snapshots)
+- Snapshot minification and screenshot compression
+- Loop detection and mitigation
+
+**Configuration**
+
+- Test, Reporting and Browser settings: [playwright.config.ts](../playwright.config.ts)
+- API & AI settings: `.env` file
+
+## Advanced Topics
+
+### Custom Tool Integration
+
+For custom tools, extensions, built-in extension composition, and custom runners, see the dedicated [Extensions guide](./EXTENSIONS.md).
+
+### Performance Optimization
+
+For large test suites:
+
+- Use faster models for simple tests (e.g., `gemini-3-flash-preview` or `gpt-5-mini`)
+- Set token budgets to prevent runaway costs
+- Disable screenshots in snapshots when visual context isn't needed
+- Consider parallel test execution with Playwright's workers
+
+### CI/CD Integration
+
+**_checkmate_** generates JUnit XML reports compatible with most CI/CD systems:
+
+```yaml
+# Example GitHub Actions
+- name: Run Tests
+  run: npm test
+
+- name: Upload Reports
+  uses: actions/upload-artifact@v3
+  with:
+    name: test-reports
+    path: test-reports/
+```
+
+## See Also
+
+- [EXTENSIONS](./EXTENSIONS.md)
+- [README](../README.md)

package/docs/ROADMAP.md

@@ -0,0 +1,47 @@
+# Roadmap
+
+## Current State:
+
+- ✅ Extension-composed runtime via `createRunner({ extensions })`
+- ✅ Clear top-level module boundaries: `runtime`, `ai`, `tools`, `integrations`, `config`, `logging`
+- ✅ Explicit tool registration and dispatch
+- ✅ Browser snapshot filtering with semantic scoring
+- ✅ Token tracking, retry handling, loop detection, and screenshot support
+- ✅ Salesforce login integration through the SF CLI
+- ✅ Published subpath entry points for `@xoxoai/checkmate/core`, `@xoxoai/checkmate/playwright`, and `@xoxoai/checkmate/salesforce`
+
+## Near Term
+
+Focus: Stability, extension points, and better contributor ergonomics.
+
+- ✅ Custom tool registration API for external integrations
+- ✅ Better public examples for programmatic runner usage
+- ✅ Publishable npm package layout with dedicated `core`, `playwright`, and `salesforce` entry points
+- [ ] Visual interactions (click, drag, etc.) in the Playwright extension
+- [ ] Snapshot filtering tuning hooks beyond top-percent selection
+- [ ] Better reporting around filtered snapshot size and selected branches
+
+## Mid Term
+
+Focus: Product usability and broader workflow support.
+
+- [ ] UI layer for recording, editing, and replaying AI-driven steps
+- [ ] Flow-level execution mode for multi-step business journeys
+- [ ] Richer debugging output for model/tool reasoning failures
+- [ ] Better parallel execution support across large suites
+
+## Long Term
+
+Focus: Production hardening and ecosystem.
+
+- [ ] Stronger observability and explainable AI
+- [ ] Test generation from specs and recorded user behavior
+- [ ] Advanced reporting with AI-assisted failure summaries
+- [ ] Enterprise-focused environment and secret management support
+
+## Ongoing Research
+
+- 🔄 Faster local retrieval/filtering for very large page snapshots
+- 🔄 Hybrid semantic plus structural ranking for element selection
+- 🔄 Multi-agent execution models for planning and validation
+- 🔄 Confidence signals for tool selection and assertions

package/package.json

@@ -0,0 +1,106 @@
+{
+	"name": "@xoxoai/checkmate",
+	"version": "0.4.0",
+	"description": "AI‑driven e2e test automation framework built on Playwright Test and OpenAI API",
+	"homepage": "https://github.com/dawiddiwad/checkmate",
+	"repository": {
+		"type": "git",
+		"url": "https://github.com/dawiddiwad/checkmate.git"
+	},
+	"keywords": [
+		"test-automation",
+		"e2e-testing",
+		"playwright",
+		"openai",
+		"ai-testing",
+		"low-code",
+		"salesforce",
+		"gemini",
+		"claude",
+		"groq"
+	],
+	"files": [
+		"dist",
+		"docs/EXTENSIONS.md",
+		"docs/GUIDE.md",
+		"docs/ROADMAP.md",
+		"README.md",
+		"LICENSE"
+	],
+	"publishConfig": {
+		"access": "public"
+	},
+	"main": "dist/core.js",
+	"types": "dist/core.d.ts",
+	"type": "module",
+	"exports": {
+		".": {
+			"types": "./dist/core.d.ts",
+			"default": "./dist/core.js"
+		},
+		"./core": {
+			"types": "./dist/core.d.ts",
+			"default": "./dist/core.js"
+		},
+		"./playwright": {
+			"types": "./dist/playwright.d.ts",
+			"default": "./dist/playwright.js"
+		},
+		"./salesforce": {
+			"types": "./dist/salesforce.d.ts",
+			"default": "./dist/salesforce.js"
+		}
+	},
+	"scripts": {
+		"checkmate:install": "npm install && npx playwright install",
+		"build": "npm run validation:check && npm run build:dist",
+		"build:dist": "tsc -p tsconfig.build.json",
+		"clean": "rm -rf dist",
+		"prepack": "npm run clean && npm run build",
+		"validation:check": "npm run compile:check && npm run lint:check && npm run format:fix && npm run test:unit:run",
+		"compile:check": "npx tsc --noEmit",
+		"lint:check": "eslint .",
+		"lint:fix": "eslint . --fix",
+		"format:check": "prettier . --check",
+		"format:fix": "prettier . --write",
+		"test:unit": "vitest --config src/test/vitest.config.ts",
+		"test:unit:ui": "vitest --ui --config src/test/vitest.config.ts",
+		"test:unit:coverage": "vitest --coverage --config src/test/vitest.config.ts",
+		"test:unit:run": "vitest run --config src/test/vitest.config.ts",
+		"test:web": "npm run build:dist && npx playwright test --project=web",
+		"test:salesforce": "npm run build:dist && npx playwright test --project=salesforce",
+		"test:web:example": "npm run build:dist && npx playwright test --project=web --grep=ollama",
+		"show:report": "npx playwright show-report test-reports/html"
+	},
+	"author": "Dawid Dobrowolski SoftQA",
+	"license": "MIT",
+	"dependencies": {
+		"@huggingface/transformers": "^4.0.1",
+		"openai": "^6.34.0",
+		"sharp": "^0.34.5",
+		"string-similarity": "^4.0.4",
+		"strip-ansi": "^7.2.0",
+		"winston": "^3.19.0",
+		"yaml": "^2.8.3",
+		"zod": "^4.3.6"
+	},
+	"peerDependencies": {
+		"@playwright/test": "^1.59.1"
+	},
+	"devDependencies": {
+		"@eslint/js": "^10.0.1",
+		"@playwright/test": "^1.59.1",
+		"@types/node": "^25.6.0",
+		"@types/string-similarity": "^4.0.2",
+		"@vitest/coverage-v8": "^4.1.4",
+		"@vitest/ui": "^4.1.2",
+		"dotenv": "^17.4.1",
+		"eslint": "^10.2.0",
+		"eslint-config-prettier": "^10.1.8",
+		"prettier": "^3.8.3",
+		"tsx": "^4.21.0",
+		"typescript": "^5.9.3",
+		"typescript-eslint": "^8.58.2",
+		"vitest": "^4.1.2"
+	}
+}