@cementic/cementic-test 0.2.3 → 0.2.5

package/README.md

@@ -1,625 +1,468 @@
-**README.md** for CementicTest (CT).
+# CementicTest CLI
 
-# CementicTest CLI (v0.2)
+CementicTest CLI (`ct`) turns plain-English test cases into runnable Playwright tests with a Page Object Model-oriented workflow.
 
-Turn messy human test ideas into runnable Playwright tests — while staying 100% compatible with classic Playwright + POM.
+Current pipeline:
 
-## Installation
-
-```bash
-# Run directly (no install needed)
-npx @cementic/cementic-test new my-app
-
-# Or install globally
-npm install -g @cementic/cementic-test
-```
-
-## Quick Start
-
-```bash
-# 1. Scaffold a new project
-ct new "Demo"
-
-# 2. Generate test cases with AI
-cd Demo
-ct tc --ai --feature "Login" --count 3
-
-# 3. Generate & Run
-ct flow
-```
-
----
-
-# 🧪 CementicTest (CT)
-
-### **AI-Assisted Playwright Automation — From Test Cases to POM Test Suites in Minutes**
-
-CementicTest (CT) is a CLI tool that turns **written or AI-generated test cases** into a complete, production-ready **Playwright + Page Object Model** automation framework.
-
-CT can:
-
-* Scaffold new Playwright projects
-* Scrape real URLs for UI context
-* Generate test cases using LLMs (Bring Your Own API Key)
-* Normalize test cases
-* Generate full Playwright tests using POM standards
-* Run everything instantly via `ct test`
-* Enhance existing Playwright projects
-* Allow optional migration from raw PW → CT-powered PW
-
-CT removes boilerplate, enforces good practices, and massively accelerates setup and test authoring.
-
----
-
-# ⭐️ Features Overview
-
-## ✅ **1. Project Initialization (POM-ready workspace)**
-
-`ct new "<projectName>"` creates a complete, ready-to-run test project:
-
-```
-pages/
-tests/
-utils/
-test-data/
-cases/
-.cementic/
-  normalized/
-playwright.config.ts
+```text
+ct tc -> ct normalize -> ct gen -> ct test
 ```
 
-Includes:
-
-* Installed Playwright & browsers
-* Sample login page
-* Sample test
-* Opinionated project structure
-* POM-first architecture layout
+It can scaffold a project, write or AI-generate case files, normalize them into JSON, generate Playwright specs plus page objects, and run the resulting suite.
 
----
+## What this version does
 
-## ✅ **2. AI-powered Test Case Writer**
-
-Using your own LLM API key (`CT_LLM_API_KEY`, `OPENAI_API_KEY`, etc.):
-
-* Scrapes page content
-* Extracts headings, buttons, inputs, metadata
-* Sends prompt + test spec to LLM
-* Generates structured Markdown test cases:
-
-```
-# AUTH-001 — Valid Login @smoke @auth
-## Steps
-1. Navigate to /login
-...
-## Expected
-- Dashboard loads
-- User profile visible
-```
+- Scaffolds new Playwright projects in JavaScript or TypeScript
+- Generates Markdown test cases manually or with AI
+- Supports URL-aware case generation with live page capture
+- Normalizes case files into structured JSON
+- Generates Playwright specs and POM classes from normalized cases
+- Avoids overwriting existing generated page-object files
+- Runs Playwright tests through a CLI wrapper
+- Opens Playwright HTML reports
+- Serves Allure reports if Allure is installed in the target project
+- Generates a GitHub Actions workflow for Playwright CI
 
-Supports:
-
-### **Interactive writing**
-
-```
-ct tc
-```
+## Installation
 
-### **AI-writing**
+```bash
+# Run without a global install
+npx @cementic/cementic-test --help
 
+# Or install globally
+npm install -g @cementic/cementic-test
 ```
-ct tc --ai
-```
-
-### **URL-aware AI-writing (scrape + understand UI)**
 
-```
-ct tc url https://mini-bank.testamplify.com/login --ai
-```
+Requirements:
 
----
+- Node.js 18+
+- npm
 
-## ✅ **3. Test Case Normalization**
+## Quick start
 
-CT normalizes Markdown test cases into structured machine-readable JSON.
+### JavaScript project
 
-```
-ct normalize ./cases
+```bash
+ct new demo-js --lang js
+cd demo-js
+ct tc --feature "Login form" --count 2
+ct normalize ./cases --and-gen --lang js
+ct test
 ```
 
-Output goes to:
+### TypeScript project
 
+```bash
+ct new demo-ts --lang ts
+cd demo-ts
+ct tc url https://mini-bank.testamplify.com/login --feature "Login" --count 2
+ct normalize ./cases --and-gen --lang ts
+ct test
 ```
-.cementic/normalized/*.json
-```
-
-Each JSON contains:
-
-* ID
-* Title
-* Steps (array)
-* Expected results
-* Metadata / tags
 
----
+## Command flow
 
-## ✅ **4. Automatic Playwright Test Generation**
+### 1. `ct new <projectName>`
 
-Once normalized, CT can generate full Playwright tests using the POM architecture:
+Scaffolds a new CementicTest + Playwright project.
 
-```
-ct gen --lang ts
+```bash
+ct new my-suite
+ct new my-suite --lang ts
+ct new my-suite --no-browsers
 ```
 
-Generated tests go into:
+Current scaffold support:
 
-```
-tests/*.spec.ts
-```
+- `--lang js`: JavaScript template with:
+  - `pages/BasePage.js`
+  - `pages/LoginPage.js`
+  - `pages/DashboardPage.js`
+  - `pages/FormPage.js`
+  - `tests/login.spec.js`
+  - `tests/dashboard.spec.js`
+- `--lang ts`: TypeScript template with:
+  - `pages/LandingPage.ts`
+  - `tests/landing.spec.ts`
+  - `tsconfig.json`
+  - `playwright.config.ts`
 
-Generated tests **import your Page Objects**:
+What `ct new` does:
 
-```ts
-import { LoginPage } from "../pages/login.page";
+- creates the project directory
+- copies the selected template
+- initializes git
+- runs `npm install`
+- optionally runs `npx playwright install`
 
-test('AUTH-001 — Valid Login', async ({ page }) => {
-  const login = new LoginPage(page);
-  await login.goto();
-  await login.login("user", "pass");
-});
-```
+### 2. `ct tc`
 
----
+Creates Markdown case files in `./cases`.
 
-## ✅ **5. Normalize + Generate in one shot**
+Interactive mode:
 
-Most developers will use:
-
-```
-ct normalize ./cases --and-gen --lang ts
+```bash
+ct tc
 ```
 
-This:
+Non-interactive mode:
 
-1. Normalizes test cases → JSON
-2. Generates Playwright tests immediately
-
----
-
-## ✅ **6. Execute Playwright tests**
-
-CT wraps Playwright for convenience:
-
-```
-ct test
+```bash
+ct tc --feature "Checkout" --desc "Guest checkout flow" --count 3
 ```
 
-Equivalent to:
+AI mode:
 
+```bash
+ct tc --ai --feature "Login" --count 3
 ```
-npx playwright test
-```
-
----
 
-## ✅ **7. Bring Your Own LLM (BYO LLM API key)**
+### 3. `ct tc url <url>`
 
-CT is **LLM-agnostic**, meaning it does NOT force you to use OpenAI.
+Generates cases with awareness of a live page.
 
-You can set:
-
-```
-CT_LLM_API_KEY
-OPENAI_API_KEY
-CT_LLM_MODEL
-CT_LLM_BASE_URL
+```bash
+ct tc url https://mini-bank.testamplify.com/login --feature "Login" --count 2
+ct tc url https://mini-bank.testamplify.com/login --ai --feature "Login" --count 2
+ct tc url https://mini-bank.testamplify.com/login --ai --headed --feature "Login"
+ct tc url https://mini-bank.testamplify.com/login --ai --capture-only --feature "Login"
 ```
 
-Works with:
-
-* OpenAI
-* Anthropic (Claude-compatible proxies)
-* Gemini (OpenAI-compatible API wrappers)
-* DeepSeek
-* Any **OpenAI-compatible LLM server**
-
-If AI fails or no key exists?
-✨ CT gracefully falls back to manual templates.
-
----
-
-## ✅ **8. URL Scraper (Headless page scanner)**
-
-Given a URL, CT extracts:
-
-* Page title
-* Headings
-* Buttons
-* Inputs
-* Basic layout info
-
-This helps the LLM generate highly accurate test cases.
-
----
+Current behavior:
 
-## ⭐️ CLI Commands Reference
+- captures a live page with Playwright and extracts headings, buttons, links, inputs, status regions, and selector candidates
+- sends that captured element map to the capture-aware AI flow when `--ai` is enabled
+- saves the full capture artifact to `.cementic/capture/capture-*.json`
+- saves a runnable preview spec to `tests/preview/spec-preview-*.spec.cjs` when AI scenarios are generated
+- writes `<!-- ct:url ... -->` metadata plus selector and `playwright` hints into generated Markdown so normalization and generation preserve the capture output
 
-Below is every command CT currently supports.
+### 4. `ct normalize <path>`
 
----
-
-# 📦 **CLI Commands**
-
----
-
-## `ct new "<projectName>"`
-
-Scaffold a new CementicTest + Playwright project from scratch.
-
-**Examples:**
+Converts case files into normalized JSON in `.cementic/normalized/`.
 
 ```bash
-ct new "Bank"
-ct new "Bank" --no-browsers  # Skip browser download
+ct normalize ./cases
+ct normalize "cases/**/*.md"
+ct normalize ./cases --and-gen --lang ts
 ```
 
-What it does:
+Current normalized output includes:
 
-* Installs Playwright
-* Runs browser install
-* Creates POM structure
-* Adds sample pages & test
-* Ready in under 30 seconds
+- `id`
+- `title`
+- `tags`
+- `steps`
+- `expected`
+- `needs_review`
+- `review_reasons`
+- `source`
+- `url`
 
----
+Important current behavior:
 
-## `ct tc`
+- title is stored without duplicating the case ID prefix
+- URL metadata is read from `<!-- ct:url ... -->` first
+- if no metadata exists, normalization falls back to URL extraction from steps
 
-Interactive test case writer.
+### 5. `ct gen`
 
-```bash
-# Interactive mode
-ct tc
+Generates Playwright specs and page-object files from normalized JSON.
 
-# Non-interactive mode (CI/Scripting)
-ct tc --feature "Login" --desc "Login page" --count 5
+```bash
+ct gen --lang ts
+ct gen --lang js
+ct gen --lang ts --out tests/e2e
 ```
 
-Prompts (Interactive):
+Current output locations:
 
-* Feature/page name
-* Optional app description
-* Number of test cases
+- specs: `tests/generated/` by default
+- page objects: `pages/`
 
-Generates Markdown inside `/cases`.
+Current generator behavior:
 
----
+- maps common natural-language steps to Playwright actions
+- maps common expected-result phrases to Playwright assertions
+- derives one POM class per feature prefix where possible
+- uses `norm.url` for page-object `goto()`
+- avoids duplicate navigation when setup already handles the target URL
+- does not overwrite existing POM files
+- emits correct relative imports for generated specs
 
-## `ct tc --ai`
+### 6. `ct test`
 
-Same as above, but uses AI (requires API key):
+Runs Playwright tests.
 
 ```bash
-ct tc --ai
+ct test
+ct test --headed
+ct test --project chromium
 ```
 
----
-
-## `ct tc url <url> --ai`
-
-Scrape + AI test case writer.
+This is a thin wrapper over:
 
 ```bash
-ct tc url https://mini-bank.testamplify.com/login --ai
+npx playwright test
 ```
 
-This is the most powerful mode.
+Arguments after `ct test` are passed through to Playwright.
 
----
+### 7. `ct flow`
 
-## `ct normalize <path>`
+Runs the end-to-end workflow:
 
-Convert human-readable test cases into machine-readable JSON format.
+1. normalize
+2. generate
+3. test
 
 ```bash
-ct normalize ./cases
-```
-
-Outputs:
-
-```
-.cementic/normalized/*.json
+ct flow
+ct flow ./cases
+ct flow --lang js
+ct flow --no-run
 ```
 
----
-
-## `ct normalize <path> --and-gen --lang ts`
+### 8. `ct report`
 
-Normalize AND generate tests in one step.
+Opens the Playwright HTML report.
 
 ```bash
-ct normalize ./cases --and-gen --lang ts
+ct report
 ```
 
----
-
-## `ct gen --lang ts`
-
-Generate Playwright test files from normalized JSON cases.
+Equivalent behavior:
 
 ```bash
-ct gen --lang ts
-ct gen --lang ts --out tests/e2e  # Custom output directory
+npx playwright show-report
 ```
 
----
-
-## `ct test`
+### 9. `ct serve`
 
-Runs Playwright test runner.
+Serves the Allure report if `allure-commandline` is available in the current project.
 
 ```bash
-ct test
+ct serve
 ```
 
----
+Current behavior:
+
+- prefers the local `node_modules/allure-commandline/bin/allure` binary
+- falls back to `npx allure serve ./allure-results`
 
-## `ct flow`
+### 10. `ct ci`
 
-**One-Shot Command**: Normalizes cases, generates tests, and runs them in sequence.
+Generates a GitHub Actions workflow at `.github/workflows/cementic.yml`.
 
 ```bash
-ct flow
-ct flow --lang js
-ct flow --no-run  # Normalize & Generate only (skip execution)
+ct ci
 ```
 
----
+Current workflow behavior:
 
-## `ct report`
+- skips generation if the workflow file already exists
+- installs dependencies
+- installs Playwright browsers
+- runs `npx playwright test`
+- uploads the Playwright HTML report as an artifact
 
-Opens the Playwright HTML report in your default browser.
+## AI provider support
 
-```bash
-ct report
-```
+Current AI flows live in `src/core/llm.ts` and `src/core/analyse.ts`.
 
----
+Provider behavior in this version:
 
-## `ct serve`
+- `ct tc --ai` uses the standard Markdown case writer in `src/core/llm.ts`
+- `ct tc url --ai` uses the capture-aware analysis flow in `src/core/analyse.ts`
+- manual template generation is used if AI generation fails
 
-Starts the Allure report server (if Allure is configured).
+Supported environment variables:
 
-```bash
-ct serve
-```
+| Variable | Purpose |
+| --- | --- |
+| `DEEPSEEK_API_KEY` | DeepSeek key for capture-aware analysis |
+| `ANTHROPIC_API_KEY` | Primary Anthropic key |
+| `CT_ANTHROPIC_API_KEY` | Alternate Anthropic key |
+| `GEMINI_API_KEY` | Gemini key for capture-aware analysis |
+| `OPENAI_API_KEY` | OpenAI key |
+| `QWEN_API_KEY` | Qwen key for capture-aware analysis |
+| `KIMI_API_KEY` | Kimi / Moonshot key for capture-aware analysis |
+| `CT_LLM_API_KEY` | Generic OpenAI-compatible API key |
+| `CT_LLM_PROVIDER` | Optional provider override (`deepseek`, `anthropic`, `gemini`, `qwen`, `kimi`, or `openai`) |
+| `CT_LLM_MODEL` | Model override |
+| `CT_LLM_BASE_URL` | OpenAI-compatible base URL override |
 
----
+Current defaults:
 
-## `ct ci`
-
-Generates a GitHub Actions workflow file (`.github/workflows/cementic.yml`) for CI/CD.
-
-```bash
-ct ci
-```
+- DeepSeek default: `deepseek-chat`
+- Anthropic default: `claude-sonnet-4-5`
+- Gemini default: `gemini-2.5-flash`
+- Qwen default: `qwen-plus`
+- Kimi default: `moonshot-v1-8k`
+- OpenAI-compatible default: `gpt-4o-mini`
 
----
+## URL capture
 
-# ⚙️ Environment Variables
+Current capture behavior lives in `src/core/capture.ts`.
 
-| Variable          | Description                             |
-| ----------------- | --------------------------------------- |
-| `CT_LLM_API_KEY`  | Primary AI key                          |
-| `OPENAI_API_KEY`  | Secondary AI key                        |
-| `CT_LLM_MODEL`    | Override default model (`gpt-4.1-mini`) |
-| `CT_LLM_BASE_URL` | Custom LLM endpoint                     |
-| `CT_DEBUG`        | Enables verbose logging                 |
+Primary path:
 
----
+- uses a real Playwright browser
+- extracts:
+  - page title
+  - headings
+  - buttons
+  - links
+  - inputs with label, placeholder, name, type, and `data-testid`
+  - status and alert regions
+  - selector confidence and alternative selectors
+- builds a structured `ElementMap`
+- passes that capture into `src/core/analyse.ts` for evidence-backed scenario generation
+- formats markdown, JSON, and preview-spec outputs through `src/core/report.ts`
 
-# 📁 Project Structure
+## Generated project structure
 
-After running `ct new “MyProject”`:
+After a typical run:
 
-```
+```text
 project/
+  cases/
   pages/
-    login.page.ts
-    ...
   tests/
-    sample.spec.ts
-  test-data/
-  utils/
-  cases/
+    generated/
   .cementic/
     normalized/
-  playwright.config.ts
+  playwright.config.js|ts
 ```
 
----
+Typical file flow:
 
-# 🧩 Architecture Overview
-
-### **Input**
+```text
+cases/*.md
+  -> .cementic/normalized/*.json
+  -> tests/generated/*.spec.ts|js
+  -> pages/*.ts|js
+```
 
-* Human test cases (Markdown)
-* AI-generated test cases
-* Scraped UI context
+## Example case file
 
-### **Processing**
+```md
+# AUTH-001 — User can log in @smoke @auth
+<!-- ct:url https://mini-bank.testamplify.com/login -->
+## Steps
+1. Navigate to https://mini-bank.testamplify.com/login
+2. Fill in email with [EMAIL_ADDRESS]'
+3. Fill in password with 'secret'
+4. Click the 'Sign In' button
 
-1. Normalize (Markdown → JSON)
-2. Generate POM-based Playwright tests
-3. Maintain test ID prefixes (AUTH-###, DASH-###, etc.)
+## Expected Results
+- Page redirects to /dashboard
+- Welcome message is visible
+```
 
-### **Output**
+## Current implementation notes
 
-* Full Playwright automation suite ready to run
+This version is intentionally heuristic and file-oriented.
 
----
+Important current characteristics:
 
-# 🚀 Developer Setup (For Internal Development)
+- generated specs are POM-oriented, not raw recorded scripts
+- capture-generated selector and assertion hints are preserved through `normalize` and `gen`
+- generator rules still cover non-capture cases heuristically, but capture-backed cases now keep exact selectors and assertions
+- Playwright itself is used for capture and test execution
+- Playwright CLI and Playwright agents are not yet first-class runtime backends in this version
 
-### Clone & install dependencies
+## Developer setup
 
 ```bash
 git clone <repo>
-cd cementic-test
+cd cementic-test-cli
 npm install
 ```
 
-### Build the CLI
+Build the CLI:
 
 ```bash
 npm run build
 ```
 
-### Link locally
+Run the repo test suite:
 
 ```bash
-npm link
+npm test
 ```
 
-Test globally:
+What `npm test` currently verifies:
+
+- build succeeds
+- `ct tc url` writes URL metadata
+- capture-generated selector and `playwright` hints survive `normalize` and `gen`
+- `ct normalize` and `ct gen` preserve the recent generator fixes
+- the JavaScript scaffold contains the expected page objects and sample specs
+
+## Local development loop
 
 ```bash
+npm run build
+npm link
+
 ct --help
 ```
 
----
-
-# 🧪 Developer Testing Loop
-
-1. Modify code
-2. Run:
+Manual smoke flow:
 
-```
-npm run build && npm link
-```
+```bash
+mkdir demo
+cd demo
 
-3. In a sample project:
+ct new sample --lang ts --no-browsers
+cd sample
 
-```
-mkdir demo && cd demo
-ct new "Shop"
-ct tc url https://example.com/login --ai
+ct tc url https://mini-bank.testamplify.com/login --ai --feature "Login" --count 1
 ct normalize ./cases --and-gen --lang ts
 ct test
 ```
 
----
-
-# 🏗 Implementation Notes (For Your Developer)
-
-### Key directories:
-
-```
-src/
-  cli.ts
-  commands/
-    new.ts
-    tc.ts
-    normalize.ts
-    gen.ts
-    test.ts
-  core/
-    prefix.ts
-    normalize.ts
-    llm.ts
-    scrape.ts
-    generator.ts
-  templates/
-```
-
-### Important details:
-
-* **`tc.ts`** manually checks `process.argv` for `--ai`
-  (because Commander is inconsistent with subcommands)
-* URL mode scrapes with `scrapePageSummary()`
-* LLM integration is fully isolated inside `core/llm.ts`
-* Test generation happens in `core/generator.ts`
-* Normalization lives in `core/normalize.ts`
-* Pages generator/templates live inside `templates/pages`
-
-### Code standards:
-
-* TypeScript everywhere
-* ESM modules
-* No external runtime dependencies except Playwright
-* Everything async
-* Clear separation between:
-
-  * Commands (CLI)
-  * Logic (core)
-  * Templates (scaffolding)
-  * Outputs (.cementic, tests, pages, cases)
+## Key source files
 
----
-
-# 📦 Release Process
-
-We use GitHub Actions for automated releases.
-
-### 1. Bump Version
-Update `package.json` version:
-```bash
-npm version patch # or minor/major
+```text
+src/cli.ts
+src/commands/new.ts
+src/commands/tc.ts
+src/commands/normalize.ts
+src/commands/gen.ts
+src/commands/test.ts
+src/commands/flow.ts
+src/core/capture.ts
+src/core/analyse.ts
+src/core/llm.ts
+src/core/report.ts
+src/core/prefix.ts
+templates/student-framework/
+templates/student-framework-ts/
 ```
 
-### 2. Push to Main
-```bash
-git push origin main
-```
-*CI will run tests and build.*
-
-### 3. Create Release Tag
-```bash
-git push --tags
-```
-*CI will detect the `v*` tag and automatically publish to NPM.*
-
----
-
-# 📅 Roadmap
-
-### Phase 1 (Current)
-
-✔️ CLI scaffolding
-✔️ AI test case generator
-✔️ URL scraping
-✔️ Normalization pipeline
-✔️ POM test generator
-✔️ Playwright runner wrapper
-
----
-
-### Phase 2 (Next)
-
-🔜 Self-healing locators
-🔜 Page synchronizer (`ct update pages`)
-🔜 API automation support (`ct api`)
-🔜 Test coverage reports
-
----
+## Changelog
 
-### Phase 3 (Future)
+### v0.2.5
 
-🔜 CT Desktop App (UI version of CLI)
-🔜 VS Code extension
-🔜 CI/CD pipeline builder
-🔜 Visual assertion generator
-🔜 CT Agent (continuous learning on test failures)
+- replaced the legacy URL scraping path with full Playwright-based live page capture
+- integrated the POC capture and AI analysis flow into `ct tc url --ai`
+- added capture artifacts at `.cementic/capture/capture-*.json`
+- added preview spec output at `tests/preview/spec-preview-*.spec.cjs`
+- preserved capture-generated selector hints and exact `playwright` assertions through `normalize` and `gen`
+- added `--headed` and `--capture-only` support for the URL capture flow
+- expanded capture-aware AI provider support to DeepSeek, Anthropic, Gemini, Qwen, Kimi, and OpenAI-compatible endpoints
 
----
+## Current limitations
 
-# 🤝 Contributing
+- generator assertions and step mapping are heuristic
+- there is no built-in self-healing test loop yet
+- there is no first-class Playwright CLI integration yet
+- POM generation intentionally avoids overwriting existing page objects, so existing pages may diverge from later normalized cases
 
-1. Fork the repo
-2. Create a feature branch
-3. Follow code conventions
-4. Build & test locally (`npm link`)
-5. Submit PR# cementic-test
-6. What's Next?
+## Contributing
 
-We created an org on NPMjs
+1. Create a branch
+2. Run `npm test`
+3. Verify a manual `tc -> normalize -> gen -> test` flow if your change affects generation
+4. Open a PR with the behavior change clearly described