retestkit 1.4.1 → 1.5.0

package/openspec/changes/archive/2025-12-18-add-webtest-orchestrator/specs/webtest-tools/spec.md

@@ -1,501 +0,0 @@
-# webtest-tools Specification
-
-## Purpose
-
-Defines the five webtest tools that compose the dynamic web testing workflow.
-
-## ADDED Requirements
-
-### Requirement: webtest_init Tool
-
-The system SHALL provide a `webtest_init` tool that initializes an analysis workspace for a target URL and focus.
-
-#### Scenario: Start analysis with valid URL
-
-- **GIVEN** the tool is called with a valid URL and focus
-- **WHEN** execution completes
-- **THEN** it SHALL generate a unique `analysisId`
-- **AND** create workspace directories
-- **AND** write initial `index.json` metadata
-- **AND** return `{ analysisId, workspaceRootUri, statusUri }`
-
-#### Scenario: Start analysis validates URL
-
-- **GIVEN** the tool is called with an invalid URL
-- **WHEN** validation occurs
-- **THEN** it SHALL return an error with message "Invalid URL format"
-
-#### Scenario: Start analysis normalizes domain for allowlist
-
-- **GIVEN** the tool is called with URL "https://example.com/path"
-- **WHEN** workspace is created
-- **THEN** the default `allowedDomains` SHALL include "example.com"
-- **AND** this SHALL be stored in workspace metadata
-
-#### Scenario: Start analysis accepts custom limits
-
-- **GIVEN** the tool is called with `limits: { maxSteps: 50, maxPages: 10, maxMinutes: 5 }`
-- **WHEN** workspace is created
-- **THEN** limits SHALL be stored in workspace metadata
-- **AND** subsequent crawls SHALL respect these limits
-
-### Requirement: webtest_crawl_app Tool
-
-The system SHALL provide a `webtest_crawl_app` tool that dynamically explores a web application to achieve a goal.
-
-#### Scenario: Crawl navigates to starting URL
-
-- **GIVEN** the tool is called with a valid analysisId
-- **WHEN** crawl begins
-- **THEN** it SHALL launch Playwright MCP browser
-- **AND** navigate to the URL from analysis metadata
-
-#### Scenario: Crawl captures artifacts at each checkpoint
-
-- **GIVEN** a crawl iteration completes an action
-- **WHEN** state is captured
-- **THEN** it SHALL call Playwright MCP `browser_snapshot` for accessibility tree
-- **AND** call `browser_take_screenshot` for visual evidence
-- **AND** optionally extract HTML DOM
-- **AND** store artifacts in workspace with unique page IDs
-
-#### Scenario: Crawl uses sampling for next action
-
-- **GIVEN** crawl has captured current state
-- **WHEN** next action is needed
-- **THEN** it SHALL construct a sampling prompt with goal, history, current state
-- **AND** request next action via `sampling/createMessage`
-- **AND** validate and execute returned Playwright actions
-
-#### Scenario: Crawl terminates when goal satisfied
-
-- **GIVEN** crawl sampling returns `goalSatisfied: true`
-- **WHEN** this is detected
-- **THEN** crawl SHALL finalize with success status
-- **AND** write crawl summary to workspace
-
-#### Scenario: Crawl terminates when limits reached
-
-- **GIVEN** crawl has executed `maxSteps` actions
-- **WHEN** limit is checked
-- **THEN** crawl SHALL finalize with "limits_reached" status
-- **AND** preserve all collected artifacts
-
-#### Scenario: Crawl handles navigation loops
-
-- **GIVEN** crawl detects same page state 3 times consecutively
-- **WHEN** loop is detected
-- **THEN** it SHALL log a warning
-- **AND** request alternative action from sampling with loop context
-
-#### Scenario: Crawl triggers elicitation for cookie consent
-
-- **GIVEN** crawl detects a cookie consent dialog
-- **WHEN** elicitation is supported
-- **THEN** it SHALL call elicitation with options: "Accept", "Reject", "Dismiss"
-- **AND** execute the chosen action
-
-#### Scenario: Crawl triggers elicitation for blocking modal
-
-- **GIVEN** crawl detects a modal blocking navigation
-- **WHEN** elicitation is supported
-- **THEN** it SHALL call elicitation with options: "Close modal", "Interact with modal content"
-
-#### Scenario: Crawl triggers elicitation for ambiguous navigation
-
-- **GIVEN** sampling returns `needsElicitation: { type: "ambiguous", options: [...] }`
-- **WHEN** elicitation is supported
-- **THEN** it SHALL present the options to the user
-- **AND** use the selection to continue crawl
-
-#### Scenario: Crawl stops on authentication required
-
-- **GIVEN** crawl detects login form or auth wall
-- **WHEN** elicitation is supported
-- **THEN** it SHALL call elicitation with options: "Stop analysis", "Continue unauthenticated"
-- **AND** never request credentials
-
-#### Scenario: Crawl emits progress notifications
-
-- **GIVEN** crawl is running
-- **WHEN** each iteration completes
-- **THEN** it SHALL emit progress notification with step count, pages discovered, current intent
-
-#### Scenario: Crawl responds to cancellation
-
-- **GIVEN** crawl receives `notifications/cancelled`
-- **WHEN** cancellation is detected
-- **THEN** it SHALL stop the crawl loop promptly
-- **AND** finalize with "cancelled" status
-- **AND** preserve collected artifacts
-
-#### Scenario: Crawl returns fallback when sampling unavailable
-
-- **GIVEN** client does not support sampling
-- **WHEN** crawl needs next action
-- **THEN** it SHALL return `{ needsManualInput: true, promptUri, currentState }`
-- **AND** accept `manualNextActions` input to continue
-
-#### Scenario: Crawl outputs complete results
-
-- **GIVEN** crawl has finalized
-- **WHEN** output is returned
-- **THEN** it SHALL include `crawlId`, `crawlIndexUri`, `pages[]`, `summaryUri`
-
-### Requirement: webtest_analyze_app Tool
-
-The system SHALL provide a `webtest_analyze_app` tool that reverse-engineers application structure from crawl data.
-
-#### Scenario: Analyze app loads crawl data
-
-- **GIVEN** the tool is called with valid analysisId and crawlId
-- **WHEN** execution begins
-- **THEN** it SHALL load crawl index and artifact references
-- **AND** load page snapshots for key pages
-
-#### Scenario: Analyze app uses sampling for analysis
-
-- **GIVEN** crawl data is loaded
-- **WHEN** analysis is performed
-- **THEN** it SHALL construct sampling prompt with crawl summary and snapshots
-- **AND** request structured analysis via `sampling/createMessage`
-
-#### Scenario: Analyze app extracts application purpose
-
-- **GIVEN** analysis sampling completes
-- **WHEN** results are processed
-- **THEN** output SHALL include identified app purpose
-- **AND** key entities (users, products, orders, etc.)
-
-#### Scenario: Analyze app identifies user flows
-
-- **GIVEN** analysis sampling completes
-- **WHEN** results are processed
-- **THEN** output SHALL include discovered user flows
-- **AND** each flow SHALL have id, name, description, steps
-
-#### Scenario: Analyze app suggests assertions
-
-- **GIVEN** analysis sampling completes
-- **WHEN** results are processed
-- **THEN** output SHALL include suggested assertions for testing
-- **AND** potential risks or edge cases
-
-#### Scenario: Analyze app writes markdown report
-
-- **GIVEN** analysis is complete
-- **WHEN** output is generated
-- **THEN** it SHALL write `app-analysis.md` resource to workspace
-
-#### Scenario: Analyze app outputs URIs
-
-- **GIVEN** analysis is complete
-- **WHEN** tool returns
-- **THEN** it SHALL include `appAnalysisUri` and `flowsIndexUri`
-
-### Requirement: webtest_generate_tests Tool
-
-The system SHALL provide a `webtest_generate_tests` tool that produces test cases from application analysis.
-
-#### Scenario: Generate tests loads analysis
-
-- **GIVEN** the tool is called with valid analysisId and appAnalysisUri
-- **WHEN** execution begins
-- **THEN** it SHALL load app analysis and flows from workspace
-
-#### Scenario: Generate tests uses sampling
-
-- **GIVEN** analysis is loaded
-- **WHEN** test generation is performed
-- **THEN** it SHALL construct sampling prompt with analysis, flows, strategy
-- **AND** request test cases via `sampling/createMessage`
-
-#### Scenario: Generate tests applies strategy
-
-- **GIVEN** tool is called with `testStrategy: { count: 5, types: ["smoke", "negative"] }`
-- **WHEN** sampling prompt is built
-- **THEN** it SHALL instruct model to generate 5 tests covering smoke and negative scenarios
-
-#### Scenario: Generate tests outputs structured format
-
-- **GIVEN** test generation completes
-- **WHEN** results are written
-- **THEN** it SHALL produce `tests.md` with human-readable format
-- **AND** `tests.json` with structured test definitions
-
-#### Scenario: Test case structure is complete
-
-- **GIVEN** tests.json is generated
-- **WHEN** a test case is examined
-- **THEN** it SHALL include: id, name, purpose, preconditions, steps[], expected results, priority
-
-#### Scenario: Generate tests outputs URIs
-
-- **GIVEN** generation is complete
-- **WHEN** tool returns
-- **THEN** it SHALL include `testsUri` and `testIndexUri`
-
-### Requirement: webtest_run_tests Tool
-
-The system SHALL provide a `webtest_run_tests` tool that executes a test case with evidence capture.
-
-#### Scenario: Run test case loads test definition
-
-- **GIVEN** the tool is called with valid analysisId and testCaseId
-- **WHEN** execution begins
-- **THEN** it SHALL load test case from tests index
-- **AND** validate test case exists
-
-#### Scenario: Run test case executes steps sequentially
-
-- **GIVEN** test case has multiple steps
-- **WHEN** execution runs
-- **THEN** it SHALL execute each step in order
-- **AND** capture state before and after each step
-
-#### Scenario: Run test case uses sampling for step translation
-
-- **GIVEN** a test step needs execution
-- **WHEN** translation is needed
-- **THEN** it SHALL use sampling to convert step description to Playwright actions
-- **AND** validate actions before execution
-
-#### Scenario: Run test case captures evidence
-
-- **GIVEN** a step is executed
-- **WHEN** evidence is captured
-- **THEN** it SHALL take screenshot after action
-- **AND** capture accessibility snapshot
-- **AND** store with step identifier
-
-#### Scenario: Run test case evaluates pass/fail
-
-- **GIVEN** a step has executed
-- **WHEN** evaluation occurs
-- **THEN** it SHALL use sampling to compare expected vs actual
-- **AND** record pass or fail with reason
-
-#### Scenario: Run test case continues on step failure
-
-- **GIVEN** a step fails
-- **WHEN** failure is recorded
-- **THEN** execution SHALL continue to next step (unless critical)
-- **AND** overall test status SHALL be "failed"
-
-#### Scenario: Run test case emits progress
-
-- **GIVEN** test is running
-- **WHEN** each step completes
-- **THEN** it SHALL emit progress notification with step number, status
-
-#### Scenario: Run test case responds to cancellation
-
-- **GIVEN** test receives `notifications/cancelled`
-- **WHEN** cancellation is detected
-- **THEN** it SHALL stop after current step
-- **AND** finalize with "cancelled" status and partial results
-
-#### Scenario: Run test case outputs report
-
-- **GIVEN** test execution completes
-- **WHEN** output is generated
-- **THEN** it SHALL write `report.md` with pass/fail summary, step details, evidence links
-- **AND** `artifacts.json` with structured run data
-
-#### Scenario: Run test case returns URIs
-
-- **GIVEN** execution is complete
-- **WHEN** tool returns
-- **THEN** it SHALL include `testRunId`, `reportUri`, `runArtifactsIndexUri`
-
-### Requirement: Playwright MCP Integration
-
-The system SHALL orchestrate an external Playwright MCP server for browser automation with dynamic tool discovery.
-
-#### Scenario: Playwright MCP is spawned on first use
-
-- **GIVEN** a webtest tool needs browser access
-- **WHEN** Playwright client is accessed
-- **THEN** it SHALL spawn Playwright MCP server as subprocess if not running
-- **AND** connect via stdio transport
-
-#### Scenario: Playwright MCP tools are discovered dynamically
-
-- **GIVEN** Playwright MCP is connected
-- **WHEN** connection is established
-- **THEN** it SHALL call `tools/list` to discover available tools
-- **AND** build a capability adapter mapping canonical operations to actual tool names
-- **AND** cache the mapping for the session lifetime
-
-#### Scenario: Capability adapter maps canonical operations
-
-- **GIVEN** Playwright MCP tools have been discovered
-- **WHEN** the adapter is queried for operation "snapshot"
-- **THEN** it SHALL return the matching tool (e.g., `browser_snapshot` or `playwright_snapshot`)
-- **AND** if multiple matches exist, prefer the most specific
-
-#### Scenario: Missing required capability logs warning
-
-- **GIVEN** Playwright MCP tools have been discovered
-- **WHEN** a required capability (snapshot, screenshot, click, type, navigate) is missing
-- **THEN** it SHALL log a warning with the missing capability
-- **AND** tools requiring that capability SHALL return an error when invoked
-
-#### Scenario: Playwright actions are executed via adapter
-
-- **GIVEN** a crawl action specifies `{ tool: "click", args: { selector: "button" } }`
-- **WHEN** action is executed
-- **THEN** it SHALL resolve "click" through the capability adapter
-- **AND** call the resolved Playwright MCP tool with appropriate arguments
-- **AND** return the result
-
-#### Scenario: Playwright MCP version differences are handled
-
-- **GIVEN** different Playwright MCP implementations may have different tool names
-- **WHEN** the adapter maps tools
-- **THEN** it SHALL check for common variants:
-  - `browser_*` prefix (Microsoft implementation)
-  - `playwright_*` prefix (alternative implementations)
-  - unprefixed names
-- **AND** log the detected implementation variant
-
-#### Scenario: Playwright MCP is terminated on shutdown
-
-- **GIVEN** the server receives shutdown signal
-- **WHEN** shutdown begins
-- **THEN** it SHALL terminate Playwright MCP subprocess
-- **AND** wait for clean exit
-
-### Requirement: Crawl Checkpointing
-
-The system SHALL implement checkpointing during crawl to enable resumption and provide partial results on failure.
-
-#### Scenario: Checkpoint is written periodically
-
-- **GIVEN** a crawl is in progress
-- **WHEN** N steps have completed (configurable, default 5)
-- **THEN** it SHALL write a checkpoint to `webtest://{analysisId}/crawls/{crawlId}/checkpoint.json`
-- **AND** the checkpoint SHALL include: current step, visited pages, action history, goal progress
-
-#### Scenario: Checkpoint is written on each page capture
-
-- **GIVEN** a crawl captures a new page
-- **WHEN** page artifacts are saved
-- **THEN** it SHALL update the crawl index immediately
-- **AND** emit `notifications/resources/list_changed` if supported
-
-#### Scenario: Crawl can resume from checkpoint
-
-- **GIVEN** a crawl was interrupted (cancelled, error, timeout)
-- **AND** a checkpoint exists
-- **WHEN** `webtest_crawl_app` is called with `resume: true`
-- **THEN** it SHALL load the checkpoint
-- **AND** continue from the last recorded state
-
-#### Scenario: Checkpoint includes DOM signature for loop detection
-
-- **GIVEN** a checkpoint is written
-- **WHEN** the current page state is recorded
-- **THEN** it SHALL include a DOM signature (hash of key structural elements)
-- **AND** this signature SHALL be used for loop detection
-
-### Requirement: Crawl Loop Detection and Prevention
-
-The system SHALL detect and prevent infinite crawl loops.
-
-#### Scenario: Same page state detected consecutively
-
-- **GIVEN** crawl detects same DOM signature 3 times consecutively
-- **WHEN** loop is detected
-- **THEN** it SHALL log a warning with loop details
-- **AND** inform sampling of the loop condition
-- **AND** request alternative action with loop context
-
-#### Scenario: URL cycle detected
-
-- **GIVEN** crawl visits the same URL more than 3 times
-- **WHEN** cycle is detected
-- **THEN** it SHALL log a warning
-- **AND** exclude that URL from future navigation suggestions
-
-#### Scenario: Action repeat detected
-
-- **GIVEN** the same action (tool + args) is attempted 3 times consecutively
-- **WHEN** repeat is detected
-- **THEN** it SHALL reject the repeated action
-- **AND** request a different action from sampling with repeat context
-
-#### Scenario: Loop detection state is included in sampling prompts
-
-- **GIVEN** loop detection has flagged potential issues
-- **WHEN** the next sampling prompt is built
-- **THEN** it SHALL include:
-  - URLs visited more than once
-  - Recently repeated actions
-  - DOM signature history
-- **AND** instruct the model to avoid these patterns
-
-### Requirement: Crawl Budget Enforcement
-
-The system SHALL enforce time, step, and page limits during crawl.
-
-#### Scenario: Step limit is enforced
-
-- **GIVEN** crawl has a `maxSteps` limit
-- **WHEN** the step count reaches the limit
-- **THEN** crawl SHALL finalize with status "limits_reached"
-- **AND** include all artifacts collected up to that point
-
-#### Scenario: Time limit is enforced
-
-- **GIVEN** crawl has a `maxMinutes` limit
-- **WHEN** elapsed time reaches the limit
-- **THEN** crawl SHALL finalize with status "timeout"
-- **AND** complete current action before stopping
-- **AND** preserve all collected artifacts
-
-#### Scenario: Page limit is enforced
-
-- **GIVEN** crawl has a `maxPages` limit
-- **WHEN** the unique page count reaches the limit
-- **THEN** crawl SHALL stop discovering new pages
-- **AND** continue actions on already-visited pages until goal or step limit
-
-#### Scenario: Budget status is reported in progress
-
-- **GIVEN** crawl is running with limits
-- **WHEN** progress notification is emitted
-- **THEN** it SHALL include budget status:
-  - `stepsUsed` / `maxSteps`
-  - `minutesElapsed` / `maxMinutes`
-  - `pagesDiscovered` / `maxPages`
-
-### Requirement: Security Domain Enforcement
-
-The system SHALL enforce domain allowlists for all navigation actions.
-
-#### Scenario: Navigation to allowed domain succeeds
-
-- **GIVEN** allowedDomains includes "example.com"
-- **WHEN** Playwright action navigates to "https://example.com/page"
-- **THEN** navigation SHALL be allowed
-
-#### Scenario: Navigation to disallowed domain is blocked
-
-- **GIVEN** allowedDomains includes only "example.com"
-- **WHEN** sampling returns action to navigate to "https://malicious.com"
-- **THEN** the action SHALL be rejected
-- **AND** error logged with attempted URL
-
-#### Scenario: Subdomain matching follows rules
-
-- **GIVEN** allowedDomains includes "example.com"
-- **WHEN** navigation to "sub.example.com" is attempted
-- **THEN** it SHALL be allowed (subdomain of allowed domain)
-
-#### Scenario: Link clicks are validated
-
-- **GIVEN** a click action may navigate to external domain
-- **WHEN** click is executed
-- **THEN** resulting URL SHALL be checked post-navigation
-- **AND** if disallowed, navigate back and report error