retestkit 1.4.1 → 1.5.0

package/openspec/changes/archive/2025-12-18-add-hybrid-artifact-paths/specs/webtest-tools/spec.md

@@ -1,304 +0,0 @@
-## MODIFIED Requirements
-
-### Requirement: webtest_init Tool
-
-The system SHALL provide a `webtest_init` tool that initializes an analysis workspace for a target URL and focus.
-
-#### Scenario: Start analysis with valid URL
-
-- **GIVEN** the tool is called with a valid URL and focus
-- **WHEN** execution completes
-- **THEN** it SHALL generate a unique `analysisId`
-- **AND** create workspace directories
-- **AND** write initial `index.json` metadata
-- **AND** return `{ analysisId, workspaceRootPath, workspaceRootUri, statusUri }`
-
-#### Scenario: Start analysis validates URL
-
-- **GIVEN** the tool is called with an invalid URL
-- **WHEN** validation occurs
-- **THEN** it SHALL return an error with message "Invalid URL format"
-
-#### Scenario: Start analysis normalizes domain for allowlist
-
-- **GIVEN** the tool is called with URL "https://example.com/path"
-- **WHEN** workspace is created
-- **THEN** the default `allowedDomains` SHALL include "example.com"
-- **AND** this SHALL be stored in workspace metadata
-
-#### Scenario: Start analysis accepts custom limits
-
-- **GIVEN** the tool is called with `limits: { maxSteps: 50, maxPages: 10, maxMinutes: 5 }`
-- **WHEN** workspace is created
-- **THEN** limits SHALL be stored in workspace metadata
-- **AND** subsequent crawls SHALL respect these limits
-
-### Requirement: webtest_crawl_app Tool
-
-The system SHALL provide a `webtest_crawl_app` tool that dynamically explores a web application to achieve a goal.
-
-#### Scenario: Crawl navigates to starting URL
-
-- **GIVEN** the tool is called with a valid analysisId
-- **WHEN** crawl begins
-- **THEN** it SHALL launch Playwright MCP browser
-- **AND** navigate to the URL from analysis metadata
-
-#### Scenario: Crawl captures artifacts at each checkpoint
-
-- **GIVEN** a crawl iteration completes an action
-- **WHEN** state is captured
-- **THEN** it SHALL call Playwright MCP `browser_snapshot` for accessibility tree
-- **AND** call `browser_take_screenshot` for visual evidence
-- **AND** optionally extract HTML DOM
-- **AND** store artifacts in workspace with unique page IDs
-
-#### Scenario: Crawl uses sampling for next action
-
-- **GIVEN** crawl has captured current state
-- **WHEN** next action is needed
-- **THEN** it SHALL construct a sampling prompt with goal, history, current state
-- **AND** request next action via `sampling/createMessage`
-- **AND** validate and execute returned Playwright actions
-
-#### Scenario: Crawl terminates when goal satisfied
-
-- **GIVEN** crawl sampling returns `goalSatisfied: true`
-- **WHEN** this is detected
-- **THEN** crawl SHALL finalize with success status
-- **AND** write crawl summary to workspace
-
-#### Scenario: Crawl terminates when limits reached
-
-- **GIVEN** crawl has executed `maxSteps` actions
-- **WHEN** limit is checked
-- **THEN** crawl SHALL finalize with "limits_reached" status
-- **AND** preserve all collected artifacts
-
-#### Scenario: Crawl handles navigation loops
-
-- **GIVEN** crawl detects same page state 3 times consecutively
-- **WHEN** loop is detected
-- **THEN** it SHALL log a warning
-- **AND** request alternative action from sampling with loop context
-
-#### Scenario: Crawl triggers elicitation for cookie consent
-
-- **GIVEN** crawl detects a cookie consent dialog
-- **WHEN** elicitation is supported
-- **THEN** it SHALL call elicitation with options: "Accept", "Reject", "Dismiss"
-- **AND** execute the chosen action
-
-#### Scenario: Crawl triggers elicitation for blocking modal
-
-- **GIVEN** crawl detects a modal blocking navigation
-- **WHEN** elicitation is supported
-- **THEN** it SHALL call elicitation with options: "Close modal", "Interact with modal content"
-
-#### Scenario: Crawl triggers elicitation for ambiguous navigation
-
-- **GIVEN** sampling returns `needsElicitation: { type: "ambiguous", options: [...] }`
-- **WHEN** elicitation is supported
-- **THEN** it SHALL present the options to the user
-- **AND** use the selection to continue crawl
-
-#### Scenario: Crawl stops on authentication required
-
-- **GIVEN** crawl detects login form or auth wall
-- **WHEN** elicitation is supported
-- **THEN** it SHALL call elicitation with options: "Stop analysis", "Continue unauthenticated"
-- **AND** never request credentials
-
-#### Scenario: Crawl emits progress notifications
-
-- **GIVEN** crawl is running
-- **WHEN** each iteration completes
-- **THEN** it SHALL emit progress notification with step count, pages discovered, current intent
-
-#### Scenario: Crawl responds to cancellation
-
-- **GIVEN** crawl receives `notifications/cancelled`
-- **WHEN** cancellation is detected
-- **THEN** it SHALL stop the crawl loop promptly
-- **AND** finalize with "cancelled" status
-- **AND** preserve collected artifacts
-
-#### Scenario: Crawl returns fallback when sampling unavailable
-
-- **GIVEN** client does not support sampling
-- **WHEN** crawl needs next action
-- **THEN** it SHALL return `{ needsManualInput: true, promptUri, currentState }`
-- **AND** accept `manualNextActions` input to continue
-
-#### Scenario: Crawl outputs complete results
-
-- **GIVEN** crawl has finalized
-- **WHEN** output is returned
-- **THEN** it SHALL include `crawlId`, `crawlIndexFilePath`, `crawlIndexUri`, `pages[]`, `summaryUri`
-
-### Requirement: webtest_analyze_app Tool
-
-The system SHALL provide a `webtest_analyze_app` tool that reverse-engineers application structure from crawl data.
-
-#### Scenario: Analyze app loads crawl data
-
-- **GIVEN** the tool is called with valid analysisId and crawlId
-- **WHEN** execution begins
-- **THEN** it SHALL load crawl index and artifact references
-- **AND** load page snapshots for key pages
-
-#### Scenario: Analyze app uses sampling for analysis
-
-- **GIVEN** crawl data is loaded
-- **WHEN** analysis is performed
-- **THEN** it SHALL construct sampling prompt with crawl summary and snapshots
-- **AND** request structured analysis via `sampling/createMessage`
-
-#### Scenario: Analyze app extracts application purpose
-
-- **GIVEN** analysis sampling completes
-- **WHEN** results are processed
-- **THEN** output SHALL include identified app purpose
-- **AND** key entities (users, products, orders, etc.)
-
-#### Scenario: Analyze app identifies user flows
-
-- **GIVEN** analysis sampling completes
-- **WHEN** results are processed
-- **THEN** output SHALL include discovered user flows
-- **AND** each flow SHALL have id, name, description, steps
-
-#### Scenario: Analyze app suggests assertions
-
-- **GIVEN** analysis sampling completes
-- **WHEN** results are processed
-- **THEN** output SHALL include suggested assertions for testing
-- **AND** potential risks or edge cases
-
-#### Scenario: Analyze app writes markdown report
-
-- **GIVEN** analysis is complete
-- **WHEN** output is generated
-- **THEN** it SHALL write `app-analysis.md` resource to workspace
-
-#### Scenario: Analyze app outputs file paths and URIs
-
-- **GIVEN** analysis is complete
-- **WHEN** tool returns
-- **THEN** it SHALL include `appAnalysisFilePath`, `appAnalysisUri`, `flowsFilePath`, and `flowsUri`
-
-### Requirement: webtest_generate_tests Tool
-
-The system SHALL provide a `webtest_generate_tests` tool that produces test cases from application analysis.
-
-#### Scenario: Generate tests loads analysis
-
-- **GIVEN** the tool is called with valid analysisId and appAnalysisUri
-- **WHEN** execution begins
-- **THEN** it SHALL load app analysis and flows from workspace
-
-#### Scenario: Generate tests uses sampling
-
-- **GIVEN** analysis is loaded
-- **WHEN** test generation is performed
-- **THEN** it SHALL construct sampling prompt with analysis, flows, strategy
-- **AND** request test cases via `sampling/createMessage`
-
-#### Scenario: Generate tests applies strategy
-
-- **GIVEN** tool is called with `testStrategy: { count: 5, types: ["smoke", "negative"] }`
-- **WHEN** sampling prompt is built
-- **THEN** it SHALL instruct model to generate 5 tests covering smoke and negative scenarios
-
-#### Scenario: Generate tests outputs structured format
-
-- **GIVEN** test generation completes
-- **WHEN** results are written
-- **THEN** it SHALL produce `tests.md` with human-readable format
-- **AND** `tests.json` with structured test definitions
-
-#### Scenario: Test case structure is complete
-
-- **GIVEN** tests.json is generated
-- **WHEN** a test case is examined
-- **THEN** it SHALL include: id, name, purpose, preconditions, steps[], expected results, priority
-
-#### Scenario: Generate tests outputs file paths and URIs
-
-- **GIVEN** generation is complete
-- **WHEN** tool returns
-- **THEN** it SHALL include `testsFilePath`, `testsUri`, and `testIndexUri`
-
-### Requirement: webtest_run_tests Tool
-
-The system SHALL provide a `webtest_run_tests` tool that executes a test case with evidence capture.
-
-#### Scenario: Run test case loads test definition
-
-- **GIVEN** the tool is called with valid analysisId and testCaseId
-- **WHEN** execution begins
-- **THEN** it SHALL load test case from tests index
-- **AND** validate test case exists
-
-#### Scenario: Run test case executes steps sequentially
-
-- **GIVEN** test case has multiple steps
-- **WHEN** execution runs
-- **THEN** it SHALL execute each step in order
-- **AND** capture state before and after each step
-
-#### Scenario: Run test case uses sampling for step translation
-
-- **GIVEN** a test step needs execution
-- **WHEN** translation is needed
-- **THEN** it SHALL use sampling to convert step description to Playwright actions
-- **AND** validate actions before execution
-
-#### Scenario: Run test case captures evidence
-
-- **GIVEN** a step is executed
-- **WHEN** evidence is captured
-- **THEN** it SHALL take screenshot after action
-- **AND** capture accessibility snapshot
-- **AND** store with step identifier
-
-#### Scenario: Run test case evaluates pass/fail
-
-- **GIVEN** a step has executed
-- **WHEN** evaluation occurs
-- **THEN** it SHALL use sampling to compare expected vs actual
-- **AND** record pass or fail with reason
-
-#### Scenario: Run test case continues on step failure
-
-- **GIVEN** a step fails
-- **WHEN** failure is recorded
-- **THEN** execution SHALL continue to next step (unless critical)
-- **AND** overall test status SHALL be "failed"
-
-#### Scenario: Run test case emits progress
-
-- **GIVEN** test is running
-- **WHEN** each step completes
-- **THEN** it SHALL emit progress notification with step number, status
-
-#### Scenario: Run test case responds to cancellation
-
-- **GIVEN** test receives `notifications/cancelled`
-- **WHEN** cancellation is detected
-- **THEN** it SHALL stop after current step
-- **AND** finalize with "cancelled" status and partial results
-
-#### Scenario: Run test case outputs report
-
-- **GIVEN** test execution completes
-- **WHEN** output is generated
-- **THEN** it SHALL write `report.md` with pass/fail summary, step details, evidence links
-- **AND** `artifacts.json` with structured run data
-
-#### Scenario: Run test case returns file paths and URIs
-
-- **GIVEN** execution is complete
-- **WHEN** tool returns
-- **THEN** it SHALL include `testRunId`, `reportFilePath`, `reportUri`, `runArtifactsIndexUri`
-- **AND** each step result SHALL include `evidenceFilePath` and `evidenceUri` for captured artifacts

package/openspec/changes/archive/2025-12-18-add-hybrid-artifact-paths/tasks.md

@@ -1,43 +0,0 @@
-## 1. Workspace Manager Updates
-
-- [x] 1.1 Add `workspaceDir` getter to expose base directory path
-- [x] 1.2 Update `saveAnalysis` to return `{ appAnalysisFilePath, appAnalysisUri, flowsFilePath, flowsUri }`
-- [x] 1.3 Update `saveTests` to return `{ testsFilePath, testsUri, testsMarkdownFilePath }`
-- [x] 1.4 Update `savePage` to return `{ snapshotFilePath, screenshotFilePath, domFilePath }` alongside URIs
-- [x] 1.5 Update `saveTestStepEvidence` to return file paths alongside URIs
-- [x] 1.6 Update `createWorkspace` to return `{ workspacePath, workspaceUri }`
-- [x] 1.7 Update `createCrawl` to return `{ crawlPath, crawlIndexUri }`
-
-## 2. Type Definitions
-
-- [x] 2.1 Update workspace types to include FilePath variants in return types
-- [x] 2.2 Add TypeScript interfaces for hybrid artifact results
-
-## 3. Tool Updates
-
-- [x] 3.1 Update `webtest_init` to include `workspaceRootPath` in result
-- [x] 3.2 Update `webtest_crawl_app` to include `crawlIndexFilePath` in result
-- [x] 3.3 Update `webtest_analyze_app` to include `appAnalysisFilePath` and `flowsFilePath` in result
-- [x] 3.4 Update `webtest_generate_tests` to include `testsFilePath` in result
-- [x] 3.5 Update `webtest_run_tests` to include `reportFilePath` and evidence file paths in result
-
-## 4. Tests
-
-- [x] 4.1 Update workspace manager tests to verify file path return values
-- [x] 4.2 Update tool tests to verify file path fields in results
-- [x] 4.3 Verify file paths are absolute and valid
-
-## 5. Workspace Index Persistence (Fix for handover issue)
-
-- [x] 5.1 Add `appAnalysisFilePath` and `flowsFilePath` to `AnalysisReference` type
-- [x] 5.2 Add `testsFilePath` and `testsMarkdownFilePath` to `TestsReference` type
-- [x] 5.3 Update `saveAnalysis` to persist file paths in workspace index
-- [x] 5.4 Update `saveTests` to persist file paths in workspace index
-
-## 6. Manual Input Support (Fix for fallback mode when sampling unavailable)
-
-- [x] 6.1 Add `manualAnalysis` parameter to `webtest_analyze_app` input schema
-- [x] 6.2 Update `webtest_analyze_app` handler to use manual input when provided
-- [x] 6.3 Update fallback instructions to guide calling tool again with manual input
-- [x] 6.4 Add `manualTests` parameter to `webtest_generate_tests` input schema
-- [x] 6.5 Update `webtest_generate_tests` handler to use manual input when provided

package/openspec/changes/archive/2025-12-18-add-mcp-server-foundation/design.md

@@ -1,209 +0,0 @@
-## Context
-
-This MCP Web Testing Server needs a solid foundation that follows MCP protocol best practices while being structured for extensibility. The server will eventually provide tools for automated web application testing, so the initial architecture must support easy addition of testing-specific tools (browser automation, HTTP requests, assertions, etc.).
-
-**Stakeholders**: Developers building testing workflows, MCP client integrations (Claude Code, ChatGPT desktop, etc.)
-
-**Constraints**:
-- Must follow MCP specification (2025-06-18)
-- Must support both stdio and HTTP transports from day one
-- Proprietary codebase - no external contributions expected
-
-## Goals / Non-Goals
-
-**Goals:**
-- Establish clean, extensible project structure for adding testing tools
-- Follow official MCP TypeScript SDK patterns
-- Provide pluggable transport layer (stdio + Streamable HTTP)
-- Create self-describing tool registry pattern
-- Enable rapid development iteration with hot-reloading
-- Include comprehensive test infrastructure (unit + integration)
-- Production-ready logging and configuration
-
-**Non-Goals:**
-- Authentication/authorization (deferred to future proposal)
-- Resource or prompt handlers (tools-focused initially)
-- Actual testing functionality (that comes in subsequent proposals)
-
-## Decisions
-
-### Decision 1: TypeScript with Official MCP SDK
-
-**What**: Use TypeScript with `@modelcontextprotocol/sdk` and Zod for schemas.
-
-**Why**:
-- Official SDK maintained by protocol authors
-- TypeScript provides type safety and IDE support
-- Zod is required peer dependency and excellent for runtime validation
-- Largest ecosystem of MCP reference implementations
-
-**Alternatives considered**:
-- Python SDK: Good option but TypeScript has better MCP ecosystem maturity
-- Go SDK: Less mature, fewer reference implementations to learn from
-
-### Decision 2: ESM-Only with Node.js >= 22.18.0
-
-**What**: Use ECMAScript modules exclusively, recommend Node.js 22.18.0+.
-
-**Why**:
-- Node.js 22.18.0+ has built-in type stripping enabled by default (instant reloads without build step)
-- Earlier 22.x versions require `--experimental-strip-types` flag
-- ESM is the modern standard and SDK uses ESM imports
-- Simplifies tooling configuration
-
-**Alternatives considered**:
-- CommonJS: Legacy, more tooling complexity
-- Dual ESM/CJS: Unnecessary complexity for internal project
-
-### Decision 3: Pluggable Transport Architecture
-
-**What**: Abstract transport selection via environment variables with implementations in `src/transports/`.
-
-```
-TRANSPORT=stdio         # Default for local dev
-TRANSPORT=http PORT=3000  # For remote deployment
-```
-
-**Why**:
-- Aligns with how the TS SDK frames transports
-- stdio for local MCP client integration (Claude Code, etc.)
-- Streamable HTTP ready for remote/cloud deployment scenarios
-- Easy to add new transports (WebSocket, etc.) following the pattern
-
-**Structure**:
-```
-src/transports/
-├── index.ts      # Transport factory based on env config
-├── stdio.ts      # StdioServerTransport wrapper
-└── http.ts       # StreamableHTTPServerTransport wrapper
-```
-
-### Decision 4: Project Structure
-
-**What**: Organize code into modular directories by concern.
-
-```
-src/
-├── index.ts           # Entry point, bootstrap
-├── server.ts          # MCP server factory
-├── config.ts          # Zod-validated configuration
-├── logger.ts          # Structured logging with redaction
-├── transports/        # Transport implementations
-│   ├── index.ts       # Factory
-│   ├── stdio.ts
-│   └── http.ts
-├── tools/             # Tool definitions
-│   ├── index.ts       # Registry
-│   └── hello.ts       # Demo tool
-├── schemas/           # Shared Zod schemas
-└── types/             # TypeScript type definitions
-```
-
-**Why**:
-- Follows patterns from official MCP reference servers
-- Tools directory with registry enables easy addition of new tools
-- Transports directory encapsulates transport-specific logic
-- Separation of concerns for maintainability
-
-### Decision 5: Self-Describing Tool Registry
-
-**What**: Each tool exports a standard interface; registry auto-discovers and registers tools.
-
-```typescript
-// Tool interface
-export interface McpTool<TInput> {
-  name: string;
-  description: string;
-  inputSchema: z.ZodType<TInput>;
-  handler: (input: TInput) => Promise<ToolResult>;
-}
-
-// tools/index.ts exports all tools
-export const tools: McpTool<unknown>[] = [helloTool, /* future tools */];
-```
-
-**Why**:
-- Single source of truth: Zod schema generates both TS types and JSON Schema for MCP
-- Consistent pattern for adding new tools
-- Easy to test tools in isolation
-- Self-documenting tool capabilities
-
-### Decision 6: Structured Logging
-
-**What**: Use structured JSON logging with automatic secret redaction.
-
-**Why**:
-- Machine-parseable logs for production observability
-- Secret redaction prevents accidental credential exposure
-- Configurable log levels via environment
-
-**Pattern**:
-```typescript
-// Redact known sensitive fields
-const REDACT_KEYS = ['password', 'token', 'secret', 'apiKey', 'authorization'];
-```
-
-### Decision 7: Vitest for Testing
-
-**What**: Use Vitest for unit and integration tests.
-
-**Why**:
-- Native ESM support
-- Fast execution with built-in watch mode
-- TypeScript support out of box
-- Compatible with Node.js test patterns
-
-**Test Strategy**:
-- **Unit tests**: Test tool handlers in isolation with mocked inputs
-- **Integration tests**: Spawn actual server, connect via StdioServerTransport, execute tools end-to-end
-
-**Alternatives considered**:
-- Jest: Requires more ESM configuration
-- Node.js built-in test runner: Less mature feature set
-
-### Decision 8: tsx for Development
-
-**What**: Use `tsx` for development with watch mode.
-
-**Why**:
-- Instant TypeScript execution without separate compile step
-- Excellent watch mode for hot-reloading during development
-- Works seamlessly with ESM
-
-### Decision 9: Package.json Hygiene
-
-**What**: Proper ESM package configuration with exports map and bin entry.
-
-```json
-{
-  "type": "module",
-  "exports": {
-    ".": "./dist/index.js"
-  },
-  "bin": {
-    "testing-mcp": "./dist/index.js"
-  }
-}
-```
-
-**Why**:
-- Enables `npx testing-mcp` execution
-- Proper exports for potential future library consumption
-- Clear module resolution
-
-## Risks / Trade-offs
-
-| Risk | Impact | Mitigation |
-|------|--------|------------|
-| SDK breaking changes | Medium | Pin SDK version, update intentionally |
-| Node.js 22.18.0+ requirement | Low | Recent LTS, document flag for earlier 22.x |
-| HTTP transport without auth | Medium | Document as internal-only; auth in future proposal |
-| Structured logging overhead | Low | Negligible for typical tool execution volumes |
-
-## Migration Plan
-
-Not applicable - this is a greenfield implementation.
-
-## Open Questions
-
-None - ready for implementation approval.

package/openspec/changes/archive/2025-12-18-add-mcp-server-foundation/proposal.md

@@ -1,41 +0,0 @@
-# Change: Add MCP Server Foundation
-
-## Why
-
-This project needs a foundational MCP server implementation to support web application testing workflows. Starting with a well-structured foundation enables incremental addition of testing-specific tools while maintaining clean architecture and consistent patterns from the start.
-
-## What Changes
-
-### Core Infrastructure
-- Initialize TypeScript project with modern tooling (Node.js >= 22.18.0 recommended, TypeScript 5.x, ESM modules)
-- Implement MCP server using the official `@modelcontextprotocol/sdk` with Zod schema validation
-- Create project structure optimized for web testing tool development
-
-### Pluggable Transport Layer
-- Abstract transport selection behind environment config (`TRANSPORT=stdio|http`, `PORT=...`)
-- Implement stdio transport for local dev/integration
-- Implement Streamable HTTP transport ready for remote deployment
-- Transport modules in `src/transports/` following SDK patterns
-
-### Self-Describing Tool Registry
-- Create tool registry pattern in `src/tools/index.ts`
-- Each tool exports `{ name, description, inputSchema, handler }`
-- Generate MCP tool input JSON schema from Zod (single source of truth)
-- Add "hello" demonstration tool following the registry pattern
-
-### Production-Ready Basics
-- Structured logging with secret redaction
-- Graceful shutdown handling (SIGINT/SIGTERM)
-- Config validation via Zod at startup
-- Package.json hygiene: `"type":"module"`, exports map, bin entry for CLI usage
-
-### Testing Infrastructure
-- Unit tests with Vitest
-- Integration tests that spawn server and speak MCP protocol end-to-end
-- Test tool execution and error handling via StdioServerTransport
-
-## Impact
-
-- Affected specs: `mcp-server-core` (new capability)
-- Affected code: New project structure in `src/`, configuration files at root
-- Dependencies: `@modelcontextprotocol/sdk`, `zod`, TypeScript toolchain