retestkit 1.4.1

package/openspec/changes/archive/2025-12-18-add-webtest-orchestrator/proposal.md

@@ -0,0 +1,66 @@
+# Change: Add Dynamic Web Testing Orchestrator
+
+## Why
+
+The current MCP server provides only a basic "hello" demonstration tool. To fulfill its purpose as a web testing server, it needs to orchestrate dynamic web application exploration, analysis, test generation, and test execution. By leveraging the full MCP protocol (sampling, elicitation, resources, prompts, progress, cancellation) and integrating with Playwright MCP for browser automation, the server can provide AI-powered exploratory testing capabilities where the LLM reasoning happens client-side via MCP Sampling.
+
+## What Changes
+
+### New Capabilities
+
+- **MCP Lifecycle & Capability Negotiation**: Proper initialize/operate/shutdown lifecycle with runtime capability detection (sampling, elicitation, logging, progress, resources.listChanged, resources.subscribe)
+- **Protocol Version Requirements**: Requires MCP protocol revision 2025-06-18+ for elicitation; graceful degradation for older clients
+- **Webtest Tools**: Five orchestration tools for the testing workflow:
+  1. `webtest_init` - Initialize analysis workspace
+  2. `webtest_crawl_app` - Dynamic goal-directed exploration with checkpointing and loop detection
+  3. `webtest_analyze_app` - Reverse-engineer app structure and flows
+  4. `webtest_generate_tests` - Produce test cases from analysis
+  5. `webtest_run_tests` - Execute tests with evidence capture
+- **Webtest Resources**: Stable `webtest://` URI-based artifacts with **listChanged/subscribe** support for live artifact surfacing during long operations
+- **Webtest Prompts**: Prompt templates for smooth client UX
+- **MCP Sampling Integration**: Client-controlled LLM reasoning for all AI decisions with **fallback mode** when sampling unavailable
+- **Elicitation Support**: Interactive user decisions during crawl with **fallback mode** when elicitation unavailable
+- **Progress & Cancellation**: Long-running operations report progress (with budget status) and respond to cancellation
+- **Playwright MCP Integration**: Orchestration with **dynamic tool discovery and capability adapter** (version/implementation resilient)
+- **Structured Logging**: MCP logging notifications with correlation IDs, log level control, and sensitive data redaction
+
+### **BREAKING** Changes
+
+- Removes `hello` tool (demonstration no longer needed)
+- Server now requires Playwright MCP server as external dependency
+
+### Security Additions
+
+- Domain allowlist enforcement with subdomain support
+- **Comprehensive prompt injection hardening** with defense-in-depth:
+  - Untrusted page content demarcation
+  - Protected system instruction prefix
+  - Scope expansion detection and blocking
+  - Data exfiltration pattern blocking
+  - Audit logging of all sampling I/O
+- **Injection test suite** validating resistance to direct/indirect injection, goal hijacking, credential phishing
+- Sensitive data redaction in logs (URLs, cookies, passwords)
+- Never requests credentials via elicitation
+
+### Robustness Additions
+
+- **Crawl checkpointing** every N steps with resume support
+- **Loop detection and prevention**: DOM signature tracking, URL cycle detection, action repeat blocking
+- **Budget enforcement**: maxSteps, maxMinutes, maxPages limits with graceful partial output
+
+## Key Features Summary
+
+| Feature | Description |
+|---------|-------------|
+| Resources listChanged/subscribe | Surface new artifacts live during crawl/test execution |
+| Runtime fallbacks | Graceful degradation when Sampling/Elicitation unsupported |
+| Playwright MCP adapter | Dynamic tool discovery; version/implementation resilient |
+| Sampling injection hardening | Defense-in-depth with audit logging and test suite |
+| Crawl checkpointing | Resume interrupted crawls; partial results on timeout |
+| Loop prevention | DOM signatures, URL cycles, action repeats detected |
+
+## Impact
+
+- Affected specs: `mcp-server-core` (lifecycle changes), plus new specs for `webtest-tools`, `webtest-resources`, `webtest-prompts`, `webtest-sampling`, `webtest-lifecycle`, `webtest-logging`
+- Affected code: `src/server.ts`, `src/tools/`, new directories for resources/prompts/sampling/lifecycle/logging
+- External dependencies: Playwright MCP server (Microsoft or compatible implementation)

package/openspec/changes/archive/2025-12-18-add-webtest-orchestrator/specs/mcp-server-core/spec.md

@@ -0,0 +1,129 @@
+# mcp-server-core Spec Delta
+
+## REMOVED Requirements
+
+### Requirement: Hello Tool Implementation
+
+**Reason**: The demonstration "hello" tool is no longer needed now that real webtest tools are implemented.
+
+**Migration**: Remove `src/tools/hello.ts` and `src/tools/hello.test.ts`. Update tool registry to exclude hello tool.
+
+## MODIFIED Requirements
+
+### Requirement: MCP Server Initialization
+
+The system SHALL provide an MCP server that initializes with proper identification, negotiates client capabilities, and connects to the configured transport.
+
+#### Scenario: Server starts with stdio transport
+
+- **GIVEN** the environment variable `TRANSPORT` is set to `stdio` or not set
+- **WHEN** the server entry point is executed
+- **THEN** it SHALL identify itself with name "testing-mcp" and version from package.json
+- **AND** it SHALL connect to stdio transport for communication
+
+#### Scenario: Server starts with HTTP transport
+
+- **GIVEN** the environment variable `TRANSPORT` is set to `http`
+- **AND** the environment variable `PORT` is set to a valid port number
+- **WHEN** the server entry point is executed
+- **THEN** it SHALL start a Streamable HTTP server on the specified port
+- **AND** it SHALL accept MCP protocol connections over HTTP
+
+#### Scenario: Server handles graceful shutdown
+
+- **GIVEN** the server is running
+- **WHEN** the process receives SIGINT or SIGTERM
+- **THEN** the server SHALL disconnect gracefully
+- **AND** any active Playwright MCP subprocess SHALL be terminated
+- **AND** the process SHALL exit with code 0
+
+#### Scenario: Server negotiates client capabilities
+
+- **GIVEN** a client connects to the server
+- **WHEN** the initialize handshake completes
+- **THEN** the server SHALL record client capabilities for sampling, elicitation, logging, and progress
+- **AND** the server SHALL adapt runtime behavior based on available capabilities
+
+## MODIFIED Requirements
+
+### Requirement: Configuration Validation
+
+The system SHALL validate configuration at startup using Zod schemas and fail fast on invalid configuration, including webtest-specific settings.
+
+#### Scenario: Valid configuration starts server
+
+- **GIVEN** all required environment variables are valid
+- **WHEN** the server starts
+- **THEN** configuration SHALL be parsed and validated
+- **AND** the server SHALL proceed with initialization
+
+#### Scenario: Invalid configuration fails fast
+
+- **GIVEN** an environment variable has an invalid value (e.g., `PORT=invalid`)
+- **WHEN** the server attempts to start
+- **THEN** it SHALL log a descriptive error message
+- **AND** the process SHALL exit with a non-zero code
+
+#### Scenario: Webtest workspace configuration is validated
+
+- **GIVEN** the environment variable `WEBTEST_WORKSPACE_DIR` is set
+- **WHEN** the server starts
+- **THEN** it SHALL validate the path is writable
+- **AND** create the directory if it does not exist
+
+## MODIFIED Requirements
+
+### Requirement: Self-Describing Tool Registry
+
+The system SHALL maintain a tool registry where each tool exports a standard interface including name, description, Zod input schema, and async handler function, supporting the webtest tool namespace.
+
+#### Scenario: Tool is registered and discoverable
+
+- **GIVEN** a tool is added to the registry
+- **WHEN** an MCP client requests the tool list
+- **THEN** the tool SHALL appear in the list with its name and description
+- **AND** the input JSON Schema SHALL be generated from the Zod schema
+
+#### Scenario: New tool follows registry pattern
+
+- **GIVEN** a developer creates a new tool
+- **WHEN** the tool exports `{ name, description, inputSchema, handler }`
+- **AND** the tool is added to the registry index
+- **THEN** it SHALL be automatically registered with the MCP server
+
+#### Scenario: Webtest tools use namespaced naming
+
+- **GIVEN** the webtest tools are registered
+- **WHEN** an MCP client requests the tool list
+- **THEN** tools SHALL appear with `webtest_` prefix (e.g., `webtest_init`, `webtest_crawl_app`)
+
+## MODIFIED Requirements
+
+### Requirement: Structured Logging
+
+The system SHALL provide structured JSON logging with configurable log levels, automatic redaction of sensitive fields, and optional emission as MCP logging notifications.
+
+#### Scenario: Log output is structured JSON
+
+- **GIVEN** the server is running
+- **WHEN** a log event occurs
+- **THEN** it SHALL be output as a JSON object with timestamp, level, and message fields
+
+#### Scenario: Sensitive fields are redacted
+
+- **GIVEN** a log message contains a field matching a sensitive key pattern (password, token, secret, apiKey, authorization)
+- **WHEN** the log is written
+- **THEN** the sensitive field value SHALL be replaced with "[REDACTED]"
+
+#### Scenario: Log level is configurable
+
+- **GIVEN** the environment variable `LOG_LEVEL` is set to a valid level (debug, info, warn, error)
+- **WHEN** the server starts
+- **THEN** only log messages at or above that level SHALL be output
+
+#### Scenario: Logs are emitted as MCP notifications when supported
+
+- **GIVEN** the client supports MCP logging notifications
+- **WHEN** a log event occurs
+- **THEN** it SHALL be emitted as a `notifications/message` to the client
+- **AND** the log level SHALL map to MCP log levels (debug, info, warning, error)

package/openspec/changes/archive/2025-12-18-add-webtest-orchestrator/specs/webtest-lifecycle/spec.md

@@ -0,0 +1,138 @@
+# webtest-lifecycle Specification
+
+## Purpose
+
+Defines the MCP lifecycle management and capability negotiation for the web testing server.
+
+## ADDED Requirements
+
+### Requirement: MCP Protocol Version Requirements
+
+The system SHALL require MCP protocol revision 2025-06-18 or later to ensure elicitation support is available.
+
+#### Scenario: Server declares required protocol version
+
+- **GIVEN** the server starts
+- **WHEN** it responds to initialize request
+- **THEN** it SHALL declare `protocolVersion: "2025-06-18"` or later
+- **AND** include elicitation in server capabilities
+
+#### Scenario: Client with older protocol version
+
+- **GIVEN** a client connects with protocol version older than 2025-06-18
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that elicitation is NOT available
+- **AND** log a warning about degraded functionality
+
+#### Scenario: Protocol version mismatch handling
+
+- **GIVEN** a client requests a protocol version the server cannot satisfy
+- **WHEN** version negotiation occurs
+- **THEN** the server SHALL negotiate to the highest mutually supported version
+- **AND** adjust available features accordingly
+
+### Requirement: MCP Lifecycle Management
+
+The system SHALL implement proper MCP lifecycle phases (initialize, operate, shutdown) and maintain lifecycle state.
+
+#### Scenario: Server transitions through lifecycle phases
+
+- **GIVEN** a client connects to the server
+- **WHEN** the connection is established
+- **THEN** the server SHALL be in "initializing" state
+- **AND** after successful initialize handshake, transition to "operating" state
+- **AND** on shutdown signal, transition to "shutdown" state
+
+#### Scenario: Server rejects operations before initialization
+
+- **GIVEN** the server is in "initializing" state
+- **WHEN** a client sends a tool call request
+- **THEN** the server SHALL return an error indicating initialization not complete
+
+#### Scenario: Server rejects new operations during shutdown
+
+- **GIVEN** the server is in "shutdown" state
+- **WHEN** a client sends a new tool call request
+- **THEN** the server SHALL return an error indicating server is shutting down
+
+### Requirement: Client Capability Negotiation
+
+The system SHALL query and record client capabilities during initialization and adapt behavior accordingly.
+
+#### Scenario: Server records sampling capability
+
+- **GIVEN** a client connects with `capabilities.sampling` present
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that sampling is available
+- **AND** webtest tools SHALL use `sampling/createMessage` for LLM reasoning
+
+#### Scenario: Server records elicitation capability
+
+- **GIVEN** a client connects with `capabilities.elicitation` present
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that elicitation is available
+- **AND** webtest tools SHALL use `elicitation/create` for user decisions
+
+#### Scenario: Server records logging capability
+
+- **GIVEN** a client connects with `capabilities.logging` present
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that logging notifications are supported
+- **AND** the logger SHALL emit `notifications/message` to the client
+
+#### Scenario: Server records progress capability
+
+- **GIVEN** a client connects with MCP progress support
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that progress notifications are supported
+- **AND** long-running tools SHALL emit `notifications/progress`
+
+#### Scenario: Server records resources listChanged capability
+
+- **GIVEN** a client connects with `capabilities.resources.listChanged` present
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that resource list change notifications are supported
+- **AND** resource creation SHALL emit `notifications/resources/list_changed`
+
+#### Scenario: Server records resources subscribe capability
+
+- **GIVEN** a client connects with `capabilities.resources.subscribe` present
+- **WHEN** initialization completes
+- **THEN** the server SHALL record that resource subscriptions are supported
+- **AND** resource updates SHALL emit `notifications/resources/updated` to subscribers
+
+#### Scenario: Fallback when sampling not supported
+
+- **GIVEN** a client connects without `capabilities.sampling`
+- **WHEN** a webtest tool requires LLM reasoning
+- **THEN** the tool SHALL return a prompt resource for manual execution
+- **AND** the tool output SHALL include `needsManualInput: true`
+
+#### Scenario: Fallback when elicitation not supported
+
+- **GIVEN** a client connects without `capabilities.elicitation`
+- **WHEN** a webtest tool needs user decision
+- **THEN** the tool SHALL include questions in its output
+- **AND** the tool output SHALL include `needsInput: true` with question details
+
+### Requirement: Capability Query API
+
+The system SHALL provide internal APIs for tools to query client capabilities at runtime.
+
+#### Scenario: Tool queries sampling availability
+
+- **GIVEN** a tool handler is executing
+- **WHEN** it calls `capabilities.hasSampling()`
+- **THEN** it SHALL receive a boolean indicating sampling support
+
+#### Scenario: Tool queries elicitation availability
+
+- **GIVEN** a tool handler is executing
+- **WHEN** it calls `capabilities.hasElicitation()`
+- **THEN** it SHALL receive a boolean indicating elicitation support
+
+#### Scenario: Tool queries all capabilities
+
+- **GIVEN** a tool handler is executing
+- **WHEN** it calls `capabilities.getAll()`
+- **THEN** it SHALL receive an object with all recorded capabilities

package/openspec/changes/archive/2025-12-18-add-webtest-orchestrator/specs/webtest-logging/spec.md

@@ -0,0 +1,211 @@
+# webtest-logging Specification
+
+## Purpose
+
+Defines structured logging with MCP logging notifications, correlation IDs, sensitive data redaction, and log level control.
+
+## ADDED Requirements
+
+### Requirement: MCP Logging Notifications
+
+The system SHALL emit structured logs as MCP logging notifications when the client supports it.
+
+#### Scenario: Log emitted as MCP notification
+
+- **GIVEN** the client supports MCP logging (`capabilities.logging` present)
+- **WHEN** a log event occurs
+- **THEN** it SHALL emit `notifications/message` with:
+  - `level`: one of "debug", "info", "warning", "error"
+  - `logger`: "webtest"
+  - `data`: structured log payload
+
+#### Scenario: Fallback to stderr when logging unsupported
+
+- **GIVEN** the client does not support MCP logging
+- **WHEN** a log event occurs
+- **THEN** it SHALL write to stderr as JSON
+- **AND** not attempt MCP notification
+
+#### Scenario: Log level is respected
+
+- **GIVEN** client log level is set to "warning"
+- **WHEN** an "info" level log is generated
+- **THEN** it SHALL NOT be emitted
+- **AND** "warning" and "error" logs SHALL be emitted
+
+### Requirement: Logging Level Control
+
+The system SHALL support dynamic log level configuration via MCP and environment.
+
+#### Scenario: Log level set via environment
+
+- **GIVEN** environment variable `LOG_LEVEL` is set to "debug"
+- **WHEN** the server starts
+- **THEN** all log levels (debug, info, warning, error) SHALL be emitted
+
+#### Scenario: Client sets log level via logging/setLevel
+
+- **GIVEN** client supports `logging/setLevel`
+- **WHEN** client sends `logging/setLevel` with level "error"
+- **THEN** only "error" level logs SHALL be emitted thereafter
+- **AND** this SHALL override the environment setting
+
+#### Scenario: Default log level
+
+- **GIVEN** no log level is configured
+- **WHEN** the server starts
+- **THEN** the default log level SHALL be "info"
+
+### Requirement: Correlation IDs
+
+The system SHALL include correlation IDs in all log messages to enable tracing across operations.
+
+#### Scenario: Analysis ID is included in logs
+
+- **GIVEN** a tool is executing within an analysis context
+- **WHEN** a log is emitted
+- **THEN** it SHALL include `analysisId` in the log data
+
+#### Scenario: Crawl ID is included in crawl logs
+
+- **GIVEN** a crawl is in progress
+- **WHEN** a log is emitted during the crawl
+- **THEN** it SHALL include `crawlId` in addition to `analysisId`
+
+#### Scenario: Test run ID is included in test logs
+
+- **GIVEN** a test case is being executed
+- **WHEN** a log is emitted during test execution
+- **THEN** it SHALL include `testRunId` in addition to `analysisId`
+
+#### Scenario: Iteration number is included in loop logs
+
+- **GIVEN** a crawl or test loop is executing
+- **WHEN** a log is emitted during an iteration
+- **THEN** it SHALL include `iteration` number
+
+#### Scenario: Request ID is included when available
+
+- **GIVEN** a tool is handling an MCP request with `_meta.requestId`
+- **WHEN** logs are emitted during that request
+- **THEN** they SHALL include `requestId` for correlation with client logs
+
+### Requirement: Structured Log Format
+
+The system SHALL emit logs in a consistent structured format.
+
+#### Scenario: Log structure is consistent
+
+- **GIVEN** any log event occurs
+- **WHEN** it is emitted
+- **THEN** it SHALL include:
+  - `timestamp`: ISO 8601 format
+  - `level`: log level
+  - `message`: human-readable message
+  - `context`: object with correlation IDs
+  - `data`: optional additional structured data
+
+#### Scenario: Playwright MCP tool calls are logged
+
+- **GIVEN** a Playwright MCP tool is called
+- **WHEN** the call completes
+- **THEN** it SHALL log:
+  - `message`: "Playwright tool executed"
+  - `data.tool`: tool name
+  - `data.duration`: execution time in ms
+  - `data.success`: boolean
+  - `data.error`: error message if failed (sensitive data redacted)
+
+#### Scenario: Sampling calls are logged
+
+- **GIVEN** a sampling request is made
+- **WHEN** the request completes
+- **THEN** it SHALL log:
+  - `message`: "Sampling completed"
+  - `data.promptTokens`: approximate prompt size
+  - `data.responseTokens`: approximate response size
+  - `data.duration`: execution time in ms
+  - `data.validationPassed`: boolean
+
+### Requirement: Sensitive Data Redaction
+
+The system SHALL redact sensitive data from logs to prevent credential exposure.
+
+#### Scenario: URL query parameters are redacted
+
+- **GIVEN** a log includes a URL
+- **WHEN** the URL contains query parameters matching sensitive patterns (token, key, password, secret, auth, session)
+- **THEN** parameter values SHALL be replaced with "[REDACTED]"
+
+#### Scenario: Cookie values are redacted
+
+- **GIVEN** a log includes cookie data
+- **WHEN** cookies are serialized
+- **THEN** cookie values SHALL be replaced with "[REDACTED]"
+- **AND** only cookie names SHALL be visible
+
+#### Scenario: Form input values are redacted
+
+- **GIVEN** a log includes form interaction (type action)
+- **WHEN** the input is to a password or sensitive field
+- **THEN** the typed value SHALL be replaced with "[REDACTED]"
+
+#### Scenario: HTML content is truncated
+
+- **GIVEN** a log includes HTML content
+- **WHEN** the HTML exceeds 500 characters
+- **THEN** it SHALL be truncated with "...[truncated]"
+- **AND** sensitive elements (script, style) SHALL be removed
+
+#### Scenario: Known sensitive field patterns are redacted
+
+- **GIVEN** a log data object is being serialized
+- **WHEN** it contains keys matching sensitive patterns:
+  - password, passwd, pwd
+  - token, apiKey, api_key
+  - secret, credential
+  - authorization, auth
+  - session, cookie
+- **THEN** those values SHALL be replaced with "[REDACTED]"
+
+### Requirement: Operation Step Logging
+
+The system SHALL log detailed step information for debugging and audit.
+
+#### Scenario: Crawl step is logged
+
+- **GIVEN** a crawl iteration completes
+- **WHEN** step logging occurs
+- **THEN** it SHALL log at "debug" level:
+  - Current URL
+  - Action taken (tool + args with sensitive data redacted)
+  - Result summary
+  - Goal progress assessment
+
+#### Scenario: Test step is logged
+
+- **GIVEN** a test step executes
+- **WHEN** step logging occurs
+- **THEN** it SHALL log at "info" level:
+  - Step number and description
+  - Actions executed
+  - Pass/fail result
+  - Evidence URIs
+
+#### Scenario: Elicitation event is logged
+
+- **GIVEN** elicitation is triggered
+- **WHEN** user response is received
+- **THEN** it SHALL log at "info" level:
+  - Elicitation type (cookie, modal, ambiguous, auth)
+  - Options presented
+  - User selection
+
+#### Scenario: Security event is logged
+
+- **GIVEN** a security check fails (domain validation, injection detection)
+- **WHEN** the violation is detected
+- **THEN** it SHALL log at "warning" level:
+  - Violation type
+  - Attempted action (redacted as needed)
+  - Remediation taken