retestkit 1.4.1 → 1.5.0

package/openspec/specs/webtest-lifecycle/spec.md

@@ -1,136 +0,0 @@
-# webtest-lifecycle Specification
-
-## Purpose
-TBD - created by archiving change add-webtest-orchestrator. Update Purpose after archive.
-## Requirements
-### Requirement: MCP Protocol Version Requirements
-
-The system SHALL require MCP protocol revision 2025-06-18 or later to ensure elicitation support is available.
-
-#### Scenario: Server declares required protocol version
-
-- **GIVEN** the server starts
-- **WHEN** it responds to initialize request
-- **THEN** it SHALL declare `protocolVersion: "2025-06-18"` or later
-- **AND** include elicitation in server capabilities
-
-#### Scenario: Client with older protocol version
-
-- **GIVEN** a client connects with protocol version older than 2025-06-18
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that elicitation is NOT available
-- **AND** log a warning about degraded functionality
-
-#### Scenario: Protocol version mismatch handling
-
-- **GIVEN** a client requests a protocol version the server cannot satisfy
-- **WHEN** version negotiation occurs
-- **THEN** the server SHALL negotiate to the highest mutually supported version
-- **AND** adjust available features accordingly
-
-### Requirement: MCP Lifecycle Management
-
-The system SHALL implement proper MCP lifecycle phases (initialize, operate, shutdown) and maintain lifecycle state.
-
-#### Scenario: Server transitions through lifecycle phases
-
-- **GIVEN** a client connects to the server
-- **WHEN** the connection is established
-- **THEN** the server SHALL be in "initializing" state
-- **AND** after successful initialize handshake, transition to "operating" state
-- **AND** on shutdown signal, transition to "shutdown" state
-
-#### Scenario: Server rejects operations before initialization
-
-- **GIVEN** the server is in "initializing" state
-- **WHEN** a client sends a tool call request
-- **THEN** the server SHALL return an error indicating initialization not complete
-
-#### Scenario: Server rejects new operations during shutdown
-
-- **GIVEN** the server is in "shutdown" state
-- **WHEN** a client sends a new tool call request
-- **THEN** the server SHALL return an error indicating server is shutting down
-
-### Requirement: Client Capability Negotiation
-
-The system SHALL query and record client capabilities during initialization and adapt behavior accordingly.
-
-#### Scenario: Server records sampling capability
-
-- **GIVEN** a client connects with `capabilities.sampling` present
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that sampling is available
-- **AND** webtest tools SHALL use `sampling/createMessage` for LLM reasoning
-
-#### Scenario: Server records elicitation capability
-
-- **GIVEN** a client connects with `capabilities.elicitation` present
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that elicitation is available
-- **AND** webtest tools SHALL use `elicitation/create` for user decisions
-
-#### Scenario: Server records logging capability
-
-- **GIVEN** a client connects with `capabilities.logging` present
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that logging notifications are supported
-- **AND** the logger SHALL emit `notifications/message` to the client
-
-#### Scenario: Server records progress capability
-
-- **GIVEN** a client connects with MCP progress support
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that progress notifications are supported
-- **AND** long-running tools SHALL emit `notifications/progress`
-
-#### Scenario: Server records resources listChanged capability
-
-- **GIVEN** a client connects with `capabilities.resources.listChanged` present
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that resource list change notifications are supported
-- **AND** resource creation SHALL emit `notifications/resources/list_changed`
-
-#### Scenario: Server records resources subscribe capability
-
-- **GIVEN** a client connects with `capabilities.resources.subscribe` present
-- **WHEN** initialization completes
-- **THEN** the server SHALL record that resource subscriptions are supported
-- **AND** resource updates SHALL emit `notifications/resources/updated` to subscribers
-
-#### Scenario: Fallback when sampling not supported
-
-- **GIVEN** a client connects without `capabilities.sampling`
-- **WHEN** a webtest tool requires LLM reasoning
-- **THEN** the tool SHALL return a prompt resource for manual execution
-- **AND** the tool output SHALL include `needsManualInput: true`
-
-#### Scenario: Fallback when elicitation not supported
-
-- **GIVEN** a client connects without `capabilities.elicitation`
-- **WHEN** a webtest tool needs user decision
-- **THEN** the tool SHALL include questions in its output
-- **AND** the tool output SHALL include `needsInput: true` with question details
-
-### Requirement: Capability Query API
-
-The system SHALL provide internal APIs for tools to query client capabilities at runtime.
-
-#### Scenario: Tool queries sampling availability
-
-- **GIVEN** a tool handler is executing
-- **WHEN** it calls `capabilities.hasSampling()`
-- **THEN** it SHALL receive a boolean indicating sampling support
-
-#### Scenario: Tool queries elicitation availability
-
-- **GIVEN** a tool handler is executing
-- **WHEN** it calls `capabilities.hasElicitation()`
-- **THEN** it SHALL receive a boolean indicating elicitation support
-
-#### Scenario: Tool queries all capabilities
-
-- **GIVEN** a tool handler is executing
-- **WHEN** it calls `capabilities.getAll()`
-- **THEN** it SHALL receive an object with all recorded capabilities
-

package/openspec/specs/webtest-logging/spec.md

@@ -1,209 +0,0 @@
-# webtest-logging Specification
-
-## Purpose
-TBD - created by archiving change add-webtest-orchestrator. Update Purpose after archive.
-## Requirements
-### Requirement: MCP Logging Notifications
-
-The system SHALL emit structured logs as MCP logging notifications when the client supports it.
-
-#### Scenario: Log emitted as MCP notification
-
-- **GIVEN** the client supports MCP logging (`capabilities.logging` present)
-- **WHEN** a log event occurs
-- **THEN** it SHALL emit `notifications/message` with:
-  - `level`: one of "debug", "info", "warning", "error"
-  - `logger`: "webtest"
-  - `data`: structured log payload
-
-#### Scenario: Fallback to stderr when logging unsupported
-
-- **GIVEN** the client does not support MCP logging
-- **WHEN** a log event occurs
-- **THEN** it SHALL write to stderr as JSON
-- **AND** not attempt MCP notification
-
-#### Scenario: Log level is respected
-
-- **GIVEN** client log level is set to "warning"
-- **WHEN** an "info" level log is generated
-- **THEN** it SHALL NOT be emitted
-- **AND** "warning" and "error" logs SHALL be emitted
-
-### Requirement: Logging Level Control
-
-The system SHALL support dynamic log level configuration via MCP and environment.
-
-#### Scenario: Log level set via environment
-
-- **GIVEN** environment variable `LOG_LEVEL` is set to "debug"
-- **WHEN** the server starts
-- **THEN** all log levels (debug, info, warning, error) SHALL be emitted
-
-#### Scenario: Client sets log level via logging/setLevel
-
-- **GIVEN** client supports `logging/setLevel`
-- **WHEN** client sends `logging/setLevel` with level "error"
-- **THEN** only "error" level logs SHALL be emitted thereafter
-- **AND** this SHALL override the environment setting
-
-#### Scenario: Default log level
-
-- **GIVEN** no log level is configured
-- **WHEN** the server starts
-- **THEN** the default log level SHALL be "info"
-
-### Requirement: Correlation IDs
-
-The system SHALL include correlation IDs in all log messages to enable tracing across operations.
-
-#### Scenario: Analysis ID is included in logs
-
-- **GIVEN** a tool is executing within an analysis context
-- **WHEN** a log is emitted
-- **THEN** it SHALL include `analysisId` in the log data
-
-#### Scenario: Crawl ID is included in crawl logs
-
-- **GIVEN** a crawl is in progress
-- **WHEN** a log is emitted during the crawl
-- **THEN** it SHALL include `crawlId` in addition to `analysisId`
-
-#### Scenario: Test run ID is included in test logs
-
-- **GIVEN** a test case is being executed
-- **WHEN** a log is emitted during test execution
-- **THEN** it SHALL include `testRunId` in addition to `analysisId`
-
-#### Scenario: Iteration number is included in loop logs
-
-- **GIVEN** a crawl or test loop is executing
-- **WHEN** a log is emitted during an iteration
-- **THEN** it SHALL include `iteration` number
-
-#### Scenario: Request ID is included when available
-
-- **GIVEN** a tool is handling an MCP request with `_meta.requestId`
-- **WHEN** logs are emitted during that request
-- **THEN** they SHALL include `requestId` for correlation with client logs
-
-### Requirement: Structured Log Format
-
-The system SHALL emit logs in a consistent structured format.
-
-#### Scenario: Log structure is consistent
-
-- **GIVEN** any log event occurs
-- **WHEN** it is emitted
-- **THEN** it SHALL include:
-  - `timestamp`: ISO 8601 format
-  - `level`: log level
-  - `message`: human-readable message
-  - `context`: object with correlation IDs
-  - `data`: optional additional structured data
-
-#### Scenario: Playwright MCP tool calls are logged
-
-- **GIVEN** a Playwright MCP tool is called
-- **WHEN** the call completes
-- **THEN** it SHALL log:
-  - `message`: "Playwright tool executed"
-  - `data.tool`: tool name
-  - `data.duration`: execution time in ms
-  - `data.success`: boolean
-  - `data.error`: error message if failed (sensitive data redacted)
-
-#### Scenario: Sampling calls are logged
-
-- **GIVEN** a sampling request is made
-- **WHEN** the request completes
-- **THEN** it SHALL log:
-  - `message`: "Sampling completed"
-  - `data.promptTokens`: approximate prompt size
-  - `data.responseTokens`: approximate response size
-  - `data.duration`: execution time in ms
-  - `data.validationPassed`: boolean
-
-### Requirement: Sensitive Data Redaction
-
-The system SHALL redact sensitive data from logs to prevent credential exposure.
-
-#### Scenario: URL query parameters are redacted
-
-- **GIVEN** a log includes a URL
-- **WHEN** the URL contains query parameters matching sensitive patterns (token, key, password, secret, auth, session)
-- **THEN** parameter values SHALL be replaced with "[REDACTED]"
-
-#### Scenario: Cookie values are redacted
-
-- **GIVEN** a log includes cookie data
-- **WHEN** cookies are serialized
-- **THEN** cookie values SHALL be replaced with "[REDACTED]"
-- **AND** only cookie names SHALL be visible
-
-#### Scenario: Form input values are redacted
-
-- **GIVEN** a log includes form interaction (type action)
-- **WHEN** the input is to a password or sensitive field
-- **THEN** the typed value SHALL be replaced with "[REDACTED]"
-
-#### Scenario: HTML content is truncated
-
-- **GIVEN** a log includes HTML content
-- **WHEN** the HTML exceeds 500 characters
-- **THEN** it SHALL be truncated with "...[truncated]"
-- **AND** sensitive elements (script, style) SHALL be removed
-
-#### Scenario: Known sensitive field patterns are redacted
-
-- **GIVEN** a log data object is being serialized
-- **WHEN** it contains keys matching sensitive patterns:
-  - password, passwd, pwd
-  - token, apiKey, api_key
-  - secret, credential
-  - authorization, auth
-  - session, cookie
-- **THEN** those values SHALL be replaced with "[REDACTED]"
-
-### Requirement: Operation Step Logging
-
-The system SHALL log detailed step information for debugging and audit.
-
-#### Scenario: Crawl step is logged
-
-- **GIVEN** a crawl iteration completes
-- **WHEN** step logging occurs
-- **THEN** it SHALL log at "debug" level:
-  - Current URL
-  - Action taken (tool + args with sensitive data redacted)
-  - Result summary
-  - Goal progress assessment
-
-#### Scenario: Test step is logged
-
-- **GIVEN** a test step executes
-- **WHEN** step logging occurs
-- **THEN** it SHALL log at "info" level:
-  - Step number and description
-  - Actions executed
-  - Pass/fail result
-  - Evidence URIs
-
-#### Scenario: Elicitation event is logged
-
-- **GIVEN** elicitation is triggered
-- **WHEN** user response is received
-- **THEN** it SHALL log at "info" level:
-  - Elicitation type (cookie, modal, ambiguous, auth)
-  - Options presented
-  - User selection
-
-#### Scenario: Security event is logged
-
-- **GIVEN** a security check fails (domain validation, injection detection)
-- **WHEN** the violation is detected
-- **THEN** it SHALL log at "warning" level:
-  - Violation type
-  - Attempted action (redacted as needed)
-  - Remediation taken
-

package/openspec/specs/webtest-prompts/spec.md

@@ -1,155 +0,0 @@
-# webtest-prompts Specification
-
-## Purpose
-TBD - created by archiving change add-webtest-orchestrator. Update Purpose after archive.
-## Requirements
-### Requirement: Prompt Template Registration
-
-The system SHALL register prompt templates with the MCP server for client discovery.
-
-#### Scenario: Prompts are listed on prompts/list
-
-- **GIVEN** a client calls `prompts/list`
-- **WHEN** the response is returned
-- **THEN** it SHALL include prompts:
-  - `webtest-start-analysis` (Start web testing analysis)
-  - `webtest-crawl` (Crawl to satisfy focus)
-  - `webtest-analyze` (Analyze application)
-  - `webtest-generate-tests` (Generate test cases)
-  - `webtest-run-test` (Run a test case)
-
-### Requirement: Start Analysis Prompt
-
-The system SHALL provide a prompt template to initiate a web testing analysis.
-
-#### Scenario: Start analysis prompt is invoked
-
-- **GIVEN** client invokes `prompts/get` with name `webtest-start-analysis`
-- **WHEN** the prompt is returned
-- **THEN** it SHALL include:
-  - Description: "Start a new web testing analysis for a URL"
-  - Arguments: `url` (required), `focus` (required), `maxSteps` (optional), `maxPages` (optional)
-
-#### Scenario: Start analysis prompt generates tool call
-
-- **GIVEN** client invokes the prompt with arguments
-- **WHEN** the prompt messages are returned
-- **THEN** it SHALL include a user message instructing to call `webtest_init`
-- **AND** include the provided arguments in the instruction
-
-### Requirement: Crawl Prompt
-
-The system SHALL provide a prompt template to start or continue a crawl.
-
-#### Scenario: Crawl prompt is invoked
-
-- **GIVEN** client invokes `prompts/get` with name `webtest-crawl`
-- **WHEN** the prompt is returned
-- **THEN** it SHALL include:
-  - Description: "Crawl a web application to explore and achieve a goal"
-  - Arguments: `analysisId` (required), `goal` (optional), `strategy` (optional)
-
-#### Scenario: Crawl prompt uses analysis goal by default
-
-- **GIVEN** client invokes crawl prompt with only `analysisId`
-- **WHEN** the prompt messages are returned
-- **THEN** it SHALL instruct to use the original focus as the crawl goal
-
-### Requirement: Analyze Prompt
-
-The system SHALL provide a prompt template to analyze crawled data.
-
-#### Scenario: Analyze prompt is invoked
-
-- **GIVEN** client invokes `prompts/get` with name `webtest-analyze`
-- **WHEN** the prompt is returned
-- **THEN** it SHALL include:
-  - Description: "Analyze a crawled web application to understand its structure"
-  - Arguments: `analysisId` (required), `crawlId` (required)
-
-#### Scenario: Analyze prompt lists available crawls
-
-- **GIVEN** client invokes analyze prompt with only `analysisId`
-- **WHEN** the prompt messages are returned
-- **THEN** it SHALL include context about available crawls for that analysis
-
-### Requirement: Generate Tests Prompt
-
-The system SHALL provide a prompt template to generate test cases.
-
-#### Scenario: Generate tests prompt is invoked
-
-- **GIVEN** client invokes `prompts/get` with name `webtest-generate-tests`
-- **WHEN** the prompt is returned
-- **THEN** it SHALL include:
-  - Description: "Generate test cases from application analysis"
-  - Arguments: `analysisId` (required), `count` (optional), `types` (optional)
-
-#### Scenario: Generate tests prompt offers strategy options
-
-- **GIVEN** client invokes generate tests prompt
-- **WHEN** the prompt messages are returned
-- **THEN** it SHALL mention available test types: "smoke", "negative", "boundary", "integration"
-
-### Requirement: Run Test Prompt
-
-The system SHALL provide a prompt template to execute a test case.
-
-#### Scenario: Run test prompt is invoked
-
-- **GIVEN** client invokes `prompts/get` with name `webtest-run-test`
-- **WHEN** the prompt is returned
-- **THEN** it SHALL include:
-  - Description: "Execute a test case and capture results"
-  - Arguments: `analysisId` (required), `testCaseId` (required)
-
-#### Scenario: Run test prompt lists available tests
-
-- **GIVEN** client invokes run test prompt with only `analysisId`
-- **WHEN** the prompt messages are returned
-- **THEN** it SHALL include context about available test cases for that analysis
-
-### Requirement: Prompt Argument Validation
-
-The system SHALL validate prompt arguments and return helpful errors.
-
-#### Scenario: Missing required argument returns error
-
-- **GIVEN** client invokes `webtest-start-analysis` without `url`
-- **WHEN** validation occurs
-- **THEN** it SHALL return error indicating `url` is required
-
-#### Scenario: Invalid analysisId returns error
-
-- **GIVEN** client invokes a prompt with non-existent `analysisId`
-- **WHEN** validation occurs
-- **THEN** it SHALL return error indicating analysis not found
-
-### Requirement: Prompt Chaining Guidance
-
-The system SHALL include guidance in prompt outputs for the recommended workflow sequence.
-
-#### Scenario: Start analysis prompt suggests next step
-
-- **GIVEN** client uses start analysis prompt
-- **WHEN** the prompt messages are returned
-- **THEN** they SHALL include guidance: "After analysis starts, use webtest-crawl to explore the application"
-
-#### Scenario: Crawl prompt suggests next step
-
-- **GIVEN** client uses crawl prompt
-- **WHEN** the prompt messages are returned
-- **THEN** they SHALL include guidance: "After crawl completes, use webtest-analyze to understand the application structure"
-
-#### Scenario: Analyze prompt suggests next step
-
-- **GIVEN** client uses analyze prompt
-- **WHEN** the prompt messages are returned
-- **THEN** they SHALL include guidance: "After analysis, use webtest-generate-tests to create test cases"
-
-#### Scenario: Generate tests prompt suggests next step
-
-- **GIVEN** client uses generate tests prompt
-- **WHEN** the prompt messages are returned
-- **THEN** they SHALL include guidance: "After test generation, use webtest-run-test to execute individual test cases"
-