@miniidealab/openlogos 0.3.0 → 0.3.2

package/skills/test-orchestrator/SKILL.en.md

@@ -0,0 +1,142 @@
+# Skill: Test Orchestrator
+
+> Design **API orchestration test** cases based on business scenarios and sequence diagrams (Phase 3 Step 3b), covering normal/exception/boundary scenarios. Automatically identify external dependencies and apply test strategies as end-to-end API acceptance criteria. **Only applicable to projects involving APIs.**
+
+## Relationship with test-writer
+
+This Skill is responsible for the **top layer** of the test pyramid — API orchestration tests (HTTP request level), executed in Phase 3 Step 3b.
+
+The lower-level unit tests and scenario tests (function call level) are completed by the `test-writer` Skill in Step 3a. Step 3a is a mandatory step for all projects; Step 3b (this Skill) is only executed when the project involves APIs.
+
+## Trigger Conditions
+
+- User requests API orchestration test design
+- User mentions "Phase 3 Step 3b", "API orchestration", or "orchestration tests"
+- After Step 3a (test-writer) is complete, AI guides the user to proceed to Step 3b
+- User needs to validate deployed API code
+
+## Prerequisites
+
+- `logos/resources/test/` contains test case specification documents (Step 3a completed)
+- `logos/resources/prd/3-technical-plan/2-scenario-implementation/` contains scenario sequence diagrams
+- `logos/resources/api/` contains API specifications (OpenAPI YAML)
+- `logos-project.yaml` contains `external_dependencies` (if applicable)
+
+If the project does not involve APIs (pure CLI tools, pure frontend, etc.), skip this Skill.
+
+## Core Capabilities
+
+1. Design normal flow orchestration from sequence diagrams and API YAML
+2. Design exception flow orchestration based on exception cases (EX-N.M)
+3. Design boundary cases (valid but non-happy-path variations)
+4. Define variable extraction and passing mechanisms
+5. **Identify external dependencies and apply test strategies**: Read `external_dependencies` from `logos-project.yaml` and automatically insert `mock` fields in steps involving external services
+6. Execute orchestration and verify results
+
+## Execution Steps
+
+### Step 1: Read Scenario Context
+
+Read the following files to establish complete context:
+
+- Scenario sequence diagrams (`logos/resources/prd/3-technical-plan/2-scenario-implementation/`)
+- API YAML (`logos/resources/api/`)
+- `logos-project.yaml` — focus on reading the `external_dependencies` field
+
+### Step 2: Identify External Dependencies
+
+Match `used_in` from `external_dependencies` with the current scenario number. If the current scenario involves external dependencies:
+
+- Record the dependency's `test_strategy` and `test_config`
+- If a dependency declares `used_in` but is missing `test_strategy`, **proactively ask the user** for the test strategy
+
+If there is no `external_dependencies` field in `logos-project.yaml`, but the sequence diagrams contain calls to external services (e.g., sending emails, payment requests, etc.), proactively remind the user to add them.
+
+### Step 3: Design Normal Flow Orchestration
+
+Design the API call chain step by step following the sequence diagram's Step numbers:
+
+- Each step includes method, url, headers, body, expected_status
+- For steps involving external dependencies, insert the `mock` field (see Output Specification)
+- For variables that need to be passed from the previous step's response, use `extract` to define extraction rules
+
+### Step 4: Design Exception Flow Orchestration
+
+Design independent orchestrations for each EX exception case, ensuring:
+
+- Exception scenarios also cover external dependency failure cases
+- Use the `mock` field to simulate external service failures (e.g., timeouts, error responses, etc.)
+
+### Step 5: Design Boundary Case Orchestration
+
+Identify valid but non-happy-path variations (e.g., password length exactly at the boundary value, empty fields, etc.) and add supplementary orchestrations.
+
+### Step 6: Output Orchestration JSON
+
+Output executable orchestration JSON files per scenario.
+
+## Output Specification
+
+- File format: JSON
+- Storage location: `logos/resources/scenario/`
+- Separate files per scenario: `user-auth.json`, `payment-flow.json`
+- Each step in the orchestration corresponds to a Step number in the sequence diagram
+
+### mock Field Structure
+
+When a step involves an external dependency, add a `mock` field to that step:
+
+```json
+{
+  "step": "Step 2: Get email verification code",
+  "mock": {
+    "dependency": "Email Service",
+    "strategy": "test-api",
+    "config": "GET /api/test/latest-email?to={email}",
+    "extract": { "code": "response.body.code" }
+  },
+  "method": "GET",
+  "url": "/api/test/latest-email?to={{email}}",
+  "expected_status": 200,
+  "extract": {
+    "verification_code": "body.code"
+  }
+}
+```
+
+`mock` field description:
+
+| Field | Type | Description |
+|------|------|------|
+| `dependency` | string | Corresponds to `name` in `external_dependencies` |
+| `strategy` | string | Test strategy (`test-api` / `fixed-value` / `env-disable` / `mock-callback` / `mock-service`) |
+| `config` | string | Specific configuration for the test strategy, from `test_config` |
+| `extract` | object | Extract variables from mock response (optional) |
+
+Orchestration behavior for different strategies:
+
+- **`test-api`**: The step's url is replaced with the backdoor API address
+- **`fixed-value`**: The step does not make an actual request; fixed values are injected directly via `extract`
+- **`env-disable`**: The step is marked as skipped, with a comment explaining the precondition
+- **`mock-callback`**: An additional mock callback request is inserted after the previous step completes
+- **`mock-service`**: The step's url is replaced with the local mock service address
+
+## Best Practices
+
+- **Normal orchestration is the skeleton**: Complete the normal flow orchestration first to ensure the happy path works end-to-end
+- **Exception orchestration is the safety net**: At least 1 exception orchestration per external call
+- **Variable passing**: Extract variables from the previous step's response (e.g., token, user_id) and pass them to subsequent steps
+- **Test data**: Prepare test data before orchestration begins and clean up afterwards to ensure idempotency
+- **Concurrency testing**: Key scenarios should account for concurrent situations (e.g., two users registering with the same email simultaneously)
+- **Check the external dependency list first**: Before starting orchestration design, read `external_dependencies` from `logos-project.yaml`; proactively remind the user to add any undeclared external calls
+- **Do not decide mock strategies on your own**: Test strategies are determined during S12 technical architecture design (Phase 3 Step 0, architecture-designer); the orchestration test phase only consumes them — do not modify them unilaterally
+- **Relationship with `openlogos verify`**: API orchestration tests can also produce JSONL results in the same format as `spec/test-results.md`. After orchestration tests run, results are also written to `logos/resources/verify/test-results.jsonl`, and `openlogos verify` reads them uniformly to determine acceptance
+
+## Recommended Prompts
+
+The following prompts can be copied directly for AI use:
+
+- `Help me design orchestration tests`
+- `Generate orchestration tests for S01 based on the API spec`
+- `Help me orchestrate all normal paths for every scenario`
+- `Help me add exception path orchestration tests for S02`

package/skills/test-writer/SKILL.en.md

@@ -0,0 +1,247 @@
+# Skill: Test Writer
+
+> Based on sequence diagrams, API specifications, and DB constraints, design unit test cases and scenario test cases for each business scenario. Applicable to all project types (API services, CLI tools, frontend applications, libraries, etc.), this is a mandatory prerequisite step before code generation.
+
+## Trigger Conditions
+
+- User requests test case or test plan design
+- User mentions "Phase 3 Step 3", "Step 3a", "test-first", "test design"
+- Sequence diagrams already exist, and tests need to be designed before writing code
+- User specifies a scenario number (e.g., S01) that needs test design
+
+## Prerequisites
+
+- `logos/resources/prd/3-technical-plan/2-scenario-implementation/` contains sequence diagrams (**required**)
+- `logos/resources/api/` contains API specifications (read if present, skip if absent — non-API projects may not have these)
+- `logos/resources/database/` contains DB DDL (read if present, skip if absent)
+- `logos/resources/prd/1-product-requirements/` contains requirements documents (for tracing acceptance criteria)
+
+**Cannot be skipped**: Regardless of project type, Step 3a (this Skill) must be executed.
+
+## Core Capabilities
+
+1. Extract unit test cases from API field constraints (type, format, length, enum)
+2. Extract unit test cases from DB constraints (UNIQUE, CHECK, NOT NULL, FK)
+3. Extract unit test cases from business rules and single-point error handling in EX exception cases
+4. Extract scenario test cases from sequence diagram Step sequences (happy path)
+5. Extract scenario test cases from EX exception cases (exception paths)
+6. Reverse-validate test coverage completeness against Phase 1/2 acceptance criteria
+
+## Execution Steps
+
+### Step 1: Load Scenario Context
+
+Read the following files to establish complete context:
+
+- Sequence diagrams (`logos/resources/prd/3-technical-plan/2-scenario-implementation/`)
+- API YAML (`logos/resources/api/`) — if present
+- DB DDL (`logos/resources/database/`) — if present
+- Phase 1 requirements documents (acceptance criteria)
+- Phase 2 product design documents (interaction-level acceptance criteria)
+
+Confirm the following for the current scenario:
+- **Step count**: How many Steps are in the sequence diagram
+- **EX count**: How many exception cases exist
+- **API endpoints**: Which endpoints are involved and their field constraints
+- **DB tables**: Which tables are involved and their constraints
+
+### Step 2: Design Unit Test Cases
+
+Extract unit test cases from three categories of sources:
+
+#### 2a: API Field Constraints
+
+Inspect `requestBody` and `parameters` for each API endpoint:
+
+- `type` → Type error cases (passing incorrect types)
+- `format` (email, uuid, date-time) → Format validation cases
+- `minLength` / `maxLength` → Boundary value cases (exactly at limit, exceeding by 1)
+- `required` → Required field missing cases
+- `enum` → Enumeration value cases (valid values + invalid values)
+- `minimum` / `maximum` → Numeric range cases
+
+#### 2b: DB Constraints
+
+Inspect constraints for each related table:
+
+- `UNIQUE` → Duplicate insertion cases
+- `NOT NULL` → Null value insertion cases
+- `CHECK` → Constraint violation cases
+- `FOREIGN KEY` → Referencing non-existent record cases
+- `DEFAULT` → Default value verification when no value is provided
+
+#### 2c: Business Rules
+
+Extract single-point business logic from sequence diagram Step descriptions and EX exception cases:
+
+- Permission checks (not logged in, insufficient permissions)
+- State machine transitions (only specific states allow certain operations)
+- Rate limiting / throttling rules
+- Data computation logic (amount calculations, discount rules)
+
+**Format for each unit test case**:
+
+| Field | Description |
+|-------|-------------|
+| ID | `UT-{scenario-number}-{sequence}`, e.g., `UT-S01-01` |
+| Description | What behavior is being tested |
+| Source | Constraint origin (e.g., `auth.yaml → register → email: format:email`) |
+| Preconditions | State required before the test |
+| Input | Specific input values |
+| Expected Output | Expected return value or error message |
+
+### Step 3: Design Scenario Test Cases
+
+Extract scenario test cases from two categories of sources:
+
+#### 3a: Happy Path (Sequence Diagram Step Sequence)
+
+Treat the complete Step 1 → Step N sequence from the sequence diagram as an end-to-end code call chain:
+
+- Determine the scenario's entry and exit points
+- Annotate data passing between each Step (previous step's output as next step's input)
+- Verify the final state (database records, return values)
+
+#### 3b: Exception Paths (EX Exception Cases)
+
+Expand each EX exception case into a scenario test case:
+
+- Annotate which Step triggers the exception
+- Verify the handling logic after the exception is triggered (error response, compensation/rollback)
+- Verify the exception did not compromise the integrity of other data
+
+**Format for each scenario test case**:
+
+| Field | Description |
+|-------|-------------|
+| ID | `ST-{scenario-number}-{sequence}`, e.g., `ST-S01-01` |
+| Description | What scenario flow is being tested |
+| Covered Steps | Which sequence diagram Steps are covered (e.g., `Step 1→6`) or which EX (e.g., `EX-2.1`) |
+| Preconditions | State and data required before the test |
+| Operation Sequence | Ordered list of operations following Step sequence |
+| Expected Result | Final state (return value + database state + side effects) |
+
+### Step 4: Coverage Validation
+
+Reverse-validate whether test cases cover all critical constraints:
+
+- [ ] Each normal acceptance criterion from Phase 1 maps to at least 1 ST case
+- [ ] Each exception acceptance criterion from Phase 1 maps to at least 1 ST or UT case
+- [ ] Each EX exception case maps to at least 1 ST case
+- [ ] Each `required` field in the API has at least 1 UT case
+- [ ] Each `UNIQUE` / `CHECK` constraint in the DB has at least 1 UT case
+
+If any items are uncovered, add supplementary cases or explain the reason to the user.
+
+### Step 5: Acceptance Criteria Traceability
+
+Extract each GIVEN/WHEN/THEN acceptance criterion from the Phase 1 requirements document, assign a traceability ID to each, and link it to the test case IDs that cover that criterion.
+
+#### Acceptance Criteria ID Rules
+
+- Format: `{scenario-number}-AC-{two-digit-sequence}`, e.g., `S01-AC-01`, `S01-AC-02`
+- Numbered in the order they appear in the requirements document; normal and exception criteria use a unified numbering sequence
+- AC IDs within the same scenario must be consecutive and unique
+
+#### Traceability Table Rules
+
+1. Read all acceptance criteria (normal + exception) for the current scenario from the requirements document
+2. Assign an AC ID to each acceptance criterion
+3. Find the test case IDs that cover each criterion (can be UT or ST), and fill in the "Covered By" column
+4. Each AC must be linked to at least 1 test case; if it cannot be covered, note the reason in the "Covered By" column
+
+`openlogos verify` parses this traceability table and links AC → test case ID → execution result across three layers to generate a complete acceptance traceability report.
+
+### Step 6: Output Test Case Specification Document
+
+Output the test case specification document in Markdown format, organized by scenario.
+
+### Step 7: Guide Next Steps
+
+Guide the user to the next step based on project type:
+
+- **Involves API** → "Continue to Step 3b to design API orchestration tests?"
+- **Does not involve API** → "Test design is complete. Recommend proceeding to code generation: say 'Implement based on the S01 specification for me'"
+
+## Output Specification
+
+- **File format**: Markdown
+- **Location**: `logos/resources/test/`
+- **Naming convention**: `{scenario-number}-test-cases.md` (e.g., `S01-test-cases.md`)
+- Each file contains: Unit test cases (grouped by source) + Scenario test cases (happy path + exception paths)
+- Case IDs are globally unique: `UT-{scenario-number}-{sequence}` / `ST-{scenario-number}-{sequence}`
+
+### Document Structure Template
+
+```markdown
+# {scenario-number}: {scenario-name} — Test Cases
+
+## 1. Unit Test Cases
+
+### 1.1 {group-name} (Source: {constraint-origin})
+
+| ID | Description | Source | Preconditions | Input | Expected Output |
+|----|-------------|--------|---------------|-------|-----------------|
+| UT-S01-01 | ... | ... | ... | ... | ... |
+
+## 2. Scenario Test Cases
+
+### 2.1 Happy Path: {scenario-name}
+
+| ID | Description | Covered Steps | Preconditions | Operation Sequence | Expected Result |
+|----|-------------|---------------|---------------|--------------------|-----------------|
+| ST-S01-01 | ... | Step 1→6 | ... | ... | ... |
+
+### 2.2 Exception Paths
+
+| ID | Description | Covered EX | Preconditions | Trigger Condition | Expected Result |
+|----|-------------|------------|---------------|-------------------|-----------------|
+| ST-S01-02 | ... | EX-2.1 | ... | ... | ... |
+
+## 3. Coverage Validation
+
+- [x] Phase 1 normal acceptance criteria: fully covered
+- [x] Phase 1 exception acceptance criteria: fully covered
+- [x] EX exception cases: fully covered
+- [x] API required fields: fully covered
+- [x] DB UNIQUE/CHECK constraints: fully covered
+
+## 4. Acceptance Criteria Traceability
+
+| AC ID | Acceptance Criterion | Covered By |
+|-------|----------------------|------------|
+| S01-AC-01 | Normal: Fresh project initialization — create complete directory structure | ST-S01-01 |
+| S01-AC-02 | Normal: Confirm when explicit project name differs from config file | ST-S01-02 |
+| S01-AC-03 | Exception: Project already initialized — display error message | ST-S01-03, UT-S01-05 |
+```
+
+## Test Case ID Contract
+
+Test case IDs (`UT-S01-01`, `ST-S01-01`) serve as a **binding contract** between design documents and runtime:
+
+- IDs defined in test-cases.md must be used as-is in the generated test code
+- The test code reporter writes each case's ID and execution result to a JSONL file
+- `openlogos verify` maps execution results back to test case specifications via IDs, automatically determining acceptance
+- When modifying case IDs, the corresponding IDs in the test code must be updated simultaneously
+
+See `spec/test-results.md` for the detailed JSONL format definition and reporter code templates for each language.
+
+## Best Practices
+
+- **Test cases are design documents, not code**: This Skill produces test case specifications in Markdown format; the actual test code is implemented by AI during Step 4 code generation based on these specifications
+- **Unit first, then scenario**: Unit test cases cover the correctness of individual functions; scenario tests cover cross-module integration — first ensure the building blocks are correct, then verify they fit together
+- **Don't overlook DB constraints**: Many bugs originate from database-level constraint violations; DB constraints are an important source of unit test cases
+- **Scenario tests focus on data passing**: Data passing between Steps (previous step's output → next step's input) is where errors most commonly occur
+- **EX exception cases must have corresponding scenario tests**: Every EX annotated in the sequence diagram should be reflected in scenario tests
+- **Boundary values first**: Unit test cases should prioritize boundary values (just valid, just invalid) over random values
+- **Complementary with test-orchestrator**: This Skill designs code-level tests (function call level); test-orchestrator designs API-level tests (HTTP request level). Together they cover different layers of the "testing pyramid"
+- **Case IDs are cross-phase contracts**: IDs span test-cases.md → test code → test-results.jsonl → acceptance-report.md; any inconsistency will cause `openlogos verify` to report incomplete results
+
+## Recommended Prompts
+
+The following prompts can be copied directly for AI use:
+
+- `Design test cases for me`
+- `Design unit tests and scenario tests for S01`
+- `Design test cases for all P0 scenarios`
+- `Check the test coverage for S01`