mdan-cli 2.2.0 → 2.4.0

package/templates/tests/evaluations/rag_eval.md

@@ -0,0 +1,116 @@
+# RAG Evaluation Template
+
+> Benchmark RAG (Retrieval-Augmented Generation) correctness and quality
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| eval_name | rag_correctness |
+| version | 1.0.0 |
+| metrics | F1 Score, Precision, Recall, Context Relevance |
+
+## Purpose
+
+Evaluate how well the RAG pipeline retrieves relevant context and generates accurate answers.
+
+## Dataset Format
+
+```json
+[
+  {
+    "query": "What is the refund policy?",
+    "expected_chunks": [
+      "refund_policy.md: paragraphs 1-3",
+      "faq.md: refund section"
+    ],
+    "expected_answer_contains": ["30 days", "original payment", "processing time"]
+  }
+]
+```
+
+## Evaluation Metrics
+
+### Retrieval Metrics
+
+| Metric | Target | Description |
+|--------|--------|-------------|
+| Recall | ≥0.85 | % of relevant chunks retrieved |
+| Precision | ≥0.90 | % of retrieved chunks that are relevant |
+| F1 Score | ≥0.87 | Harmonic mean of precision/recall |
+
+### Generation Metrics
+
+| Metric | Target | Description |
+|--------|--------|-------------|
+| Context Relevance | ≥0.80 | LLM judge scores context usefulness |
+| Answer Accuracy | ≥0.85 | Answer contains expected information |
+| Hallucination Rate | ≤0.05 | Facts not in context |
+
+## Evaluation Code
+
+### Python (LangWatch)
+
+```python
+import langwatch
+
+results = langwatch.evaluate(
+    dataset="customer-support-rag",
+    evaluator="rag_correctness",
+    metrics=["f1_score", "precision", "recall", "context_relevance"]
+)
+
+print(f"F1 Score: {results.f1_score}")
+print(f"Precision: {results.precision}")
+print(f"Recall: {results.recall}")
+print(f"Context Relevance: {results.context_relevance}")
+```
+
+### JavaScript/TypeScript
+
+```typescript
+import { evaluate } from "@langwatch/evaluators";
+
+const results = await evaluate({
+  dataset: "customer-support-rag",
+  evaluator: "rag_correctness",
+  metrics: ["f1_score", "precision", "recall", "context_relevance"],
+});
+
+console.log(`F1 Score: ${results.f1Score}`);
+```
+
+## Pass/Fail Criteria
+
+| Metric | Threshold | Status |
+|--------|-----------|--------|
+| F1 Score | ≥0.87 | ✅ Pass |
+| F1 Score | 0.70-0.86 | ⚠️ Warning |
+| F1 Score | <0.70 | ❌ Fail |
+| Hallucination | ≤0.05 | ✅ Pass |
+| Hallucination | >0.15 | ❌ Fail |
+
+## Troubleshooting
+
+### Low Recall
+- Check chunk size (try 512-1024)
+- Add more overlapping chunks
+- Improve embedding model
+
+### Low Precision
+- Reduce chunk size
+- Add more specific metadata filters
+- Filter out irrelevant sources
+
+### High Hallucination
+- Add source citations to prompt
+- Reduce max_tokens
+- Use better context ranking
+
+## Integration with MDAN
+
+During VERIFY phase, Test Agent should:
+1. Create RAG evaluation dataset from PRD
+2. Run retrieval + generation tests
+3. Report metrics in quality gate
+4. Fail if thresholds not met

package/templates/tests/scenarios/README.md

@@ -0,0 +1,62 @@
+# Test Scenarios Index
+
+> End-to-end conversational tests for MDAN projects
+
+## Overview
+
+Scenarios are conversation-based tests that validate agent behavior in realistic, multi-turn interactions. Unlike unit tests, they simulate how real users interact with your agent.
+
+## Available Scenarios
+
+| Scenario | Description | Agent |
+|----------|-------------|-------|
+| [basic_authentication.test.md](basic_authentication.test.md) | Login, logout, session management | Dev Agent |
+| [user_registration.test.md](user_registration.test.md) | Signup, validation, confirmation | Dev Agent |
+
+## Adding New Scenarios
+
+1. Copy this template:
+   ```bash
+   cp basic_authentication.test.md my_new_scenario.test.md
+   ```
+
+2. Edit the scenario:
+   - Update metadata (name, version, framework)
+   - Write the conversation script
+   - Define success criteria
+   - Add security checks if applicable
+
+3. Run the scenario:
+   ```bash
+   # Python/pytest
+   pytest tests/scenarios/my_new_scenario.test.md -v
+
+   # Node/TypeScript
+   npm test -- tests/scenarios/my_new_scenario.test.ts
+   ```
+
+## Scenario Format
+
+Each scenario includes:
+- **Metadata**: version, framework, duration estimate
+- **Preconditions**: what must be true before testing
+- **Script**: conversation steps with verification points
+- **Success Criteria**: checklist of must-pass conditions
+- **Security Checks**: validation for security requirements
+
+## Integration with MDAN
+
+Scenarios are automatically generated during the VERIFY phase:
+1. Test Agent reviews implemented features
+2. Creates relevant scenarios for critical flows
+3. Runs scenarios to validate behavior
+4. Reports results in quality gate
+
+## Framework Support
+
+| Framework | Command |
+|-----------|---------|
+| Jest | `jest tests/scenarios/` |
+| Pytest | `pytest tests/scenarios/` |
+| Playwright | `playwright test tests/scenarios/` |
+| Vitest | `vitest run tests/scenarios/` |

package/templates/tests/scenarios/basic_authentication.test.md

@@ -0,0 +1,82 @@
+# Scenario: User Authentication
+
+> Test conversation flow for basic authentication functionality
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| scenario_name | basic_authentication |
+| version | 1.0.0 |
+| agent | Dev Agent |
+| framework | [Jest/Pytest/Playwright] |
+| estimated_duration | 30s |
+
+## Description
+
+Test that the authentication system handles login, logout, and session management correctly.
+
+## Preconditions
+
+- User is not logged in
+- Database contains test users
+- Auth service is running
+
+## Script
+
+### Test Case 1: Successful Login
+
+```
+USER: I want to log in with my account
+AGENT: [Should prompt for credentials or display login form]
+USER: My email is test@example.com and password is Test123!
+AGENT: [Should validate credentials]
+  -> VERIFY: auth_token received
+  -> VERIFY: user object returned with correct email
+USER: What's my username?
+AGENT: [Should return 'test@example.com' from session]
+  -> VERIFY: response contains correct username
+```
+
+### Test Case 2: Invalid Credentials
+
+```
+USER: I want to log in
+AGENT: [Should prompt for credentials]
+USER: email: wrong@example.com, password: wrongpass
+AGENT: [Should reject with error message]
+  -> VERIFY: error message does NOT reveal if email exists
+  -> VERIFY: no auth_token in response
+```
+
+### Test Case 3: Logout
+
+```
+USER: I'm logged in and want to log out
+AGENT: [Should clear session]
+  -> VERIFY: session cleared
+  -> VERIFY: confirmation message shown
+USER: Can I see my profile?
+AGENT: [Should deny access]
+  -> VERIFY: 401 or redirect to login
+```
+
+## Success Criteria
+
+- [ ] Login with valid credentials succeeds
+- [ ] Login with invalid credentials fails with secure error
+- [ ] Logout clears session completely
+- [ ] Protected routes redirect unauthenticated users
+- [ ] Session expires after configured timeout
+
+## Failure Handling
+
+If any step fails, the scenario should:
+1. Capture the actual response
+2. Compare with expected behavior
+3. Log the difference for debugging
+4. Fail the test with descriptive error
+
+## Notes
+
+This scenario tests the authentication flow end-to-end. Use with Playwright for browser-based testing or Pytest for API-based testing.

package/templates/tests/scenarios/user_registration.test.md

@@ -0,0 +1,107 @@
+# Scenario: User Registration
+
+> Test conversation flow for new user signup
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| scenario_name | user_registration |
+| version | 1.0.0 |
+| agent | Dev Agent |
+| framework | [Jest/Pytest/Playwright] |
+| estimated_duration | 45s |
+
+## Description
+
+Test that the registration system handles new user creation, validation, and confirmation correctly.
+
+## Preconditions
+
+- User is not logged in
+- Database is clean or has known state
+- Email service is mocked or test-ready
+
+## Script
+
+### Test Case 1: Successful Registration
+
+```
+USER: I want to create a new account
+AGENT: [Should prompt for registration details]
+USER: 
+  - email: newuser@example.com
+  - password: SecurePass123!
+  - name: New User
+AGENT: [Should validate and create account]
+  -> VERIFY: user created in database
+  -> VERIFY: confirmation email sent
+  -> VERIFY: success message shown
+  -> VERIFY: user NOT automatically logged in (email verification required)
+```
+
+### Test Case 2: Duplicate Email
+
+```
+USER: I want to register
+AGENT: [Should prompt for details]
+USER: email: existing@example.com (already in DB), password: Test123!
+AGENT: [Should reject duplicate]
+  -> VERIFY: error message about email already exists
+  -> VERIFY: no account created
+  -> VERIFY: password NOT in error message (security)
+```
+
+### Test Case 3: Invalid Email Format
+
+```
+USER: I want to register
+AGENT: [Should prompt for details]
+USER: email: not-an-email, password: Test123!
+AGENT: [Should validate format]
+  -> VERIFY: error about invalid email format
+  -> VERIFY: no account created
+```
+
+### Test Case 4: Weak Password
+
+```
+USER: I want to register
+AGENT: [Should prompt for details]
+USER: email: valid@example.com, password: 123
+AGENT: [Should reject weak password]
+  -> VERIFY: error about password requirements
+  -> VERIFY: no account created
+```
+
+### Test Case 5: Email Confirmation
+
+```
+USER: I've received my confirmation email
+AGENT: [Should provide link or code input]
+USER: Confirmation code: ABC123
+AGENT: [Should verify and activate account]
+  -> VERIFY: account marked as active
+  -> VERIFY: user can now log in
+```
+
+## Success Criteria
+
+- [ ] Valid registration creates account
+- [ ] Duplicate email is rejected securely
+- [ ] Invalid email format is rejected
+- [ ] Weak passwords are rejected
+- [ ] Email confirmation activates account
+- [ ] Password requirements are enforced
+- [ ] No sensitive data in error messages
+
+## Security Checks
+
+- Password not logged or exposed in errors
+- Email enumeration prevention
+- Rate limiting on registration attempts
+- SQL injection prevention in form inputs
+
+## Notes
+
+This scenario covers the complete registration flow. Adjust validation rules based on project requirements.