mdan-cli 2.5.1 → 2.5.2

package/templates/tests/evaluations/classification_eval.md

@@ -1,136 +0,0 @@
-# Classification Evaluation Template
-
-> Benchmark routing accuracy and categorization correctness
-
-## Metadata
-
-| Field | Value |
-|-------|-------|
-| eval_name | classification_accuracy |
-| version | 1.0.0 |
-| metrics | Accuracy, Precision, Recall, F1 Score |
-
-## Purpose
-
-Evaluate how well the agent correctly routes requests, categorizes inputs, or makes routing decisions.
-
-## Use Cases
-
-- Intent classification (chatbot routing)
-- Content moderation
-- Priority/ticket categorization
-- Language detection
-- Sentiment analysis
-
-## Dataset Format
-
-```json
-[
-  {
-    "input": "I want to get a refund",
-    "expected_category": "refund_request",
-    "expected_confidence": 0.90
-  },
-  {
-    "input": "How do I change my password?",
-    "expected_category": "account_settings",
-    "expected_confidence": 0.85
-  }
-]
-```
-
-## Evaluation Metrics
-
-| Metric | Target | Description |
-|--------|--------|-------------|
-| Accuracy | ≥0.95 | % of correct classifications |
-| Macro F1 | ≥0.90 | F1 averaged across all categories |
-| Precision (per class) | ≥0.85 | True positives / predicted positives |
-| Recall (per class) | ≥0.85 | True positives / actual positives |
-| Confidence Calibration | ≤0.10 | Mean absolute error between confidence and accuracy |
-
-## Evaluation Code
-
-### Python (LangWatch)
-
-```python
-import langwatch
-
-results = langwatch.evaluate(
-    dataset="intent-classification",
-    evaluator="classification_accuracy",
-    metrics=["accuracy", "precision", "recall", "f1_macro", "calibration"]
-)
-
-print(f"Accuracy: {results.accuracy}")
-print(f"Macro F1: {results.f1_macro}")
-print(f"Precision: {results.precision}")
-print(f"Recall: {results.recall}")
-```
-
-### JavaScript/TypeScript
-
-```typescript
-import { evaluate } from "@langwatch/evaluators";
-
-const results = await evaluate({
-  dataset: "intent-classification",
-  evaluator: "classification_accuracy",
-  metrics: ["accuracy", "precision", "recall", "f1_macro"],
-});
-
-console.log(`Accuracy: ${results.accuracy}`);
-```
-
-## Pass/Fail Criteria
-
-| Metric | Threshold | Status |
-|--------|-----------|--------|
-| Accuracy | ≥0.95 | ✅ Pass |
-| Accuracy | 0.90-0.94 | ⚠️ Warning |
-| Accuracy | <0.90 | ❌ Fail |
-| Macro F1 | ≥0.90 | ✅ Pass |
-| Macro F1 | <0.85 | ❌ Fail |
-| Confidence Error | ≤0.10 | ✅ Pass |
-| Confidence Error | >0.15 | ⚠️ Warning |
-
-## Per-Class Analysis
-
-Generate confusion matrix:
-
-|  | Predicted A | Predicted B | Predicted C |
-|--|-------------|-------------|-------------|
-| Actual A | 45 | 3 | 2 |
-| Actual B | 5 | 38 | 7 |
-| Actual C | 1 | 4 | 40 |
-
-Identify:
-- **High-confusion pairs**: A↔B need better differentiation
-- **Low-recall classes**: More training data needed
-- **Low-precision classes**: Overlapping with other categories
-
-## Common Issues
-
-### Low Precision (many false positives)
-- Add negative examples
-- Make categories more distinct
-- Add disambiguation prompts
-
-### Low Recall (many false negatives)
-- Add more training data
-- Expand category definitions
-- Check for data quality issues
-
-### Poor Calibration
-- Retrain with temperature scaling
-- Add more diverse examples
-- Use calibration-aware loss
-
-## Integration with MDAN
-
-During VERIFY phase, Test Agent should:
-1. Identify all classification/routing points in the system
-2. Create evaluation datasets from real user queries
-3. Run classification evaluation
-4. Report per-class performance
-5. Fail if overall accuracy < 90%

package/templates/tests/evaluations/rag_eval.md

@@ -1,116 +0,0 @@
-# RAG Evaluation Template
-
-> Benchmark RAG (Retrieval-Augmented Generation) correctness and quality
-
-## Metadata
-
-| Field | Value |
-|-------|-------|
-| eval_name | rag_correctness |
-| version | 1.0.0 |
-| metrics | F1 Score, Precision, Recall, Context Relevance |
-
-## Purpose
-
-Evaluate how well the RAG pipeline retrieves relevant context and generates accurate answers.
-
-## Dataset Format
-
-```json
-[
-  {
-    "query": "What is the refund policy?",
-    "expected_chunks": [
-      "refund_policy.md: paragraphs 1-3",
-      "faq.md: refund section"
-    ],
-    "expected_answer_contains": ["30 days", "original payment", "processing time"]
-  }
-]
-```
-
-## Evaluation Metrics
-
-### Retrieval Metrics
-
-| Metric | Target | Description |
-|--------|--------|-------------|
-| Recall | ≥0.85 | % of relevant chunks retrieved |
-| Precision | ≥0.90 | % of retrieved chunks that are relevant |
-| F1 Score | ≥0.87 | Harmonic mean of precision/recall |
-
-### Generation Metrics
-
-| Metric | Target | Description |
-|--------|--------|-------------|
-| Context Relevance | ≥0.80 | LLM judge scores context usefulness |
-| Answer Accuracy | ≥0.85 | Answer contains expected information |
-| Hallucination Rate | ≤0.05 | Facts not in context |
-
-## Evaluation Code
-
-### Python (LangWatch)
-
-```python
-import langwatch
-
-results = langwatch.evaluate(
-    dataset="customer-support-rag",
-    evaluator="rag_correctness",
-    metrics=["f1_score", "precision", "recall", "context_relevance"]
-)
-
-print(f"F1 Score: {results.f1_score}")
-print(f"Precision: {results.precision}")
-print(f"Recall: {results.recall}")
-print(f"Context Relevance: {results.context_relevance}")
-```
-
-### JavaScript/TypeScript
-
-```typescript
-import { evaluate } from "@langwatch/evaluators";
-
-const results = await evaluate({
-  dataset: "customer-support-rag",
-  evaluator: "rag_correctness",
-  metrics: ["f1_score", "precision", "recall", "context_relevance"],
-});
-
-console.log(`F1 Score: ${results.f1Score}`);
-```
-
-## Pass/Fail Criteria
-
-| Metric | Threshold | Status |
-|--------|-----------|--------|
-| F1 Score | ≥0.87 | ✅ Pass |
-| F1 Score | 0.70-0.86 | ⚠️ Warning |
-| F1 Score | <0.70 | ❌ Fail |
-| Hallucination | ≤0.05 | ✅ Pass |
-| Hallucination | >0.15 | ❌ Fail |
-
-## Troubleshooting
-
-### Low Recall
-- Check chunk size (try 512-1024)
-- Add more overlapping chunks
-- Improve embedding model
-
-### Low Precision
-- Reduce chunk size
-- Add more specific metadata filters
-- Filter out irrelevant sources
-
-### High Hallucination
-- Add source citations to prompt
-- Reduce max_tokens
-- Use better context ranking
-
-## Integration with MDAN
-
-During VERIFY phase, Test Agent should:
-1. Create RAG evaluation dataset from PRD
-2. Run retrieval + generation tests
-3. Report metrics in quality gate
-4. Fail if thresholds not met

package/templates/tests/scenarios/README.md

@@ -1,62 +0,0 @@
-# Test Scenarios Index
-
-> End-to-end conversational tests for MDAN projects
-
-## Overview
-
-Scenarios are conversation-based tests that validate agent behavior in realistic, multi-turn interactions. Unlike unit tests, they simulate how real users interact with your agent.
-
-## Available Scenarios
-
-| Scenario | Description | Agent |
-|----------|-------------|-------|
-| [basic_authentication.test.md](basic_authentication.test.md) | Login, logout, session management | Dev Agent |
-| [user_registration.test.md](user_registration.test.md) | Signup, validation, confirmation | Dev Agent |
-
-## Adding New Scenarios
-
-1. Copy this template:
-   ```bash
-   cp basic_authentication.test.md my_new_scenario.test.md
-   ```
-
-2. Edit the scenario:
-   - Update metadata (name, version, framework)
-   - Write the conversation script
-   - Define success criteria
-   - Add security checks if applicable
-
-3. Run the scenario:
-   ```bash
-   # Python/pytest
-   pytest tests/scenarios/my_new_scenario.test.md -v
-
-   # Node/TypeScript
-   npm test -- tests/scenarios/my_new_scenario.test.ts
-   ```
-
-## Scenario Format
-
-Each scenario includes:
-- **Metadata**: version, framework, duration estimate
-- **Preconditions**: what must be true before testing
-- **Script**: conversation steps with verification points
-- **Success Criteria**: checklist of must-pass conditions
-- **Security Checks**: validation for security requirements
-
-## Integration with MDAN
-
-Scenarios are automatically generated during the VERIFY phase:
-1. Test Agent reviews implemented features
-2. Creates relevant scenarios for critical flows
-3. Runs scenarios to validate behavior
-4. Reports results in quality gate
-
-## Framework Support
-
-| Framework | Command |
-|-----------|---------|
-| Jest | `jest tests/scenarios/` |
-| Pytest | `pytest tests/scenarios/` |
-| Playwright | `playwright test tests/scenarios/` |
-| Vitest | `vitest run tests/scenarios/` |

package/templates/tests/scenarios/basic_authentication.test.md

@@ -1,82 +0,0 @@
-# Scenario: User Authentication
-
-> Test conversation flow for basic authentication functionality
-
-## Metadata
-
-| Field | Value |
-|-------|-------|
-| scenario_name | basic_authentication |
-| version | 1.0.0 |
-| agent | Dev Agent |
-| framework | [Jest/Pytest/Playwright] |
-| estimated_duration | 30s |
-
-## Description
-
-Test that the authentication system handles login, logout, and session management correctly.
-
-## Preconditions
-
-- User is not logged in
-- Database contains test users
-- Auth service is running
-
-## Script
-
-### Test Case 1: Successful Login
-
-```
-USER: I want to log in with my account
-AGENT: [Should prompt for credentials or display login form]
-USER: My email is test@example.com and password is Test123!
-AGENT: [Should validate credentials]
-  -> VERIFY: auth_token received
-  -> VERIFY: user object returned with correct email
-USER: What's my username?
-AGENT: [Should return 'test@example.com' from session]
-  -> VERIFY: response contains correct username
-```
-
-### Test Case 2: Invalid Credentials
-
-```
-USER: I want to log in
-AGENT: [Should prompt for credentials]
-USER: email: wrong@example.com, password: wrongpass
-AGENT: [Should reject with error message]
-  -> VERIFY: error message does NOT reveal if email exists
-  -> VERIFY: no auth_token in response
-```
-
-### Test Case 3: Logout
-
-```
-USER: I'm logged in and want to log out
-AGENT: [Should clear session]
-  -> VERIFY: session cleared
-  -> VERIFY: confirmation message shown
-USER: Can I see my profile?
-AGENT: [Should deny access]
-  -> VERIFY: 401 or redirect to login
-```
-
-## Success Criteria
-
-- [ ] Login with valid credentials succeeds
-- [ ] Login with invalid credentials fails with secure error
-- [ ] Logout clears session completely
-- [ ] Protected routes redirect unauthenticated users
-- [ ] Session expires after configured timeout
-
-## Failure Handling
-
-If any step fails, the scenario should:
-1. Capture the actual response
-2. Compare with expected behavior
-3. Log the difference for debugging
-4. Fail the test with descriptive error
-
-## Notes
-
-This scenario tests the authentication flow end-to-end. Use with Playwright for browser-based testing or Pytest for API-based testing.

package/templates/tests/scenarios/user_registration.test.md

@@ -1,107 +0,0 @@
-# Scenario: User Registration
-
-> Test conversation flow for new user signup
-
-## Metadata
-
-| Field | Value |
-|-------|-------|
-| scenario_name | user_registration |
-| version | 1.0.0 |
-| agent | Dev Agent |
-| framework | [Jest/Pytest/Playwright] |
-| estimated_duration | 45s |
-
-## Description
-
-Test that the registration system handles new user creation, validation, and confirmation correctly.
-
-## Preconditions
-
-- User is not logged in
-- Database is clean or has known state
-- Email service is mocked or test-ready
-
-## Script
-
-### Test Case 1: Successful Registration
-
-```
-USER: I want to create a new account
-AGENT: [Should prompt for registration details]
-USER: 
-  - email: newuser@example.com
-  - password: SecurePass123!
-  - name: New User
-AGENT: [Should validate and create account]
-  -> VERIFY: user created in database
-  -> VERIFY: confirmation email sent
-  -> VERIFY: success message shown
-  -> VERIFY: user NOT automatically logged in (email verification required)
-```
-
-### Test Case 2: Duplicate Email
-
-```
-USER: I want to register
-AGENT: [Should prompt for details]
-USER: email: existing@example.com (already in DB), password: Test123!
-AGENT: [Should reject duplicate]
-  -> VERIFY: error message about email already exists
-  -> VERIFY: no account created
-  -> VERIFY: password NOT in error message (security)
-```
-
-### Test Case 3: Invalid Email Format
-
-```
-USER: I want to register
-AGENT: [Should prompt for details]
-USER: email: not-an-email, password: Test123!
-AGENT: [Should validate format]
-  -> VERIFY: error about invalid email format
-  -> VERIFY: no account created
-```
-
-### Test Case 4: Weak Password
-
-```
-USER: I want to register
-AGENT: [Should prompt for details]
-USER: email: valid@example.com, password: 123
-AGENT: [Should reject weak password]
-  -> VERIFY: error about password requirements
-  -> VERIFY: no account created
-```
-
-### Test Case 5: Email Confirmation
-
-```
-USER: I've received my confirmation email
-AGENT: [Should provide link or code input]
-USER: Confirmation code: ABC123
-AGENT: [Should verify and activate account]
-  -> VERIFY: account marked as active
-  -> VERIFY: user can now log in
-```
-
-## Success Criteria
-
-- [ ] Valid registration creates account
-- [ ] Duplicate email is rejected securely
-- [ ] Invalid email format is rejected
-- [ ] Weak passwords are rejected
-- [ ] Email confirmation activates account
-- [ ] Password requirements are enforced
-- [ ] No sensitive data in error messages
-
-## Security Checks
-
-- Password not logged or exposed in errors
-- Email enumeration prevention
-- Rate limiting on registration attempts
-- SQL injection prevention in form inputs
-
-## Notes
-
-This scenario covers the complete registration flow. Adjust validation rules based on project requirements.