mdan-cli 2.2.0 → 2.3.0

package/templates/prompts/orchestrator.yaml

@@ -0,0 +1,97 @@
+handle: orchestrator
+scope: PROJECT
+model: openai/gpt-4o
+version: 2.2.0
+last_updated: "2026-02-24"
+maintainer: khalilbenaz
+
+description: |
+  MDAN Core v2 - Adaptive central orchestrator that manages the entire
+  development lifecycle. Handles project profiling, session memory,
+  and agent coordination.
+
+system_prompt: |
+  You are MDAN Core v2, the adaptive central orchestrator of the MDAN method.
+
+  You do everything MDAN Core v1 does, plus two critical upgrades:
+
+  1. PROJECT PROFILE DETECTION — You detect the type of project early in DISCOVER 
+     and automatically adapt the agent workflow, phase depth, and artifact requirements.
+
+  2. SESSION MEMORY — You can resume any project from a MDAN-STATE.json file, 
+     reconstructing full context and picking up exactly where work left off.
+
+  Your responsibilities:
+  - Guide user through 5 phases: DISCOVER, DESIGN, BUILD, VERIFY, SHIP
+  - Activate appropriate agents based on project profile
+  - Maintain session state in MDAN-STATE.json
+  - Ensure quality gates are met before phase transitions
+  - Support slash commands: /mdan-help, /party
+
+  Project profiles you can detect:
+  - MICRO: Solo dev, <1 week, single feature
+  - STANDARD: Small team, 2-8 weeks (default)
+  - PRODUCT: Team 5+, multi-month, stakeholders
+  - ENTERPRISE: Large team, compliance, SLA
+  - API-ONLY: No frontend, developer-focused
+  - DATA/ML: Machine learning, data pipelines
+
+constraints: |
+  - Never skip quality gates
+  - Always ask for user name in first interaction
+  - Use agent names: Khalil (Product), Reda (Architect), Jihane (UX),
+    Haytame (Dev), Youssef (Test), Said (Security), Anas (DevOps), Amina (Doc)
+  - Log all agent activations with version numbers
+
+output_format: |
+  Always include these in your responses:
+  - Current phase indicator
+  - Next recommended action
+  - Quality gate status
+
+examples:
+  - scenario: Start new project
+    user: "I want to build a SaaS app"
+    response: |
+      [MDAN CORE — PROJECT PROFILE DETECTED]
+      
+      Based on your description, I'm classifying this as a STANDARD project.
+      
+      This means:
+      - Full 5-phase workflow
+      - All agents will be activated
+      - Standard artifact depth
+      
+      Does this feel right? Also, before we proceed, what is your name?
+
+  - scenario: Resume session
+    user: "Here's my MDAN-STATE.json"
+    response: |
+      [MDAN CORE — SESSION RESUMED]
+      
+      📁 Project: my-saas (STANDARD)
+      📍 Current Phase: DESIGN
+      ...
+      
+      → Recommended next action: Complete Architecture document
+      
+      Shall I continue?
+
+changelog:
+  - version: 2.2.0
+    date: "2026-02-24"
+    changes:
+      - Added project profile detection
+      - Added session resume protocol
+      - Added agent version tracking
+      - Added quality gates for each profile
+  - version: 2.1.0
+    date: "2025-12-01"
+    changes:
+      - Added /party command
+      - Improved /mdan-help
+  - version: 2.0.0
+    date: "2025-10-01"
+    changes:
+      - Initial v2 release
+      - 5-phase workflow

package/templates/prompts.json

@@ -0,0 +1,81 @@
+{
+  "registry_version": "1.0",
+  "last_updated": "2026-02-24",
+  "prompts": {
+    "orchestrator": {
+      "handle": "orchestrator",
+      "version": "2.2.0",
+      "file": "templates/prompts/orchestrator.yaml",
+      "active": true,
+      "model": "openai/gpt-4o",
+      "changelog": [
+        {
+          "version": "2.2.0",
+          "date": "2026-02-24",
+          "changes": ["Added project profile detection", "Added session resume protocol"]
+        }
+      ]
+    },
+    "dev-agent": {
+      "handle": "dev-agent",
+      "version": "2.0.0",
+      "file": "templates/prompts/dev-agent.yaml",
+      "active": true,
+      "model": "openai/gpt-4o",
+      "changelog": [
+        {
+          "version": "2.0.0",
+          "date": "2026-02-24",
+          "changes": ["Added implementation plan format", "Added setup instructions"]
+        }
+      ]
+    },
+    "product-agent": {
+      "handle": "product-agent",
+      "version": "2.0.0",
+      "file": "templates/prompts/product-agent.yaml",
+      "active": true,
+      "model": "openai/gpt-4o",
+      "changelog": [
+        {
+          "version": "2.0.0",
+          "date": "2026-02-24",
+          "changes": ["Initial release with MoSCoW, risk matrix"]
+        }
+      ]
+    },
+    "architect-agent": {
+      "handle": "architect-agent",
+      "version": "2.0.0",
+      "file": "templates/prompts/architect-agent.yaml",
+      "active": true,
+      "model": "openai/gpt-4o",
+      "changelog": [
+        {
+          "version": "2.0.0",
+          "date": "2026-02-24",
+          "changes": ["Added Mermaid diagrams", "Added ADR format"]
+        }
+      ]
+    },
+    "test-agent": {
+      "handle": "test-agent",
+      "version": "2.0.0",
+      "file": "templates/prompts/test-agent.yaml",
+      "active": true,
+      "model": "openai/gpt-4o",
+      "changelog": [
+        {
+          "version": "2.0.0",
+          "date": "2026-02-24",
+          "changes": ["Added scenarios support", "Added evaluations"]
+        }
+      ]
+    }
+  },
+  "sync_settings": {
+    "auto_sync": false,
+    "langwatch_integration": false,
+    "commit_on_change": true
+  }
+}

package/templates/tests/evaluations/README.md

@@ -0,0 +1,80 @@
+# Test Evaluations Index
+
+> Structured benchmarking for agent components
+
+## Overview
+
+Evaluations provide quantitative testing for specific components of your agent pipeline. Unlike scenarios (end-to-end), evaluations test isolated components.
+
+## Available Evaluations
+
+| Evaluation | Description | Metrics |
+|------------|-------------|---------|
+| [rag_eval.md](rag_eval.md) | RAG correctness | F1, Precision, Recall, Context Relevance |
+| [classification_eval.md](classification_eval.md) | Routing/categorization | Accuracy, F1, Precision, Recall |
+
+## Adding New Evaluations
+
+1. Copy an existing template:
+   ```bash
+   cp rag_eval.md my_eval.md
+   ```
+
+2. Customize:
+   - Update dataset format
+   - Set target thresholds
+   - Add domain-specific checks
+
+3. Run:
+   ```python
+   import langwatch
+   results = langwatch.evaluate(
+       dataset="my-dataset",
+       evaluator="my_eval",
+       metrics=["accuracy", "f1"]
+   )
+   ```
+
+## Built-in Evaluators
+
+LangWatch provides extensive evaluators:
+
+| Evaluator | Description |
+|-----------|-------------|
+| `rag_correctness` | Retrieval and generation quality |
+| `classification_accuracy` | Routing and categorization |
+| `answer_correctness` | Factual accuracy |
+| `safety_check` | Jailbreak, PII, toxicity |
+| `format_validation` | JSON, XML, markdown structure |
+| `tool_calling` | Correct tool selection and args |
+| `latency` | Response time benchmarking |
+
+## Running Evaluations
+
+```bash
+# Python
+pytest tests/evaluations/ -v
+
+# JavaScript
+npm test -- tests/evaluations/
+
+# With LangWatch
+langwatch evaluate --dataset my-project
+```
+
+## Integration with MDAN
+
+During VERIFY phase:
+1. Test Agent identifies components needing evaluation
+2. Creates evaluation datasets from PRD/user stories
+3. Runs relevant evaluations
+4. Reports metrics in quality gate
+5. Fails if thresholds not met
+
+## Best Practices
+
+- Create evaluation datasets from real user queries
+- Include edge cases and negative examples
+- Set realistic thresholds (not 100%)
+- Track metrics over time (regression detection)
+- Run evaluations in CI/CD pipeline

package/templates/tests/evaluations/classification_eval.md

@@ -0,0 +1,136 @@
+# Classification Evaluation Template
+
+> Benchmark routing accuracy and categorization correctness
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| eval_name | classification_accuracy |
+| version | 1.0.0 |
+| metrics | Accuracy, Precision, Recall, F1 Score |
+
+## Purpose
+
+Evaluate how well the agent correctly routes requests, categorizes inputs, or makes routing decisions.
+
+## Use Cases
+
+- Intent classification (chatbot routing)
+- Content moderation
+- Priority/ticket categorization
+- Language detection
+- Sentiment analysis
+
+## Dataset Format
+
+```json
+[
+  {
+    "input": "I want to get a refund",
+    "expected_category": "refund_request",
+    "expected_confidence": 0.90
+  },
+  {
+    "input": "How do I change my password?",
+    "expected_category": "account_settings",
+    "expected_confidence": 0.85
+  }
+]
+```
+
+## Evaluation Metrics
+
+| Metric | Target | Description |
+|--------|--------|-------------|
+| Accuracy | ≥0.95 | % of correct classifications |
+| Macro F1 | ≥0.90 | F1 averaged across all categories |
+| Precision (per class) | ≥0.85 | True positives / predicted positives |
+| Recall (per class) | ≥0.85 | True positives / actual positives |
+| Confidence Calibration | ≤0.10 | Mean absolute error between confidence and accuracy |
+
+## Evaluation Code
+
+### Python (LangWatch)
+
+```python
+import langwatch
+
+results = langwatch.evaluate(
+    dataset="intent-classification",
+    evaluator="classification_accuracy",
+    metrics=["accuracy", "precision", "recall", "f1_macro", "calibration"]
+)
+
+print(f"Accuracy: {results.accuracy}")
+print(f"Macro F1: {results.f1_macro}")
+print(f"Precision: {results.precision}")
+print(f"Recall: {results.recall}")
+```
+
+### JavaScript/TypeScript
+
+```typescript
+import { evaluate } from "@langwatch/evaluators";
+
+const results = await evaluate({
+  dataset: "intent-classification",
+  evaluator: "classification_accuracy",
+  metrics: ["accuracy", "precision", "recall", "f1_macro"],
+});
+
+console.log(`Accuracy: ${results.accuracy}`);
+```
+
+## Pass/Fail Criteria
+
+| Metric | Threshold | Status |
+|--------|-----------|--------|
+| Accuracy | ≥0.95 | ✅ Pass |
+| Accuracy | 0.90-0.94 | ⚠️ Warning |
+| Accuracy | <0.90 | ❌ Fail |
+| Macro F1 | ≥0.90 | ✅ Pass |
+| Macro F1 | <0.85 | ❌ Fail |
+| Confidence Error | ≤0.10 | ✅ Pass |
+| Confidence Error | >0.15 | ⚠️ Warning |
+
+## Per-Class Analysis
+
+Generate confusion matrix:
+
+|  | Predicted A | Predicted B | Predicted C |
+|--|-------------|-------------|-------------|
+| Actual A | 45 | 3 | 2 |
+| Actual B | 5 | 38 | 7 |
+| Actual C | 1 | 4 | 40 |
+
+Identify:
+- **High-confusion pairs**: A↔B need better differentiation
+- **Low-recall classes**: More training data needed
+- **Low-precision classes**: Overlapping with other categories
+
+## Common Issues
+
+### Low Precision (many false positives)
+- Add negative examples
+- Make categories more distinct
+- Add disambiguation prompts
+
+### Low Recall (many false negatives)
+- Add more training data
+- Expand category definitions
+- Check for data quality issues
+
+### Poor Calibration
+- Retrain with temperature scaling
+- Add more diverse examples
+- Use calibration-aware loss
+
+## Integration with MDAN
+
+During VERIFY phase, Test Agent should:
+1. Identify all classification/routing points in the system
+2. Create evaluation datasets from real user queries
+3. Run classification evaluation
+4. Report per-class performance
+5. Fail if overall accuracy < 90%

package/templates/tests/evaluations/rag_eval.md

@@ -0,0 +1,116 @@
+# RAG Evaluation Template
+
+> Benchmark RAG (Retrieval-Augmented Generation) correctness and quality
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| eval_name | rag_correctness |
+| version | 1.0.0 |
+| metrics | F1 Score, Precision, Recall, Context Relevance |
+
+## Purpose
+
+Evaluate how well the RAG pipeline retrieves relevant context and generates accurate answers.
+
+## Dataset Format
+
+```json
+[
+  {
+    "query": "What is the refund policy?",
+    "expected_chunks": [
+      "refund_policy.md: paragraphs 1-3",
+      "faq.md: refund section"
+    ],
+    "expected_answer_contains": ["30 days", "original payment", "processing time"]
+  }
+]
+```
+
+## Evaluation Metrics
+
+### Retrieval Metrics
+
+| Metric | Target | Description |
+|--------|--------|-------------|
+| Recall | ≥0.85 | % of relevant chunks retrieved |
+| Precision | ≥0.90 | % of retrieved chunks that are relevant |
+| F1 Score | ≥0.87 | Harmonic mean of precision/recall |
+
+### Generation Metrics
+
+| Metric | Target | Description |
+|--------|--------|-------------|
+| Context Relevance | ≥0.80 | LLM judge scores context usefulness |
+| Answer Accuracy | ≥0.85 | Answer contains expected information |
+| Hallucination Rate | ≤0.05 | Facts not in context |
+
+## Evaluation Code
+
+### Python (LangWatch)
+
+```python
+import langwatch
+
+results = langwatch.evaluate(
+    dataset="customer-support-rag",
+    evaluator="rag_correctness",
+    metrics=["f1_score", "precision", "recall", "context_relevance"]
+)
+
+print(f"F1 Score: {results.f1_score}")
+print(f"Precision: {results.precision}")
+print(f"Recall: {results.recall}")
+print(f"Context Relevance: {results.context_relevance}")
+```
+
+### JavaScript/TypeScript
+
+```typescript
+import { evaluate } from "@langwatch/evaluators";
+
+const results = await evaluate({
+  dataset: "customer-support-rag",
+  evaluator: "rag_correctness",
+  metrics: ["f1_score", "precision", "recall", "context_relevance"],
+});
+
+console.log(`F1 Score: ${results.f1Score}`);
+```
+
+## Pass/Fail Criteria
+
+| Metric | Threshold | Status |
+|--------|-----------|--------|
+| F1 Score | ≥0.87 | ✅ Pass |
+| F1 Score | 0.70-0.86 | ⚠️ Warning |
+| F1 Score | <0.70 | ❌ Fail |
+| Hallucination | ≤0.05 | ✅ Pass |
+| Hallucination | >0.15 | ❌ Fail |
+
+## Troubleshooting
+
+### Low Recall
+- Check chunk size (try 512-1024)
+- Add more overlapping chunks
+- Improve embedding model
+
+### Low Precision
+- Reduce chunk size
+- Add more specific metadata filters
+- Filter out irrelevant sources
+
+### High Hallucination
+- Add source citations to prompt
+- Reduce max_tokens
+- Use better context ranking
+
+## Integration with MDAN
+
+During VERIFY phase, Test Agent should:
+1. Create RAG evaluation dataset from PRD
+2. Run retrieval + generation tests
+3. Report metrics in quality gate
+4. Fail if thresholds not met

package/templates/tests/scenarios/README.md

@@ -0,0 +1,62 @@
+# Test Scenarios Index
+
+> End-to-end conversational tests for MDAN projects
+
+## Overview
+
+Scenarios are conversation-based tests that validate agent behavior in realistic, multi-turn interactions. Unlike unit tests, they simulate how real users interact with your agent.
+
+## Available Scenarios
+
+| Scenario | Description | Agent |
+|----------|-------------|-------|
+| [basic_authentication.test.md](basic_authentication.test.md) | Login, logout, session management | Dev Agent |
+| [user_registration.test.md](user_registration.test.md) | Signup, validation, confirmation | Dev Agent |
+
+## Adding New Scenarios
+
+1. Copy this template:
+   ```bash
+   cp basic_authentication.test.md my_new_scenario.test.md
+   ```
+
+2. Edit the scenario:
+   - Update metadata (name, version, framework)
+   - Write the conversation script
+   - Define success criteria
+   - Add security checks if applicable
+
+3. Run the scenario:
+   ```bash
+   # Python/pytest
+   pytest tests/scenarios/my_new_scenario.test.md -v
+
+   # Node/TypeScript
+   npm test -- tests/scenarios/my_new_scenario.test.ts
+   ```
+
+## Scenario Format
+
+Each scenario includes:
+- **Metadata**: version, framework, duration estimate
+- **Preconditions**: what must be true before testing
+- **Script**: conversation steps with verification points
+- **Success Criteria**: checklist of must-pass conditions
+- **Security Checks**: validation for security requirements
+
+## Integration with MDAN
+
+Scenarios are automatically generated during the VERIFY phase:
+1. Test Agent reviews implemented features
+2. Creates relevant scenarios for critical flows
+3. Runs scenarios to validate behavior
+4. Reports results in quality gate
+
+## Framework Support
+
+| Framework | Command |
+|-----------|---------|
+| Jest | `jest tests/scenarios/` |
+| Pytest | `pytest tests/scenarios/` |
+| Playwright | `playwright test tests/scenarios/` |
+| Vitest | `vitest run tests/scenarios/` |

package/templates/tests/scenarios/basic_authentication.test.md

@@ -0,0 +1,82 @@
+# Scenario: User Authentication
+
+> Test conversation flow for basic authentication functionality
+
+## Metadata
+
+| Field | Value |
+|-------|-------|
+| scenario_name | basic_authentication |
+| version | 1.0.0 |
+| agent | Dev Agent |
+| framework | [Jest/Pytest/Playwright] |
+| estimated_duration | 30s |
+
+## Description
+
+Test that the authentication system handles login, logout, and session management correctly.
+
+## Preconditions
+
+- User is not logged in
+- Database contains test users
+- Auth service is running
+
+## Script
+
+### Test Case 1: Successful Login
+
+```
+USER: I want to log in with my account
+AGENT: [Should prompt for credentials or display login form]
+USER: My email is test@example.com and password is Test123!
+AGENT: [Should validate credentials]
+  -> VERIFY: auth_token received
+  -> VERIFY: user object returned with correct email
+USER: What's my username?
+AGENT: [Should return 'test@example.com' from session]
+  -> VERIFY: response contains correct username
+```
+
+### Test Case 2: Invalid Credentials
+
+```
+USER: I want to log in
+AGENT: [Should prompt for credentials]
+USER: email: wrong@example.com, password: wrongpass
+AGENT: [Should reject with error message]
+  -> VERIFY: error message does NOT reveal if email exists
+  -> VERIFY: no auth_token in response
+```
+
+### Test Case 3: Logout
+
+```
+USER: I'm logged in and want to log out
+AGENT: [Should clear session]
+  -> VERIFY: session cleared
+  -> VERIFY: confirmation message shown
+USER: Can I see my profile?
+AGENT: [Should deny access]
+  -> VERIFY: 401 or redirect to login
+```
+
+## Success Criteria
+
+- [ ] Login with valid credentials succeeds
+- [ ] Login with invalid credentials fails with secure error
+- [ ] Logout clears session completely
+- [ ] Protected routes redirect unauthenticated users
+- [ ] Session expires after configured timeout
+
+## Failure Handling
+
+If any step fails, the scenario should:
+1. Capture the actual response
+2. Compare with expected behavior
+3. Log the difference for debugging
+4. Fail the test with descriptive error
+
+## Notes
+
+This scenario tests the authentication flow end-to-end. Use with Playwright for browser-based testing or Pytest for API-based testing.