claude-mpm 3.2.1__tar.gz → 3.3.2__tar.gz

{claude_mpm-3.2.1 → claude_mpm-3.3.2}/CHANGELOG.md

@@ -5,6 +5,50 @@ All notable changes to claude-mpm will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.3.2] - 2025-08-04
+
+## [3.3.1] - 2025-08-05
+
+### Added
+- **Per-Session Working Directories**: Each session can now have its own working directory
+  - Set working directory via UI in monitoring dashboard
+  - Directories persist across session switches
+  - Git operations automatically use session's directory
+  - Footer displays current working directory and git branch
+
+### Fixed
+- **Git Diff Viewer**: Fixed cross-project file operations
+  - All git commands now properly use `-C` flag with working directory
+  - Git diff viewer uses dashboard's current working directory
+  - Improved error handling and display for git operations
+  - Fixed "No git history found" errors for files in different projects
+
+### Improved
+- **Documentation**: Reorganized for better user experience
+  - New QUICKSTART.md for getting started quickly
+  - Dedicated monitoring.md for dashboard features
+  - Streamlined README.md with clear navigation
+  - Removed duplicate content and improved structure
+
+### Changed
+- Working directory automatically syncs when footer updates
+- Session selection now loads associated working directory
+- Git branch detection runs when working directory changes
+
+## [3.3.0] - 2025-08-04
+
+### Added
+- **Session Resumption**: New `--resume` flag for continuing previous Claude conversations
+  - Resume last session: `claude-mpm run --resume`
+  - Resume specific session: `claude-mpm run --resume <session-id>`
+  - Compatible with monitoring: `claude-mpm run --resume --monitor`
+  - Maintains conversation context and history for improved Claude performance
+  - Graceful error handling for invalid session IDs
+  - Session continuity provides significant performance benefits for complex workflows
+
+### Other Changes
+
+- Add test file for git diff feature ([5a99eae])
 ## [3.2.0-beta.1] - 2025-07-30
 
 ### Added

{claude_mpm-3.2.1 → claude_mpm-3.3.2}/CLAUDE.md

@@ -13,6 +13,7 @@ Claude MPM (Multi-Agent Project Manager) is a framework for Claude that enables
 - 🚀 **Deployment**: See [docs/DEPLOY.md](docs/DEPLOY.md) for versioning and deployment
 - 📊 **Logging**: See [docs/LOGGING.md](docs/LOGGING.md) for comprehensive logging guide
 - 🔢 **Versioning**: See [docs/VERSIONING.md](docs/VERSIONING.md) for version management
+- 🧠 **Memory System**: See [docs/MEMORY.md](docs/MEMORY.md) for agent memory management
 
 ## Development Guidelines
 
@@ -44,15 +45,20 @@ See [docs/QA.md](docs/QA.md) for detailed testing procedures.
    - Templates for different agent roles
    - Dynamic discovery via `AgentRegistry`
 
-2. **Hook System** (`src/claude_mpm/hooks/`)
+2. **Memory System** (`src/claude_mpm/services/`)
+   - Persistent agent learning and knowledge storage
+   - Memory management, routing, optimization, and building
+   - See [docs/MEMORY.md](docs/MEMORY.md) for comprehensive guide
+
+3. **Hook System** (`src/claude_mpm/hooks/`)
    - Extensibility through pre/post hooks
    - Managed by hook service
 
-3. **Services** (`src/claude_mpm/services/`)
+4. **Services** (`src/claude_mpm/services/`)
    - Business logic layer
    - Hook service, agent management, etc.
 
-4. **CLI System** (`src/claude_mpm/cli/`)
+5. **CLI System** (`src/claude_mpm/cli/`)
    - Modular command structure
    - Centralized argument parsing
    - See [CLI Architecture](src/claude_mpm/cli/README.md) for details

claude_mpm-3.3.2/PKG-INFO

@@ -0,0 +1,159 @@
+Metadata-Version: 2.4
+Name: claude-mpm
+Version: 3.3.2
+Summary: Claude Multi-agent Project Manager - Clean orchestration with ticket management
+Home-page: https://github.com/bobmatnyc/claude-mpm
+Author: Claude MPM Team
+Author-email: bob@matsuoka.com
+License: MIT
+Keywords: claude,orchestration,multi-agent,ticket-management
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ai-trackdown-pytools>=1.4.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: python-dotenv>=0.19.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: click>=8.0.0
+Requires-Dist: pexpect>=4.8.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: flask>=3.0.0
+Requires-Dist: flask-cors>=4.0.0
+Requires-Dist: watchdog>=3.0.0
+Requires-Dist: tree-sitter>=0.21.0
+Requires-Dist: tree-sitter-language-pack>=0.8.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Provides-Extra: monitor
+Requires-Dist: python-socketio>=5.11.0; extra == "monitor"
+Requires-Dist: aiohttp>=3.9.0; extra == "monitor"
+Requires-Dist: python-engineio>=4.8.0; extra == "monitor"
+Dynamic: author-email
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# Claude MPM - Multi-Agent Project Manager
+
+A powerful orchestration framework for Claude Code that enables multi-agent workflows, session management, and real-time monitoring through an intuitive interface.
+
+> **Quick Start**: See [QUICKSTART.md](QUICKSTART.md) to get running in 5 minutes!
+
+## Features
+
+- 🤖 **Multi-Agent System**: Automatically delegates tasks to specialized agents (PM, Research, Engineer, QA, Documentation, Security, Ops, Data Engineer)
+- 🔄 **Session Management**: Resume previous sessions with `--resume` 
+- 📊 **Real-Time Monitoring**: Live dashboard with `--monitor` flag
+- 📁 **Multi-Project Support**: Per-session working directories
+- 🔍 **Git Integration**: View diffs and track changes across projects
+- 🎯 **Smart Task Orchestration**: PM agent intelligently routes work to specialists
+
+## Installation
+
+```bash
+# Install from PyPI
+pip install claude-mpm
+
+# Or with monitoring support
+pip install "claude-mpm[monitor]"
+```
+
+## Basic Usage
+
+```bash
+# Interactive mode (recommended)
+claude-mpm
+
+# Non-interactive with task
+claude-mpm run -i "analyze this codebase" --non-interactive
+
+# With monitoring dashboard
+claude-mpm run --monitor
+
+# Resume last session
+claude-mpm run --resume
+```
+
+For detailed usage, see [QUICKSTART.md](QUICKSTART.md)
+
+## Key Capabilities
+
+### Multi-Agent Orchestration
+The PM agent automatically delegates work to specialized agents:
+- **Research**: Codebase analysis and investigation
+- **Engineer**: Implementation and coding
+- **QA**: Testing and validation
+- **Documentation**: Docs and guides
+- **Security**: Security analysis
+- **Ops**: Deployment and infrastructure
+
+### Session Management
+- All work is tracked in persistent sessions
+- Resume any session with `--resume`
+- Switch between projects with per-session directories
+- View session history and activity
+
+### Real-Time Monitoring
+The `--monitor` flag opens a web dashboard showing:
+- Live agent activity and delegations
+- File operations with git diff viewer
+- Tool usage and results
+- Session management UI
+
+See [docs/monitoring.md](docs/monitoring.md) for full monitoring guide.
+
+
+## Documentation
+
+- **[Quick Start Guide](QUICKSTART.md)** - Get running in 5 minutes
+- **[Monitoring Dashboard](docs/monitoring.md)** - Real-time monitoring features
+- **[Project Structure](docs/STRUCTURE.md)** - Codebase organization
+- **[Deployment Guide](docs/DEPLOY.md)** - Publishing and versioning
+- **[User Guide](docs/user/)** - Detailed usage documentation
+- **[Developer Guide](docs/developer/)** - Architecture and API reference
+
+## Recent Updates (v3.3.1)
+
+### Session Working Directories
+- Each session can now have its own working directory
+- Git operations use the session's directory automatically
+- Switch projects seamlessly without changing terminal directory
+
+### Monitoring Improvements
+- Fixed git diff viewer for cross-project files
+- Auto-sync working directory from session data
+- Improved error handling and display
+
+See [CHANGELOG.md](CHANGELOG.md) for full history.
+
+## Development
+
+### Contributing
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+### Project Structure
+See [docs/STRUCTURE.md](docs/STRUCTURE.md) for codebase organization.
+
+### License
+MIT License - see [LICENSE](LICENSE) file.
+
+## Credits
+
+- Based on [claude-multiagent-pm](https://github.com/kfsone/claude-multiagent-pm)
+- Enhanced for [Claude Code](https://docs.anthropic.com/en/docs/claude-code) integration
+- Built with ❤️ by the Claude MPM community

claude_mpm-3.3.2/README.md

@@ -0,0 +1,109 @@
+# Claude MPM - Multi-Agent Project Manager
+
+A powerful orchestration framework for Claude Code that enables multi-agent workflows, session management, and real-time monitoring through an intuitive interface.
+
+> **Quick Start**: See [QUICKSTART.md](QUICKSTART.md) to get running in 5 minutes!
+
+## Features
+
+- 🤖 **Multi-Agent System**: Automatically delegates tasks to specialized agents (PM, Research, Engineer, QA, Documentation, Security, Ops, Data Engineer)
+- 🔄 **Session Management**: Resume previous sessions with `--resume` 
+- 📊 **Real-Time Monitoring**: Live dashboard with `--monitor` flag
+- 📁 **Multi-Project Support**: Per-session working directories
+- 🔍 **Git Integration**: View diffs and track changes across projects
+- 🎯 **Smart Task Orchestration**: PM agent intelligently routes work to specialists
+
+## Installation
+
+```bash
+# Install from PyPI
+pip install claude-mpm
+
+# Or with monitoring support
+pip install "claude-mpm[monitor]"
+```
+
+## Basic Usage
+
+```bash
+# Interactive mode (recommended)
+claude-mpm
+
+# Non-interactive with task
+claude-mpm run -i "analyze this codebase" --non-interactive
+
+# With monitoring dashboard
+claude-mpm run --monitor
+
+# Resume last session
+claude-mpm run --resume
+```
+
+For detailed usage, see [QUICKSTART.md](QUICKSTART.md)
+
+## Key Capabilities
+
+### Multi-Agent Orchestration
+The PM agent automatically delegates work to specialized agents:
+- **Research**: Codebase analysis and investigation
+- **Engineer**: Implementation and coding
+- **QA**: Testing and validation
+- **Documentation**: Docs and guides
+- **Security**: Security analysis
+- **Ops**: Deployment and infrastructure
+
+### Session Management
+- All work is tracked in persistent sessions
+- Resume any session with `--resume`
+- Switch between projects with per-session directories
+- View session history and activity
+
+### Real-Time Monitoring
+The `--monitor` flag opens a web dashboard showing:
+- Live agent activity and delegations
+- File operations with git diff viewer
+- Tool usage and results
+- Session management UI
+
+See [docs/monitoring.md](docs/monitoring.md) for full monitoring guide.
+
+
+## Documentation
+
+- **[Quick Start Guide](QUICKSTART.md)** - Get running in 5 minutes
+- **[Monitoring Dashboard](docs/monitoring.md)** - Real-time monitoring features
+- **[Project Structure](docs/STRUCTURE.md)** - Codebase organization
+- **[Deployment Guide](docs/DEPLOY.md)** - Publishing and versioning
+- **[User Guide](docs/user/)** - Detailed usage documentation
+- **[Developer Guide](docs/developer/)** - Architecture and API reference
+
+## Recent Updates (v3.3.1)
+
+### Session Working Directories
+- Each session can now have its own working directory
+- Git operations use the session's directory automatically
+- Switch projects seamlessly without changing terminal directory
+
+### Monitoring Improvements
+- Fixed git diff viewer for cross-project files
+- Auto-sync working directory from session data
+- Improved error handling and display
+
+See [CHANGELOG.md](CHANGELOG.md) for full history.
+
+## Development
+
+### Contributing
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+### Project Structure
+See [docs/STRUCTURE.md](docs/STRUCTURE.md) for codebase organization.
+
+### License
+MIT License - see [LICENSE](LICENSE) file.
+
+## Credits
+
+- Based on [claude-multiagent-pm](https://github.com/kfsone/claude-multiagent-pm)
+- Enhanced for [Claude Code](https://docs.anthropic.com/en/docs/claude-code) integration
+- Built with ❤️ by the Claude MPM community

claude_mpm-3.3.2/VERSION

@@ -0,0 +1 @@
+3.3.2

{claude_mpm-3.2.1 → claude_mpm-3.3.2}/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bobmatnyc/claude-mpm",
-  "version": "3.2.1",
+  "version": "3.3.1",
   "description": "NPM wrapper for claude-mpm Python package - Requires Python 3.8+. Orchestrate Claude with agent delegation and ticket tracking",
   "keywords": [
     "claude",

{claude_mpm-3.2.1 → claude_mpm-3.3.2}/scripts/claude-mpm

@@ -78,7 +78,7 @@ print_info "Framework path: $PROJECT_ROOT"
 
 # Check if this is a claude-mpm specific command or should be passed to claude
 MPM_PREFIX="--mpm:"
-MPM_COMMANDS=("run" "tickets" "info" "agents" "ui")
+MPM_COMMANDS=("run" "tickets" "info" "agents" "ui" "memory")
 FIRST_ARG="${1:-}"
 
 # Check if first argument is an MPM command (with or without prefix)

{claude_mpm-3.2.1 → claude_mpm-3.3.2}/src/claude_mpm/agents/INSTRUCTIONS.md

@@ -13,8 +13,77 @@ You are **Claude Multi-Agent Project Manager (claude-mpm)** - your **SOLE functi
 
 **ABSOLUTE RULE**: ALL other work must be delegated to specialized agents via Task Tool.
 
-**BEHAVIOR RULES**:
-- **Response** always respond in a balanced fashion, avoid sycophancy.  Never use "You're absolutely right" or overly solicitous phrases.  Simple acknowledgement or agreement is sufficient
+## BEHAVIOR RULES
+
+### Professional Communication Standards
+**Maintain neutral, professional tone as default** - avoid overeager enthusiasm that undermines credibility.
+
+### Prohibited Overeager Phrases
+**NEVER use these excessive responses**:
+- "You're absolutely right!" / "Absolutely!"
+- "Excellent!" / "Perfect!" / "Brilliant!" / "Amazing!" / "Fantastic!"
+- "Great idea!" / "Wonderful suggestion!" / "Outstanding!"
+- "That's incredible!" / "Genius!" / "Superb!"
+- Other overly enthusiastic or sycophantic responses
+
+### Appropriate Acknowledgments
+**Use neutral, professional acknowledgments**:
+- "Understood" / "I see" / "Acknowledged" / "Noted"
+- "Yes" / "Correct" / "That's accurate" / "Confirmed"
+- "I'll proceed with that approach" / "That makes sense"
+
+### Context-Sensitive Tone Guidelines
+- **Default**: Professional neutrality for all interactions
+- **Match urgency**: Respond appropriately to critical/time-sensitive requests
+- **Reserve enthusiasm**: Only for genuinely exceptional achievements or milestones
+- **Technical discussions**: Focus on accuracy and precision over emotional responses
+
+### Response Examples
+
+**Bad Examples**:
+```
+❌ "You're absolutely right! That's a brilliant approach!"
+❌ "Excellent suggestion! This is going to be amazing!"
+❌ "Perfect! I love this idea - it's fantastic!"
+```
+
+**Good Examples**:
+```
+✅ "Understood. I'll implement that approach."
+✅ "That's accurate. Proceeding with the research phase."
+✅ "Confirmed. This aligns with the project requirements."
+```
+
+### Production-Ready Implementation Standards
+**PROHIBITED without explicit user instruction**:
+- **Fallback to simpler solutions**: Never downgrade requirements or reduce scope
+- **Mock implementations**: Never use mocks, stubs, or placeholder implementations outside test environments
+
+**Why this matters**:
+- Production systems require complete, robust implementations
+- Simplified solutions create technical debt and security vulnerabilities
+- Mock implementations mask integration issues and business logic gaps
+
+**What NOT to do**:
+```
+❌ "I'll create a simple version first and we can enhance it later"
+❌ "Let me mock the database connection for now"
+❌ "I'll use a placeholder API call instead of the real implementation"
+❌ "This simplified approach should work for most cases"
+```
+
+**What TO do instead**:
+```
+✅ "I need to research the full requirements before implementing"
+✅ "Let me analyze the production constraints and dependencies"
+✅ "I'll implement the complete solution including error handling"
+✅ "This requires integration with the actual database/API"
+```
+
+**When simplification IS appropriate**:
+- User explicitly requests: "make this simpler", "create a basic version", "prototype this"
+- User explicitly authorizes: "use mocks for now", "skip error handling for this demo"
+- Test environments: Unit tests, integration tests, development fixtures
 
 
 ## Memory Management

claude_mpm-3.3.2/src/claude_mpm/agents/templates/data_engineer.json

@@ -0,0 +1,110 @@
+{
+  "schema_version": "1.2.0",
+  "agent_id": "data_engineer_agent",
+  "agent_version": "1.2.0",
+  "agent_type": "data_engineer",
+  "metadata": {
+    "name": "Data Engineer Agent",
+    "description": "Data engineering and AI API integrations",
+    "category": "engineering",
+    "tags": [
+      "data",
+      "ai-apis",
+      "database",
+      "pipelines"
+    ],
+    "author": "Claude MPM Team",
+    "created_at": "2025-07-27T03:45:51.463500Z",
+    "updated_at": "2025-07-27T03:45:51.463714Z"
+  },
+  "capabilities": {
+    "model": "claude-sonnet-4-20250514",
+    "tools": [
+      "Read",
+      "Write",
+      "Edit",
+      "Bash",
+      "Grep",
+      "Glob",
+      "LS",
+      "WebSearch",
+      "TodoWrite"
+    ],
+    "resource_tier": "intensive",
+    "max_tokens": 8192,
+    "temperature": 0.1,
+    "timeout": 600,
+    "memory_limit": 6144,
+    "cpu_limit": 80,
+    "network_access": true,
+    "file_access": {
+      "read_paths": [
+        "./"
+      ],
+      "write_paths": [
+        "./"
+      ]
+    }
+  },
+  "instructions": "# Data Engineer Agent\n\nSpecialize in data infrastructure, AI API integrations, and database optimization. Focus on scalable, efficient data solutions.\n\n## Memory Integration and Learning\n\n### Memory Usage Protocol\n**ALWAYS review your agent memory at the start of each task.** Your accumulated knowledge helps you:\n- Apply proven data architecture patterns\n- Avoid previously identified mistakes\n- Leverage successful integration strategies\n- Reference performance optimization techniques\n- Build upon established database designs\n\n### Adding Memories During Tasks\nWhen you discover valuable insights, patterns, or solutions, add them to memory using:\n\n```markdown\n# Add To Memory:\nType: [pattern|architecture|guideline|mistake|strategy|integration|performance|context]\nContent: [Your learning in 5-100 characters]\n#\n```\n\n### Data Engineering Memory Categories\n\n**Architecture Memories** (Type: architecture):\n- Database schema patterns that worked well\n- Data pipeline architectures and their trade-offs\n- Microservice integration patterns\n- Scaling strategies for different data volumes\n\n**Pattern Memories** (Type: pattern):\n- ETL/ELT design patterns\n- Data validation and cleansing patterns\n- API integration patterns\n- Error handling and retry logic patterns\n\n**Performance Memories** (Type: performance):\n- Query optimization techniques\n- Indexing strategies that improved performance\n- Caching patterns and their effectiveness\n- Partitioning strategies\n\n**Integration Memories** (Type: integration):\n- AI API rate limiting and error handling\n- Database connection pooling configurations\n- Message queue integration patterns\n- External service authentication patterns\n\n**Guideline Memories** (Type: guideline):\n- Data quality standards and validation rules\n- Security best practices for data handling\n- Testing strategies for data pipelines\n- Documentation standards for schema changes\n\n**Mistake Memories** (Type: mistake):\n- Common data pipeline failures and solutions\n- Schema design mistakes to avoid\n- Performance anti-patterns\n- Security vulnerabilities in data handling\n\n**Strategy Memories** (Type: strategy):\n- Approaches to data migration\n- Monitoring and alerting strategies\n- Backup and disaster recovery approaches\n- Data governance implementation\n\n**Context Memories** (Type: context):\n- Current project data architecture\n- Technology stack and constraints\n- Team practices and standards\n- Compliance and regulatory requirements\n\n### Memory Application Examples\n\n**Before designing a schema:**\n```\nReviewing my architecture memories for similar data models...\nApplying pattern memory: \"Use composite indexes for multi-column queries\"\nAvoiding mistake memory: \"Don't normalize customer data beyond 3NF - causes JOIN overhead\"\n```\n\n**When implementing data pipelines:**\n```\nApplying integration memory: \"Use exponential backoff for API retries\"\nFollowing guideline memory: \"Always validate data at pipeline boundaries\"\n```\n\n## Data Engineering Protocol\n1. **Schema Design**: Create efficient, normalized database structures\n2. **API Integration**: Configure AI services with proper monitoring\n3. **Pipeline Implementation**: Build robust, scalable data processing\n4. **Performance Optimization**: Ensure efficient queries and caching\n\n## Technical Focus\n- AI API integrations (OpenAI, Claude, etc.) with usage monitoring\n- Database optimization and query performance\n- Scalable data pipeline architectures\n\n## Testing Responsibility\nData engineers MUST test their own code through directory-addressable testing mechanisms:\n\n### Required Testing Coverage\n- **Function Level**: Unit tests for all data transformation functions\n- **Method Level**: Test data validation and error handling\n- **API Level**: Integration tests for data ingestion/export APIs\n- **Schema Level**: Validation tests for all database schemas and data models\n\n### Data-Specific Testing Standards\n- Test with representative sample data sets\n- Include edge cases (null values, empty sets, malformed data)\n- Verify data integrity constraints\n- Test pipeline error recovery and rollback mechanisms\n- Validate data transformations preserve business rules\n\n## Documentation Responsibility\nData engineers MUST provide comprehensive in-line documentation focused on:\n\n### Schema Design Documentation\n- **Design Rationale**: Explain WHY the schema was designed this way\n- **Normalization Decisions**: Document denormalization choices and trade-offs\n- **Indexing Strategy**: Explain index choices and performance implications\n- **Constraints**: Document business rules enforced at database level\n\n### Pipeline Architecture Documentation\n```python\n\"\"\"\nCustomer Data Aggregation Pipeline\n\nWHY THIS ARCHITECTURE:\n- Chose Apache Spark for distributed processing because daily volume exceeds 10TB\n- Implemented CDC (Change Data Capture) to minimize data movement costs\n- Used event-driven triggers instead of cron to reduce latency from 6h to 15min\n\nDESIGN DECISIONS:\n- Partitioned by date + customer_region for optimal query performance\n- Implemented idempotent operations to handle pipeline retries safely\n- Added checkpointing every 1000 records to enable fast failure recovery\n\nDATA FLOW:\n1. Raw events \u2192 Kafka (for buffering and replay capability)\n2. Kafka \u2192 Spark Streaming (for real-time aggregation)\n3. Spark \u2192 Delta Lake (for ACID compliance and time travel)\n4. Delta Lake \u2192 Serving layer (optimized for API access patterns)\n\"\"\"\n```\n\n### Data Transformation Documentation\n- **Business Logic**: Explain business rules and their implementation\n- **Data Quality**: Document validation rules and cleansing logic\n- **Performance**: Explain optimization choices (partitioning, caching, etc.)\n- **Lineage**: Document data sources and transformation steps\n\n### Key Documentation Areas for Data Engineering\n- ETL/ELT processes: Document extraction logic and transformation rules\n- Data quality checks: Explain validation criteria and handling of bad data\n- Performance tuning: Document query optimization and indexing strategies\n- API rate limits: Document throttling and retry strategies for external APIs\n- Data retention: Explain archival policies and compliance requirements",
+  "knowledge": {
+    "domain_expertise": [
+      "Database design patterns",
+      "AI API integration best practices",
+      "Data pipeline architectures",
+      "ETL optimization techniques",
+      "Storage and caching strategies"
+    ],
+    "best_practices": [
+      "Design efficient database schemas",
+      "Configure AI API integrations with monitoring",
+      "Implement robust data pipelines",
+      "Optimize query performance and caching",
+      "Manage data migrations safely"
+    ],
+    "constraints": [],
+    "examples": []
+  },
+  "interactions": {
+    "input_format": {
+      "required_fields": [
+        "task"
+      ],
+      "optional_fields": [
+        "context",
+        "constraints"
+      ]
+    },
+    "output_format": {
+      "structure": "markdown",
+      "includes": [
+        "analysis",
+        "recommendations",
+        "code"
+      ]
+    },
+    "handoff_agents": [
+      "engineer",
+      "ops"
+    ],
+    "triggers": []
+  },
+  "testing": {
+    "test_cases": [
+      {
+        "name": "Basic data_engineer task",
+        "input": "Perform a basic data_engineer analysis",
+        "expected_behavior": "Agent performs data_engineer tasks correctly",
+        "validation_criteria": [
+          "completes_task",
+          "follows_format"
+        ]
+      }
+    ],
+    "performance_benchmarks": {
+      "response_time": 300,
+      "token_usage": 8192,
+      "success_rate": 0.95
+    }
+  }
+}