massgen 0.1.4__tar.gz → 0.1.6__tar.gz

{massgen-0.1.4 → massgen-0.1.6}/CHANGELOG.md

@@ -7,16 +7,137 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Recent Releases
 
+**v0.1.6 (November 2025)** - Framework Interoperability & Backend Refactoring
+External agent framework integration as tools (AG2, LangGraph, AgentScope, OpenAI Assistants, SmoLAgent), unified tool execution architecture, Gemini backend simplification, and comprehensive configuration validation.
+
+**v0.1.5 (October 2025)** - Memory System
+Long-term memory with semantic retrieval via mem0, automatic context compression, and memory sharing for multi-turn conversations.
+
 **v0.1.4 (October 2025)** - Multimodal Generation Tools & Binary File Protection
 Comprehensive generation tools for images, videos, audio, and documents with OpenAI APIs, binary file blocking for security, web crawling integration, and enhanced documentation infrastructure.
 
-**v0.1.3 (October 2025)** - Post-Evaluation Tools & Multimodal Understanding
-Post-evaluation workflow with submit/restart capabilities, custom multimodal understanding tools, Docker sudo mode, and enhanced config builder.
+---
 
-**v0.1.2 (October 2025)** - Intelligent Planning Mode & Model Updates
-Automatic irreversibility analysis for MCP tools, selective tool blocking, Claude 4.5 Haiku support, and Grok web search improvements.
+## [0.1.6] - 2025-10-31
 
----
+### Added
+- **Framework Interoperability**: External agent framework integration as MassGen custom tools
+  - New `massgen/tool/_extraframework_agents/` module with 5 framework integrations
+  - **AG2 Lesson Planner Tool**: Nested chat functionality wrapped as custom tool for multi-agent lesson planning (supports streaming)
+  - **LangGraph Lesson Planner Tool**: LangGraph graph-based workflows integrated as tool
+  - **AgentScope Lesson Planner Tool**: AgentScope agent system wrapped for lesson creation
+  - **OpenAI Assistants Lesson Planner Tool**: OpenAI Assistants API integrated as tool
+  - **SmoLAgent Lesson Planner Tool**: HuggingFace SmoLAgent integration for lesson planning
+  - Enables MassGen agents to delegate tasks to specialized external frameworks
+  - Each framework runs autonomously and returns results to MassGen orchestrator
+  - Note: Only AG2 currently supports streaming; other frameworks return complete results
+
+- **Configuration Validator**: Comprehensive YAML configuration validation system
+  - New `ConfigValidator` class in `massgen/config_validator.py` for pre-flight validation
+  - Memory configuration validation with detailed error messages
+  - Pre-commit hook integration for automatic config validation
+  - Comprehensive test suite in `massgen/tests/test_config_validator.py`
+  - Validates agent configurations, backend parameters, tool settings, and memory options
+  - Provides actionable error messages with suggestions for common mistakes
+
+### Changed
+- **Backend Architecture Refactoring**: Unified tool execution with ToolExecutionConfig
+  - New `ToolExecutionConfig` dataclass in `base_with_custom_tool_and_mcp.py` for standardized tool handling
+  - Refactored `ResponseBackend` with unified tool execution flow
+  - Refactored `ChatCompletionsBackend` with unified tool execution flow
+  - Refactored `ClaudeBackend` with unified tool execution methods
+  - Eliminates duplicate code paths between custom tools and MCP tools
+  - Consistent error handling and status reporting across all tool types
+  - Improved maintainability and extensibility for future tool systems
+
+- **Gemini Backend Simplification**: Major architectural cleanup and consolidation
+  - Removed `gemini_mcp_manager.py` module
+  - Removed `gemini_trackers.py` module
+  - Refactored `gemini.py` to use manual tool execution via base class
+  - Streamlined tool handling and cleanup logic
+  - Removed continuation logic and duplicate code
+  - Updated `_gemini_formatter.py` for simplified tool conversion
+  - Net reduction of 1,598 lines through consolidation
+  - Improved maintainability and performance
+
+- **Custom Tool System Enhancement**: Improved tool management and execution
+  - Enhanced `ToolManager` with category management capabilities
+  - Improved tool registration and validation system
+  - Enhanced tool result handling and error reporting
+  - Better support for async tool execution
+  - Improved tool schema generation for LLM consumption
+
+### Documentations, Configurations and Resources
+
+- **Framework Interoperability Examples**: 8 new configuration files demonstrating external framework integration
+  - **AG2 Examples**: `ag2_lesson_planner_example.yaml`, `ag2_and_langgraph_lesson_planner.yaml`, `ag2_and_openai_assistant_lesson_planner.yaml`
+  - **LangGraph Examples**: `langgraph_lesson_planner_example.yaml`
+  - **AgentScope Examples**: `agentscope_lesson_planner_example.yaml`
+  - **OpenAI Assistants Examples**: `openai_assistant_lesson_planner_example.yaml`
+  - **SmoLAgent Examples**: `smolagent_lesson_planner_example.yaml`
+  - **Multi-Framework Examples**: `two_models_with_tools_example.yaml`
+
+### Technical Details
+- **Major Focus**: Framework interoperability for external agent integration, unified tool execution architecture, Gemini backend simplification, and configuration validation system
+- **Contributors**: @Eric-Shang @praneeth999 @ncrispino @qidanrui @sonichi @Henry-811 and the MassGen team
+
+## [0.1.5] - 2025-10-29
+
+### Added
+- **Memory System**: Complete long-term memory implementation with semantic retrieval
+  - New `massgen/memory/` module with comprehensive memory management
+  - **PersistentMemory** via mem0 integration for semantic fact storage and retrieval
+  - **ConversationMemory** for short-term verbatim message tracking
+  - **Automatic Context Compression** when approaching token limits
+  - **Memory Sharing for Multi-Turn Conversations** with turn-aware filtering to prevent temporal leakage
+  - **Session Management** for memory isolation and continuation across runs
+  - **Qdrant Vector Database Integration** for efficient semantic search (server and local modes)
+  - **Context Monitoring** with real-time token usage tracking
+  - Fact extraction prompts with customizable LLM and embedding providers
+  - Supports OpenAI, Anthropic, Groq, and other mem0-compatible providers
+
+- **Memory Configuration Support**: New YAML configuration options
+  - Memory enable/disable toggle at global and per-agent levels
+  - Configurable compression thresholds (trigger_threshold, target_ratio)
+  - Retrieval settings (limit, exclude_recent for smart retrieval)
+  - Session naming for continuation and cross-session memory
+  - LLM and embedding provider configuration for mem0
+  - Qdrant connection settings (server/local mode, host, port, path)
+
+### Changed
+- **Chat Agent Enhancement**: Memory integration for agent workflows
+  - Memory recording after agent responses (conversation and persistent)
+  - Memory retrieval on restart/reset for context restoration
+  - Integration with compression and context monitoring modules
+
+- **Orchestrator Enhancement**: Memory coordination for multi-agent workflows
+  - Memory initialization and management across agent lifecycles
+  - Memory cleanup on orchestrator shutdown
+
+### Documentations, Configurations and Resources
+
+- **Memory Documentation**: Comprehensive memory system user guide
+  - New `docs/source/user_guide/memory.rst`
+  - Complete usage guide with quick start, configuration reference, and examples
+  - Design decisions documentation explaining architecture choices
+  - Troubleshooting guide for common memory issues
+  - Monitoring and debugging instructions with log examples
+  - API reference for PersistentMemory, ConversationMemory, and ContextMonitor
+
+- **Configuration Examples**: 5 new memory-focused YAML configurations
+  - `gpt5mini_gemini_context_window_management.yaml`: Multi-agent with context compression
+  - `gpt5mini_gemini_research_to_implementation.yaml`: Research to implementation workflow
+  - `gpt5mini_high_reasoning_gemini.yaml`: High reasoning agents with memory
+  - `gpt5mini_gemini_baseline_research_to_implementation.yaml`: Baseline research workflow
+  - `single_agent_compression_test.yaml`: Testing compression behavior
+
+- **Infrastructure and Testing**:
+  - Memory test suite with 4 test files in `massgen/tests/memory/`
+  - Additional memory tests: `test_agent_memory.py`, `test_conversation_memory.py`, `test_orchestrator_memory.py`, `test_persistent_memory.py`
+
+### Technical Details
+- **Major Focus**: Long-term memory system with semantic retrieval and memory sharing for multi-turn conversations
+- **Contributors**: @ncrispino @qidanrui @kitrakrev @sonichi @Henry-811 and the MassGen team
 
 ## [0.1.4] - 2025-10-27
 
@@ -61,14 +182,6 @@ Automatic irreversibility analysis for MCP tools, selective tool blocking, Claud
   - Updated `docs/source/user_guide/protected_paths.rst` with binary file protection section
   - Documents 40+ protected binary file types and specialized tool suggestions
 
-- **Enhanced Pre-commit Hooks**: Improved code quality automation
-  - Updated `.pre-commit-config.yaml` with additional checks
-  - Better README synchronization and validation
-
-- **Documentation Automation**: CI/CD for documentation deployment
-  - New `.github/workflows/docs-automation.yml` for automated doc builds
-  - Removed legacy docs.yml workflow
-
 - **Configuration Examples**: 9 new YAML configuration files
   - **Generation Tools**: 8 multimodal generation configurations
     - `text_to_image_generation_single.yaml` and `text_to_image_generation_multi.yaml`

{massgen-0.1.4 → massgen-0.1.6}/CONTRIBUTING.md

@@ -95,9 +95,85 @@ To add support for a new model provider:
 2. Inherit from the base backend class in `massgen/backend/base.py`
 3. Implement the required methods for message processing and completion parsing
 4. Add the model mapping in `massgen/utils.py`
-5. Update configuration templates in `massgen/configs/`
-6. Add tests in `massgen/tests/`
-7. Update documentation
+5. **Add backend capabilities to `massgen/backend/capabilities.py`** - Required for config validation
+6. Update configuration templates in `massgen/configs/`
+7. Add tests in `massgen/tests/`
+8. **Update the config validator** - See [Updating the Config Validator](#updating-the-config-validator) below
+9. Update documentation
+
+### Updating the Config Validator
+
+**⚠️ IMPORTANT**: Whenever you add new configuration options, backends, or change the config schema, you MUST update the config validator.
+
+The config validator (`massgen/config_validator.py`) ensures users get helpful error messages when they make configuration mistakes. When you add a new feature:
+
+**1. Add to Backend Capabilities** (for new backends)
+```python
+# In massgen/backend/capabilities.py
+BACKEND_CAPABILITIES = {
+    # ...
+    "your_backend": BackendCapabilities(
+        backend_type="your_backend",
+        provider_name="Your Backend Name",
+        supported_capabilities={"mcp", "web_search", ...},
+        builtin_tools=[...],
+        filesystem_support="mcp",
+        models=["model-1", "model-2"],
+        default_model="model-1",
+        env_var="YOUR_API_KEY",
+        notes="Description of your backend",
+    ),
+}
+```
+
+**2. Update Validator Enums** (for new config options)
+```python
+# In massgen/config_validator.py
+# If adding new valid values for existing fields:
+VALID_PERMISSION_MODES = {"default", "acceptEdits", "bypassPermissions", "plan"}
+VALID_DISPLAY_TYPES = {"rich_terminal", "simple"}
+```
+
+**3. Add Validation Logic** (for new config fields)
+```python
+# In massgen/config_validator.py
+# Add validation in the appropriate _validate_* method
+def _validate_your_feature(self, config, result):
+    if "your_field" in config:
+        # Validate type
+        # Validate values
+        # Add helpful error messages
+```
+
+**4. Add Tests for Common Mistakes**
+```python
+# In massgen/tests/test_config_validator.py
+def test_invalid_your_feature(self):
+    """Test invalid your_feature value."""
+    config = {
+        "your_field": "invalid_value",
+        # ...
+    }
+    validator = ConfigValidator()
+    result = validator.validate_config(config)
+    assert not result.is_valid()
+    assert any("your helpful error message" in error.message for error in result.errors)
+```
+
+**5. Test Validation**
+```bash
+# Validate all example configs still pass
+uv run python scripts/validate_all_configs.py
+
+# Run validator tests
+uv run pytest massgen/tests/test_config_validator.py -v
+```
+
+**Why This Matters:**
+- Users get clear, actionable error messages instead of cryptic failures
+- Prevents common configuration mistakes
+- Makes the framework more accessible to new users
+- Pre-commit hooks automatically validate configs in `massgen/configs/`
 
 ## 🔒 API Stability & Versioning
 
@@ -204,10 +280,12 @@ orchestrator:
   snapshot_storage: "snapshots"
   agent_temporary_workspace: "temp_workspaces"
 
-# Memory configuration (schema not yet implemented but may evolve when it does)
+# Memory configuration (implemented in v0.1.5 - see docs/source/user_guide/memory.rst)
 memory:
   enabled: true
-  provider: mem0
+  persistent_memory:
+    enabled: true
+    vector_store: "qdrant"
 ```
 
 ## 📋 Prerequisites
@@ -267,7 +345,7 @@ Create a `.env` file in the `massgen` directory as described in [README](README.
 
 ## 🔧 Development Workflow
 
-> **Important**: Our next version is v0.1.5. If you want to contribute, please contribute to the `dev/v0.1.5` branch.
+> **Important**: Our next version is v0.1.7. If you want to contribute, please contribute to the `dev/v0.1.7` branch (or `main` if dev/v0.1.7 doesn't exist yet).
 
 ### 1. Create Feature Branch
 
@@ -275,8 +353,8 @@ Create a `.env` file in the `massgen` directory as described in [README](README.
 # Fetch latest changes from upstream
 git fetch upstream
 
-# Create feature branch from dev/v0.1.5
-git checkout -b feature/your-feature-name upstream/dev/v0.1.5
+# Create feature branch from dev/v0.1.7 (or main if dev branch doesn't exist yet)
+git checkout -b feature/your-feature-name upstream/dev/v0.1.7
 ```
 
 ### 2. Make Your Changes
@@ -373,7 +451,7 @@ git push origin feature/your-feature-name
 ```
 
 Then create a pull request on GitHub:
-- Base branch: `dev/v0.1.5`
+- Base branch: `dev/v0.1.7` (or `main` if dev branch doesn't exist yet)
 - Compare branch: `feature/your-feature-name`
 - Add clear description of changes
 - Link any related issues
@@ -479,7 +557,7 @@ Have a significant feature idea not covered by existing tracks?
 - [ ] Tests pass locally
 - [ ] Documentation is updated if needed
 - [ ] Commit messages follow convention
-- [ ] PR targets `dev/v0.1.5` branch
+- [ ] PR targets `dev/v0.1.7` branch (or `main` if dev branch doesn't exist yet)
 
 ### PR Description Should Include
 
@@ -566,15 +644,20 @@ Every feature needs documentation! Here's how to decide where and what to write.
    - Location: `docs/source/quickstart/configuration.rst`
    - What to include: YAML examples, parameter descriptions
 
-3. ✅ **API Reference** - If feature changes Python API
+3. ✅ **Config Validator** - **REQUIRED if feature changes config schema**
+   - Location: `massgen/config_validator.py` and `massgen/backend/capabilities.py`
+   - What to include: Validation logic, error messages, tests
+   - **See**: [Updating the Config Validator](#updating-the-config-validator) section above
+
+4. ✅ **API Reference** - If feature changes Python API
    - Location: `docs/source/api/`
    - What to include: Docstrings, function signatures, examples
 
-4. ✅ **CHANGELOG.md** - What changed in this version
+5. ✅ **CHANGELOG.md** - What changed in this version
    - Location: Root directory
    - What to include: Brief description under "Added", "Changed", or "Fixed"
 
-5. ✅ **Examples** - **REQUIRED for every feature**
+6. ✅ **Examples** - **REQUIRED for every feature**
    - Location: `docs/source/examples/basic_examples.rst` or feature-specific example files
    - What to include: Runnable code showing feature in action
    - **Note**: Examples are ALWAYS required, even if you write a case study. Case studies showcase real-world usage; examples show basic functionality.
@@ -655,6 +738,7 @@ Every feature needs documentation! Here's how to decide where and what to write.
 |--------------|-------------|----------|-----------|
 | **User Guide** | Every feature | `docs/source/user_guide/` | ✅ Yes |
 | **Config Docs** | Config changes | `docs/source/quickstart/configuration.rst` | ✅ Yes |
+| **Config Validator** | Config schema changes | `massgen/config_validator.py` | ✅ Yes (if config changes) |
 | **API Docs** | API changes | `docs/source/api/` | ✅ Yes |
 | **CHANGELOG** | Every PR | `CHANGELOG.md` | ✅ Yes |
 | **Examples** | **Every feature** | `docs/source/examples/` | ✅ **ALWAYS** |