@michelabboud/visual-forge-mcp 0.6.1 → 0.9.0

package/CHANGELOG.md

@@ -5,6 +5,496 @@ All notable changes to Visual Forge MCP will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+---
+
+## [0.9.0] - 2026-01-16
+
+### Added 🧪 **Model Testing & Comparison System**
+
+#### Model Selection Workflow (Phase 1)
+- **Enhanced Provider Configuration**: `configure_provider` now returns available models
+  - Shows list of all models offered by the provider
+  - Displays model details: name, cost, description, capabilities
+  - Indicates which model is currently set as default
+  - Guides user to use `set_default_model` for customization
+- **New MCP Tool: `set_default_model`**
+  - Choose preferred model for a provider
+  - Validates model exists before saving
+  - Persists choice in `~/.visual-forge-mcp/user-config.json`
+  - Applies to all future generations unless overridden
+- **New MCP Tool: `get_model_info`**
+  - Retrieve detailed information about a specific model
+  - Shows capabilities, pricing, and description
+  - Displays test results if model was previously tested
+  - Helps users make informed model selection decisions
+- **User Configuration Manager**: `src/utils/user-config-manager.ts`
+  - Persistent storage for user preferences
+  - Default model per provider
+  - Model test results with quality scores
+  - Atomic file writes for crash safety
+
+#### Model Testing (Phase 2)
+- **ModelTester Utility**: `src/quality/model-tester.ts`
+  - Comprehensive testing framework for AI models
+  - Standard automated tests and custom prompt tests
+  - Multi-provider comparison with side-by-side results
+  - Quality scoring algorithm with weighted metrics
+
+- **Quality Scoring Algorithm**:
+  - **Sharpness (30%)**: Laplacian variance analysis for edge detection
+  - **Brightness (20%)**: Average brightness in optimal range (30-240)
+  - **Text Rendering (40%)**: Estimated OCR accuracy and text clarity
+  - **Color Accuracy (10%)**: Heuristic-based color validation
+  - **Overall Score**: Weighted average, pass threshold 60/100
+  - **Auto-Recording**: Test results saved for future reference
+
+- **Standard Test Prompt**:
+  - Professional quality validation image
+  - Multiple text elements (title, subtitle, detailed text)
+  - Geometric shapes (red circle, blue square, green triangle)
+  - Color gradient background (#1a365d to #0891b2)
+  - Technical diagram (simple flowchart)
+  - Designed to test sharpness, color accuracy, and text rendering
+
+- **New MCP Tool: `test_model`**
+  - Two modes:
+    - **Standard Test**: Automated quality validation with predefined elements
+    - **Custom Prompt**: Test with user's actual use case
+  - Cost permission flow (requires confirmation for paid models)
+  - Automatic quality scoring and pass/fail determination
+  - Test images saved to `generated-images/tests/`
+  - Results include quality metrics, generation time, and cost
+
+- **New MCP Tool: `compare_models`**
+  - Side-by-side comparison of multiple providers/models
+  - Same prompt tested across all selected models
+  - Automatic ranking by quality score
+  - Intelligent recommendation with reasoning
+  - Alternative suggestions for budget-conscious users
+  - Total cost and time tracking
+  - Detailed results per model with success/failure handling
+
+- **Permission Flow**:
+  - First call shows cost estimate and requires confirmation
+  - User confirms by calling again with `skipPermission: true`
+  - Prevents accidental spending on paid models
+  - Free models (Gemini, HuggingFace) can skip permission
+
+#### Files Added
+- `src/quality/model-tester.ts` - Model testing and comparison utility
+- `docs/design/model-selection-workflow.md` - Complete workflow specification
+- `docs/design/IMPLEMENTATION_STATUS.md` - Implementation tracking
+
+#### Files Modified
+- `src/server/mcp-server.ts`:
+  - Added `userConfigManager` import
+  - Added `ModelTester` instance
+  - Updated `configure_provider` handler to return models
+  - Added `handleSetDefaultModel()` handler
+  - Added `handleGetModelInfo()` handler
+  - Added `handleTestModel()` handler
+  - Added `handleCompareModels()` handler
+  - Added routing for 5 new tools
+  - Added 'zai' to all provider enums
+- `src/quality/index.ts` - Exported ModelTester
+- `src/utils/index.ts` - Exported UserConfigManager (from 0.8.0)
+
+### Benefits
+
+#### For Users
+- ✅ **Informed Decisions**: Test before committing to a model
+- ✅ **Quality Assurance**: Validate model performance on your use case
+- ✅ **Cost Awareness**: See estimated costs before generation
+- ✅ **Traceability**: Historical test results for comparison
+- ✅ **Flexibility**: Different models for different use cases
+- ✅ **Side-by-Side Comparison**: Objective ranking with recommendations
+
+#### For Workflows
+- ✅ **Standard Tests**: Quick automated validation of model quality
+- ✅ **Custom Tests**: Real-world testing with actual prompts
+- ✅ **Multi-Provider**: Compare 2-8 providers in single call
+- ✅ **Automatic Ranking**: Data-driven recommendations
+- ✅ **Error Handling**: Graceful failures don't block comparisons
+
+### Example Usage
+
+```typescript
+// 1. Configure provider and see models
+configure_provider({ provider: "zai", apiKey: "zai-..." })
+// Returns: List of available models with costs
+
+// 2. Set preferred model
+set_default_model({ provider: "zai", modelId: "glm-image" })
+
+// 3. Test with standard prompt (automated)
+test_model({
+  provider: "zai",
+  modelId: "glm-image",
+  useStandardTest: true
+})
+
+// 4. Test with custom prompt (real use case)
+test_model({
+  provider: "gemini",
+  modelId: "gemini-2.5-flash-image",
+  prompt: "AWS VPC architecture diagram with public/private subnets"
+})
+
+// 5. Compare multiple models
+compare_models({
+  prompt: "Technical diagram showing microservices architecture",
+  providers: [
+    { provider: "gemini", model: "gemini-2.5-flash-image" },
+    { provider: "zai", model: "glm-image" },
+    { provider: "huggingface", model: "black-forest-labs/FLUX.1-dev" }
+  ]
+})
+// Returns: Ranked results with recommendation
+```
+
+### Technical Details
+
+- **Test Image Storage**: `generated-images/tests/`
+- **Config Storage**: `~/.visual-forge-mcp/user-config.json`
+- **Quality Metrics**: Sharpness, brightness, text rendering, color accuracy
+- **Pass Threshold**: 60/100 overall score
+- **Concurrent Testing**: Up to 3 parallel model tests in comparison mode
+
+---
+
+## [0.8.0] - 2026-01-16
+
+### Added 🎯 **Multi-Model Architecture & Z.ai Provider**
+
+#### Multi-Model Architecture
+- **Provider-Model Separation**: Distinguish between providers (companies) and models (specific AI implementations)
+  - One provider can offer multiple models with different capabilities and pricing
+  - Example: OpenAI offers `gpt-image-1` (standard, $0.04) and `gpt-image-1-hd` (HD, $0.12)
+  - Example: Gemini offers `gemini-2.5-flash-image` (2K) and `gemini-2.5-flash-image-pro` (4K)
+  - Example: Replicate offers `flux-schnell` ($0.003), `flux-dev` ($0.025), `flux-pro` ($0.055)
+- **New Type System**:
+  - `ProviderModel` interface: Represents individual models with id, name, cost, capabilities
+  - `IImageProvider.getModels()`: Returns array of available models
+  - `IImageProvider.getDefaultModel()`: Returns provider's default model
+  - `IImageProvider.estimateCost(spec, modelId?)`: Model-aware cost estimation
+- **Backward Compatibility**:
+  - Optional `models` and `defaultModel` fields in `ProviderConfig`
+  - Legacy `model` field still supported for single-model providers
+  - Automatic fallback to legacy config if models array not provided
+- **Files Modified**:
+  - `src/types/provider.ts` - Added ProviderModel interface, updated IImageProvider
+  - `src/providers/base-provider.ts` - Implemented getModels(), getDefaultModel(), updated estimateCost()
+  - `src/providers/index.ts` - Updated all provider initializations
+  - `config/pricing.json` - Restructured to support models array per provider (v2.1.0)
+- **Benefits**:
+  - ✅ Future-proof architecture for providers with multiple model offerings
+  - ✅ Granular cost control per model
+  - ✅ Better model selection and capabilities reporting
+  - ✅ Full backward compatibility with existing code
+
+#### Z.ai GLM-Image Provider (8th Provider)
+- **New Provider**: Z.ai (Zhipu AI) with GLM-Image model
+  - **Model**: GLM-Image - 16B parameter hybrid autoregressive + diffusion model
+  - **Specialty**: Excellent for text-heavy diagrams, posters, and knowledge-dense images
+  - **Performance**: Beats Google Gemini on CVTG-2k text rendering benchmark (0.9116 vs 0.7788)
+  - **Pricing**: $0.015 per image (2nd cheapest paid option after Replicate $0.003)
+  - **Rate Limit**: 15 images/minute
+  - **Resolution**: Up to 2048x2048
+  - **Supported Formats**: PNG
+- **Files Added**:
+  - `src/providers/zai/zai-provider.ts` - Complete Z.ai provider implementation
+  - `src/types/generation.ts` - Added 'zai' to ProviderType
+- **Configuration**:
+  - Environment variable: `ZAI_API_KEY=zai-...`
+  - Runtime configuration via `configure_provider` MCP tool
+- **Features**:
+  - Custom prompt adaptation for text-heavy content
+  - Emphasizes accurate text rendering and clear labels
+  - Professional diagram focus
+  - Multilingual support
+- **Benefits**:
+  - ✅ Best-in-class text rendering for technical diagrams
+  - ✅ Cost-effective at $0.015/image
+  - ✅ Perfect for documentation with text-heavy visualizations
+
+#### Comprehensive Documentation
+- **New Guide**: `docs/guides/comprehensive-guide.md` - Complete user and developer reference
+  - **Description**: Overview of Visual Forge MCP features and use cases
+  - **Installation**: Step-by-step setup from prerequisites to MCP client configuration
+  - **Architecture**: Detailed system components and data flow diagrams
+  - **Environment Variables**: Complete reference for all configuration options
+  - **Provider & Model System**: Guide to 8 providers, model selection, capabilities
+  - **Usage Workflows**: Basic to advanced workflows with code examples
+  - **MCP Tools Reference**: All 13+ MCP tools with parameters and response examples
+  - **Testing**: Test infrastructure, running tests, writing tests, manual testing
+  - **Troubleshooting**: Common issues and debug strategies
+  - **Advanced Topics**: Custom providers, pricing config, backups, optimization
+- **Updated**:
+  - `README.md` - Added Z.ai provider, multi-model architecture feature, comprehensive guide link
+  - Version badge updated to 0.8.0
+  - Provider count updated to 8
+  - Added `ZAI_API_KEY` to environment variable examples
+
+### Changed
+
+#### Provider Priority Order
+- **New Default Priority**: Updated to prioritize cost-effectiveness
+  1. `replicate` - Cheapest ($0.003)
+  2. `zai` - 2nd cheapest ($0.015) ✨ NEW
+  3. `gemini` - Free tier
+  4. `huggingface` - Free tier
+  5. Others by cost
+
+#### Pricing Configuration
+- **Updated**: `config/pricing.json` version 2.1.0
+  - Added Z.ai provider with GLM-Image model
+  - Updated cost comparison to highlight Z.ai as 2nd cheapest
+  - Added recommended provider section (Z.ai for technical documentation)
+  - Complete model specifications for all providers
+
+### Fixed
+
+#### TypeScript Type Safety
+- **Fixed**: Handling of optional `models` field in ProviderConfig
+  - BaseProvider.init() now safely accesses models array
+  - BaseProvider.getDefaultModel() handles undefined models array
+  - BaseProvider.estimateCost() checks models existence before access
+  - ZaiProvider.generateImage() properly extracts prompt text
+- **Fixed**: Removed unused imports (ProviderModel, fs, path, fileURLToPath from ProviderFactory)
+- **Resolved**: All TypeScript compilation errors for multi-model architecture
+
+#### Backward Compatibility
+- **Ensured**: Legacy single-model providers continue to work
+  - `config.model` field still supported
+  - Automatic conversion to ProviderModel format
+  - Fallback to legacy cost if models array missing
+
+### Testing
+
+#### Test Results
+- **All Tests Passed**: 77 tests across 4 test suites
+  - ProviderFactory: 21 tests (provider initialization, selection, fallback, priority)
+  - All existing tests remain green with multi-model changes
+  - Backward compatibility verified
+
+---
+
+## [0.7.0] - 2026-01-16
+
+### Added 🎨 **Professional Image Generation Pipeline**
+
+#### Index-Based Directory Structure
+- **Collision-Free Directories**: Sequential index-based naming prevents filename conflicts
+  - Format: `001-filename/`, `002-filename/`, `003-filename/`
+  - Automatic index assignment and tracking via `IndexManager`
+  - Human-readable directory names with sanitized filenames
+  - Guaranteed unique directories even with identical source filenames
+- **Files Added**:
+  - `src/utils/index-manager.ts` - Sequential index management with persistence
+  - `src/utils/source-metadata.ts` - Per-directory metadata tracking
+- **Benefits**:
+  - ✅ No more directory collisions (e.g., `lessons/01-lesson.md` vs `docs/01-lesson.md`)
+  - ✅ Easy navigation with sequential prefixes
+  - ✅ Automatic cleanup of old indices on reset
+
+#### Per-Image Metadata & Logging System
+- **Comprehensive Metadata Tracking**: Each generated image gets complete metadata and logs
+  - `img-01.meta.json`: Prompts, generation data, quality scores, costs
+  - `img-01.log`: Timestamped generation log with all events
+  - Stored in `original/` subdirectory alongside pristine image
+- **Metadata Includes**:
+  - Full prompt (original, global context, enhanced final)
+  - Generation parameters (provider, model, timing, cost)
+  - Quality metrics (sharpness, brightness, dimensions, file size)
+  - Optimization results (formats generated, size reductions)
+  - Regeneration history (if auto-regenerated)
+- **Files Added**:
+  - `src/utils/image-metadata-manager.ts` - Metadata and log management
+- **Benefits**:
+  - ✅ Complete audit trail for every image
+  - ✅ Debug generation issues with full context
+  - ✅ Track costs and quality over time
+  - ✅ Reproduce exact generation conditions
+
+#### Multi-Format Image Optimization with Watermarking
+- **Automatic Format Generation**: Create multiple optimized formats from single source
+  - **WebP**: 94% size reduction, modern web use (default for markdown)
+  - **JPEG**: 85% size reduction, PDF export compatibility
+  - **PNG**: 70% size reduction with lossy palette mode (optional)
+  - **Original**: Pristine backup, NO watermark, always preserved
+- **Professional Watermarking**:
+  - SVG text overlay using Sharp composite
+  - Default: "Generated with Visual Forge MCP ©" at bottom-right
+  - Customizable: text, position, opacity, font size, color, font family
+  - Applied to web formats only (WebP, JPEG, PNG), never to original
+- **Directory Structure**:
+  ```
+  {provider}/
+  ├── original/
+  │   ├── img-01.png        (NO watermark, pristine)
+  │   ├── img-01.meta.json
+  │   └── img-01.log
+  ├── webp/
+  │   └── img-01.webp       (watermarked, 94% smaller)
+  ├── jpg/
+  │   └── img-01.jpg        (watermarked, 85% smaller)
+  └── png/
+      └── img-01.png        (watermarked, 70% smaller, optional)
+  ```
+- **Lossy PNG Optimization**:
+  - 8-bit color palette mode (256 colors)
+  - Floyd-Steinberg dithering for smooth gradients
+  - Near-perfect quality for technical diagrams/screenshots
+  - Configurable: quality, palette size, dithering amount
+- **Files Added**:
+  - `src/utils/multi-format-optimizer.ts` - Format generation and watermarking
+- **Configuration**:
+  ```bash
+  VF_GENERATE_PNG=true          # Enable PNG generation
+  VF_PNG_QUALITY=85             # Quality (1-100)
+  VF_PNG_PALETTE=true           # Lossy mode
+  VF_PNG_COLORS=256             # Palette size
+  VF_PNG_DITHER=1.0             # Dithering (0-1)
+  ```
+- **Benefits**:
+  - ✅ Optimized web delivery (WebP default)
+  - ✅ PDF compatibility (JPEG)
+  - ✅ Professional watermarking
+  - ✅ Pristine originals preserved
+  - ✅ Massive file size savings (70-94% reduction)
+
+#### Quality Validation with OCR & Auto-Regeneration
+- **OCR Text Detection**: Tesseract.js integration for text quality validation
+  - Detects gibberish and malformed text in generated images
+  - Analyzes OCR confidence scores
+  - Identifies common AI text rendering errors
+  - Configurable gibberish ratio threshold (default: 30%)
+- **Image Quality Metrics**:
+  - **Sharpness**: Laplacian variance edge detection (min: 50)
+  - **Brightness**: Average pixel brightness (30-240 range)
+  - **Dimensions**: Width/height verification
+  - **File Size**: Valid image size check (>10KB, <10MB)
+  - **Overall Score**: 0-100 weighted quality score
+- **Automatic Regeneration**:
+  - Retry loop with configurable attempts (default: 3)
+  - Triggers on quality validation failure
+  - Logs each attempt with quality scores
+  - Saves regeneration metadata for debugging
+  - Configurable via environment variables
+- **Files Added**:
+  - `src/utils/quality-validator.ts` - OCR and quality analysis
+- **Configuration**:
+  ```bash
+  VF_AUTO_REGENERATE=true       # Enable auto-retry
+  VF_MAX_RETRIES=3              # Max attempts
+  VF_QUALITY_VALIDATION=true    # Enable quality checks
+  ```
+- **Benefits**:
+  - ✅ Catches AI text rendering failures
+  - ✅ Ensures high-quality outputs
+  - ✅ Automatic retry without user intervention
+  - ✅ Complete quality audit trail
+
+#### Enhanced Prompt Engineering
+- **Type-Specific Enhancement**: Automatic prompt optimization by image type
+  - Diagrams: "technical diagram, clean lines, no text or labels"
+  - Flowcharts: "professional flowchart, clear arrows, minimal text"
+  - Icons: "minimalist design, single object, flat style"
+  - Hero images: "high quality, professional, no text overlay"
+  - Screenshots: "clean interface, realistic UI, no watermarks"
+- **Text Prevention Instructions**: Minimizes AI text rendering errors
+  - Adds explicit "no text", "no labels", "no watermarks" directives
+  - Prevents gibberish and malformed text in generated images
+  - Type-aware enhancement based on image purpose
+- **Files Added**:
+  - `src/utils/prompt-enhancer.ts` - Type-specific prompt optimization
+- **Integration**:
+  - Automatic enhancement in `BaseProvider.generate()`
+  - Applied before provider-specific adaptations
+  - Logged for debugging and reproducibility
+- **Benefits**:
+  - ✅ Better image quality with minimal text artifacts
+  - ✅ Type-appropriate styling and composition
+  - ✅ Reduced need for manual prompt tweaking
+  - ✅ Consistent enhancement across all providers
+
+### Changed
+- **BaseProvider**: Integrated quality validation, auto-regeneration, and multi-format optimization
+  - Now orchestrates complete generation pipeline: enhance → generate → optimize → validate → retry
+  - Updated `generate()` method with retry loop
+  - Automatic format selection for markdown (WebP default)
+  - Comprehensive metadata and logging
+- **Filename Sanitizer**: Made async to support index-based directories
+  - `createImagePath()` now returns `Promise<string>`
+  - `createImageDirectory()` creates indexed directories
+  - All dependent code updated for async paths
+- **MCP Server**: Added `initializePathManager()` call on startup
+  - Initializes IndexManager for session persistence
+  - Ensures index tracking across server restarts
+
+### Dependencies
+- **Optional**: `tesseract.js` for OCR quality validation (install with `npm install tesseract.js`)
+  - Graceful degradation if not installed
+  - Quality validation continues without OCR
+
+### Documentation
+- Updated `src/utils/index.ts` to export all new utilities
+- Comprehensive TSDoc comments in all new modules
+- Environment variable documentation in code comments
+
+### Technical Details
+
+#### Files Created (5)
+1. `src/utils/index-manager.ts` (188 lines)
+2. `src/utils/source-metadata.ts` (215 lines)
+3. `src/utils/image-metadata-manager.ts` (267 lines)
+4. `src/utils/multi-format-optimizer.ts` (553 lines)
+5. `src/utils/quality-validator.ts` (518 lines)
+6. `src/utils/prompt-enhancer.ts` (176 lines)
+
+#### Files Modified (4)
+1. `src/utils/filename-sanitizer.ts` - Async path creation with indexing
+2. `src/providers/base-provider.ts` - Integrated all new features
+3. `src/server/mcp-server.ts` - Initialize IndexManager
+4. `src/utils/index.ts` - Export new utilities
+
+#### Test Coverage
+- All 77 existing tests still passing
+- Build successful with strict TypeScript checks
+- No breaking changes to existing workflows
+
+### Backward Compatibility
+- ✅ Fully backward compatible
+- ✅ All new features activated automatically
+- ✅ Graceful degradation (OCR optional)
+- ✅ Existing workflows unchanged
+- ✅ Environment variables optional (sensible defaults)
+
+### User Impact ⭐
+**Before (v0.6.1)**
+- ❌ Directory name collisions possible
+- ❌ No metadata or logs for generated images
+- ❌ Single PNG format only (large files)
+- ❌ No watermarking
+- ❌ No quality validation
+- ❌ Manual regeneration required
+- ❌ Basic prompts only
+
+**After (v0.7.0)**
+- ✅ Collision-free indexed directories
+- ✅ Complete metadata and generation logs
+- ✅ Multi-format optimization (WebP, JPEG, PNG)
+- ✅ Professional watermarking
+- ✅ Automatic quality validation
+- ✅ Auto-regeneration on quality failure
+- ✅ Enhanced prompts with type-specific optimization
+- ✅ 70-94% file size reduction
+- ✅ Pristine originals preserved
+
+---
+
 ## [0.6.1] - 2026-01-15
 
 ### Fixed