jekyll-imgflow 0.1.5

data/README.md

@@ -0,0 +1,214 @@
+# ImgFlow
+
+A [Jekyll](https://jekyllrb.com/) plugin for automatic image optimization with multiple providers and formats.
+
+## 🚀 Quick Start
+
+```bash
+# Add to your Gemfile
+gem 'jekyll-imgflow'
+
+# Install
+bundle install
+
+# Configure (_config.yml)
+imgflow:
+  originals: "assets/images/originals"
+  output: "assets/images/optimized"
+  sizes: {sm: 400, md: 800, lg: 1200}
+  formats: [webp, avif, jpg]
+
+# Use in templates
+{% imgflow photo.jpg resize width:800 %}
+```
+
+## ✨ Key Features
+
+- **Multiple Providers**: [Sharp](https://github.com/lovell/sharp), [ImageMagick](https://github.com/ImageMagick/ImageMagick), [LibVips](https://github.com/libvips/libvips), [Imgproxy](https://github.com/imgproxy/imgproxy), [Weserv](https://github.com/weserv/images), [Flyimg](https://github.com/flyimg/flyimg)
+- **Modern Formats**: WebP, AVIF, JPG, PNG with automatic fallbacks
+- **Picture Tag Support**: Full [Jekyll Picture Tag](https://github.com/robwierzbowski/jekyll_picture_tag) compatibility
+- **Build-Time Processing**: Pre-generate images for optimal performance
+- **Docker Services**: Ready-to-use image processing containers
+- **Parallel Testing**: Fast test execution with CPU optimization
+
+## 📚 Documentation
+
+### Getting Started
+- **[Installation Guide](docs/installation.md)** - Detailed setup instructions
+- **[Basic Usage](docs/usage/)** - Core functionality and examples
+- **[Picture Tag Migration](docs/picture_tag_migration.md)** - Migrate from Picture Tag
+
+### Configuration
+- **[Provider Setup](docs/providers.md)** - Configure image providers
+- **[Docker Services](docs/docker.md)** - Using Docker containers
+- **[Presets System](docs/usage/presets.md)** - Predefined configurations
+
+### Development
+- **[Development Guide](docs/development.md)** - Contributing and setup
+- **[Testing Guide](docs/testing.md)** - Running and writing tests
+- **[Scripts Reference](docs/scripts.md)** - Development utilities
+
+### Performance
+- **[Provider Comparison](docs/providers.md)** - Performance benchmarks
+- **[Parallel Testing](docs/parallel_testing.md)** - Fast test execution
+
+## 🏗️ Architecture Overview
+
+ImgFlow has two processing flows: **Build-Time** (pre-generation) and **Runtime** (on-demand).
+
+**See:** [ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed component architecture and data flow
+
+### Quick Overview
+- **Build-Time Flow** - Pre-generate images during Jekyll build
+- **Runtime Flow** - Process images on-demand during template rendering
+- **Providers** - Image processing backends (CLI tools, HTTP APIs)
+- **Tags** - Jekyll template tags for image optimization
+
+## 🎯 Provider Support
+
+### HTTP API Services (Recommended)
+- **[Imgproxy](https://github.com/imgproxy/imgproxy)** - Fast, reliable (Port 33001)
+- **[Weserv](https://github.com/weserv/images)** - Battle-tested (Port 33007)  
+- **[Flyimg](https://github.com/flyimg/flyimg)** - On-the-fly processing (Port 33008)
+
+### CLI Tools (Local)
+- **[Sharp](https://github.com/lovell/sharp)** - Fastest (Node.js/libvips)
+- **[ImageMagick](https://github.com/ImageMagick/ImageMagick)** - Feature-rich
+- **[LibVips](https://github.com/libvips/libvips)** - Memory efficient
+
+**See:** [providers.md](docs/providers.md) for detailed provider comparison and setup
+
+## 🚀 Quick Commands
+
+```bash
+# Development
+bundle exec jekyll serve              # Start development server
+bundle exec jekyll build              # Build site
+
+# Testing
+rake quick                           # Quick checks (style + tests)
+rake test                            # All tests
+rake parallel:test                   # Parallel testing (faster)
+
+# Docker Services
+docker-compose -f docker-compose.test.yml up -d  # Start services
+rake check_services                  # Verify services
+```
+
+**See:** [testing.md](docs/testing.md) for comprehensive testing guide, [scripts.md](docs/scripts.md) for development utilities, and [rake.md](docs/rake.md) for all available Rake tasks
+
+## 📦 Installation
+
+### Quick Setup
+```bash
+# Add to Gemfile
+gem 'jekyll-imgflow'
+
+# Install
+bundle install
+
+# Configure (_config.yml)
+imgflow:
+  originals: "assets/images/originals"
+  output: "assets/images/optimized"
+  sizes: {sm: 400, md: 800, lg: 1200}
+  formats: [webp, avif, jpg]
+
+# Use in templates
+{% imgflow photo.jpg resize width:800 %}
+```
+
+**See:** [installation.md](docs/installation.md) for detailed installation instructions and configuration options
+
+## 🐳 Docker Setup (Recommended)
+
+```bash
+# Start all services
+docker-compose -f docker-compose.test.yml --env-file .env.test up -d
+
+# Check services
+rake check_services
+
+# Stop services
+docker-compose -f docker-compose.test.yml down
+```
+
+**See:** [docker.md](docs/docker.md) for detailed Docker configuration and services
+
+## 🔄 Picture Tag Migration
+
+ImgFlow provides full compatibility with [Jekyll Picture Tag](https://github.com/robwierzbowski/jekyll_picture_tag):
+
+```bash
+# Preview migration
+bin/imgflow_migrate_presets --preview
+
+# Migrate presets
+bin/imgflow_migrate_presets
+```
+
+**See:** [Picture Tag Migration Guide](docs/picture_tag_migration.md)
+
+## 📊 Performance
+
+Latest benchmarks show **Sharp** as the fastest provider:
+
+| Provider | Time (s) | Speed |
+|----------|----------|-------|
+| [Sharp](https://github.com/lovell/sharp) | 13.97 | 🏆 Fastest |
+| [LibVips](https://github.com/libvips/libvips) | 21.89 | Very Fast |
+| [Weserv](https://github.com/weserv/images) | 30.19 | Fast |
+| [ImageMagick](https://github.com/ImageMagick/ImageMagick) | 30.62 | Fast |
+| [Imgproxy](https://github.com/imgproxy/imgproxy) | 31.27 | Fast |
+
+**See:** [Provider Comparison](docs/providers.md) for detailed benchmarks
+
+## 🛠️ Development
+
+### Quick Setup
+```bash
+# Clone and setup
+git clone <repository>
+cd jekyll-imgflow
+bundle install
+
+# Download test images
+rake download_test_images
+
+# Run tests
+rake quick
+```
+
+### Key Files
+- **`lib/jekyll-imgflow.rb`** - Main module
+- **`lib/jekyll-imgflow/imgflow_tag.rb`** - Main tag implementation
+- **`lib/jekyll-imgflow/html_generator.rb`** - HTML generation
+- **`lib/jekyll-imgflow/picture_tag_adaptor.rb`** - Picture Tag compatibility
+
+**See:** [development.md](docs/development.md) for complete development setup, scripts, and workflows
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Add tests for new functionality
+5. Run `rake quality` to check all quality metrics
+6. Submit a pull request
+
+**See:** [development.md](docs/development.md) for detailed guidelines and [scripts.md](docs/scripts.md) for development utilities
+
+## 📄 License
+
+This project is licensed under the AGPL-3.0-or-later License - see the [LICENSE](LICENSE) file for details.
+
+## 🔗 Links
+
+- **[Documentation](docs/README.md)** - Complete documentation
+- **[GitHub Repository](https://github.com/gundestrup/jekyll-imgflow)** - Source code
+- **[Issue Tracker](https://github.com/gundestrup/jekyll-imgflow/issues)** - Report issues
+- **[Releases](https://github.com/gundestrup/jekyll-imgflow/releases)** - Latest versions
+
+---
+
+**ImgFlow** - Fast, flexible image optimization for Jekyll sites 🚀

data/lib/jekyll-imgflow/batch_manager.rb

@@ -0,0 +1,263 @@
+# frozen_string_literal: true
+
+require_relative "filename_generator"
+
+module JekyllImgFlow
+  # BatchManager - manages batch processing of image operations
+  # Queues tasks and processes them efficiently
+  class BatchManager
+    attr_reader :queue, :completed, :failed
+
+    def initialize(operation_processor)
+      @operation_processor = operation_processor
+      @filename_generator = FilenameGenerator.new
+      @queue = []
+      @completed = []
+      @failed = []
+      @mutex = Mutex.new
+    end
+
+    # Add a task to the batch queue
+    # @param task [Hash] Task definition
+    # @option task [String] :original_name Original image filename
+    # @option task [Symbol] :operation_type Type of operation
+    # @option task [String] :input_path Path to input image
+    # @option task [String] :output_path Path to output image
+    # @option task [Hash] :params Operation parameters
+    # @option task [Symbol] :version_type :default or :specialized
+    # @option task [String] :page_path Page using this image
+    def add_task(task)
+      @mutex.synchronize do
+        @queue << task
+      end
+    end
+
+    # Add multiple tasks to the queue
+    # @param tasks [Array<Hash>] Array of task definitions
+    def add_tasks(tasks)
+      @mutex.synchronize do
+        @queue.concat(tasks)
+      end
+    end
+
+    # Process all queued tasks
+    # @param parallel [Boolean] Whether to process in parallel (future enhancement)
+    # @return [Hash] Results summary
+    def process_all(parallel: false)
+      start_time = Time.now
+
+      if parallel
+        # Future: implement parallel processing
+        process_parallel
+      else
+        process_sequential
+      end
+
+      end_time = Time.now
+      duration = end_time - start_time
+
+      {
+        total: @queue.length + @completed.length + @failed.length,
+        completed: @completed.length,
+        failed: @failed.length,
+        duration: duration
+      }
+    end
+
+    # Process tasks sequentially
+    def process_sequential
+      while (task = next_task)
+        process_task(task)
+      end
+    end
+
+    # Process tasks in parallel (future enhancement)
+    def process_parallel
+      # Future: implement thread pool for parallel processing
+      # For now, fall back to sequential
+      process_sequential
+    end
+
+    # Get the next task from the queue
+    # @return [Hash, nil] Next task or nil if queue is empty
+    def next_task
+      @mutex.synchronize do
+        @queue.shift
+      end
+    end
+
+    # Process a single task
+    # @param task [Hash] Task to process
+    def process_task(task)
+      # Check if processing is needed
+      # Check if output is up-to-date
+      if task[:skip_if_exists] && File.exist?(task[:output_path]) && !@operation_processor.needs_processing?(
+        task[:input_path],
+        task[:output_path],
+        task[:params]
+      )
+        Jekyll.logger.debug "⏭️  Skipping #{task[:original_name]} - already up-to-date"
+        mark_completed(task, :skipped)
+        return
+      end
+
+      # Process the operation
+      operation = {
+        type: task[:operation_type],
+        params: task[:params]
+      }
+
+      Jekyll.logger.debug "🔍 BatchManager: Processing task - input: #{task[:input_path]}, original: #{task[:original_name]}"
+      result = @operation_processor.process_operation(
+        task[:original_name],
+        operation,
+        task[:input_path]
+      )
+      Jekyll.logger.debug "🔍 BatchManager: OperationProcessor returned - result: #{result}"
+
+      # Store result for manifest registration
+      task[:result_path] = result
+
+      mark_completed(task, :processed, result)
+      Jekyll.logger.debug "✅ Processed: #{task[:original_name]} → #{File.basename(result)}"
+    rescue StandardError => e
+      mark_failed(task, e)
+      Jekyll.logger.error "❌ Failed: #{task[:original_name]} - #{e.message}"
+    end
+
+    # Mark a task as completed
+    # @param task [Hash] Completed task
+    # @param status [Symbol] Completion status (:processed or :skipped)
+    # @param result [String] Result path
+    def mark_completed(task, status, result = nil)
+      @mutex.synchronize do
+        @completed << {
+          task: task,
+          status: status,
+          result: result,
+          completed_at: Time.now
+        }
+      end
+    end
+
+    # Mark a task as failed
+    # @param task [Hash] Failed task
+    # @param error [Exception] Error that occurred
+    def mark_failed(task, error)
+      @mutex.synchronize do
+        @failed << {
+          task: task,
+          error: error.message,
+          backtrace: error.backtrace&.first(5),
+          failed_at: Time.now
+        }
+      end
+    end
+
+    # Get queue status
+    # @return [Hash] Current queue status
+    def status
+      @mutex.synchronize do
+        {
+          queued: @queue.length,
+          completed: @completed.length,
+          failed: @failed.length,
+          total: @queue.length + @completed.length + @failed.length
+        }
+      end
+    end
+
+    # Clear all queues
+    def clear
+      @mutex.synchronize do
+        @queue.clear
+        @completed.clear
+        @failed.clear
+      end
+    end
+
+    # Get failed tasks for retry
+    # @return [Array<Hash>] Array of failed tasks
+    def failed_tasks
+      @mutex.synchronize do
+        @failed.map { |f| f[:task] }
+      end
+    end
+
+    # Retry failed tasks
+    # @return [Integer] Number of tasks added back to queue
+    def retry_failed
+      tasks = failed_tasks
+      @mutex.synchronize do
+        @failed.clear
+        @queue.concat(tasks)
+      end
+      tasks.length
+    end
+
+    # Build batch tasks for default image versions
+    # @param original_name [String] Original image filename
+    # @param input_path [String] Path to original image
+    # @param config [Config] ImgFlow configuration
+    # @param site [Jekyll::Site] Jekyll site object
+    # @return [Array<Hash>] Array of tasks
+    def self.build_default_tasks(original_name, input_path, config, _site)
+      tasks = []
+
+      # Use FilenameGenerator for consistent naming
+      filename_generator = JekyllImgFlow::FilenameGenerator.new
+      path_resolver = JekyllImgFlow::PathResolver.new(config)
+
+      config.sizes.each_value do |width|
+        config.formats.each do |format|
+          # Use FilenameGenerator to generate proper filename
+          operations = { width: width, format: format, quality: config.quality }
+          filename = filename_generator.generate_filename(original_name, operations)
+          # Write to _site during build (after Jekyll copies assets)
+          output_path = path_resolver.resolve_output_path(filename)
+
+          tasks << {
+            original_name: original_name,
+            operation_type: :resize,
+            input_path: input_path,
+            output_path: output_path,
+            params: {
+              width: width,
+              format: format,
+              quality: config.quality
+            },
+            version_type: :default,
+            page_path: nil, # Default versions not tied to specific page
+            skip_if_exists: true
+          }
+        end
+      end
+
+      tasks
+    end
+
+    # Build batch tasks for specialized image version
+    # @param original_name [String] Original image filename
+    # @param input_path [String] Path to original image
+    # @param operations [Array<Hash>] Operations to apply
+    # @param output_path [String] Path to output image
+    # @param page_path [String] Page using this image
+    # @return [Hash] Task definition
+    def self.build_specialized_task(original_name, input_path, operations, output_path, page_path)
+      # For now, assume single operation
+      # Future: support chaining multiple operations
+      operation = operations.first
+
+      {
+        original_name: original_name,
+        operation_type: operation[:type],
+        input_path: input_path,
+        output_path: output_path,
+        params: operation[:params],
+        version_type: :specialized,
+        page_path: page_path,
+        skip_if_exists: false
+      }
+    end
+  end
+end

data/lib/jekyll-imgflow/build_time_processor.rb

@@ -0,0 +1,152 @@
+# frozen_string_literal: true
+
+module JekyllImgFlow
+  # BuildTimeProcessor - processes images during Jekyll build
+  # Generates default image versions for all originals
+  class BuildTimeProcessor
+    def initialize(site)
+      @site = site
+      @config = Config.new(site)
+      @manifest = ManifestManager.new(site)
+      @path_resolver = PathResolver.new(@config)
+      @registry = ProviderRegistry.new(@config)
+
+      # Initialize operation processor with current provider
+      provider = @registry.current_provider
+      @operation_processor = OperationProcessor.new(provider, @path_resolver, @manifest, @config)
+
+      # Initialize batch manager
+      @batch_manager = BatchManager.new(@operation_processor)
+
+      # Store all components in site for sharing with ImgFlow tag
+      return if @site.imgflow_components
+
+      filename_generator = JekyllImgFlow::FilenameGenerator.new
+
+      @site.imgflow_components = {
+        config: @config,
+        manifest: @manifest,
+        path_resolver: @path_resolver,
+        filename_generator: filename_generator,
+        registry: @registry,
+        provider: provider,
+        operation_processor: @operation_processor
+      }
+    end
+
+    # Process all changed images in originals folder
+    def process_changed_images
+      Jekyll.logger.info "🔄 BuildTimeProcessor: Processing changed images"
+
+      # Get all original images
+      original_paths = find_original_images
+      Jekyll.logger.info "📸 Found #{original_paths.length} original images"
+
+      # Build tasks for each original image
+      original_paths.each do |path|
+        original_name = File.basename(path)
+
+        # Check if needs processing (file changed or provider changed)
+        if needs_processing?(path)
+          Jekyll.logger.info "🔧 Queuing default versions for: #{original_name}"
+
+          # Build default tasks
+          tasks = BatchManager.build_default_tasks(original_name, path, @config, @site)
+          @batch_manager.add_tasks(tasks)
+        else
+          Jekyll.logger.debug "⏭️  Skipping #{original_name} - already up-to-date"
+        end
+      end
+
+      # Process all queued tasks
+      results = @batch_manager.process_all
+
+      # Register completed tasks in manifest
+      register_completed_tasks(@batch_manager.completed)
+
+      # Save manifest so it's available when ImgflowTag runs
+      @manifest.save
+
+      Jekyll.logger.info "✅ BuildTimeProcessor complete: #{results[:completed]} completed, #{results[:failed]} failed"
+
+      results
+    end
+
+    private
+
+    # Register completed tasks in manifest
+    # @param completed_tasks [Array<Hash>] Array of completed tasks from BatchManager
+    def register_completed_tasks(completed_tasks)
+      completed_tasks.each do |task|
+        next unless task[:task] && task[:result]
+
+        # Get provider name
+        provider_name = @registry.current_provider.class.name.split("::").last.downcase
+
+        # Convert absolute path to relative for manifest storage
+        absolute_path = task[:result]
+        relative_path = absolute_path.sub(@site.dest, "")
+
+        # Register the version in manifest
+        @manifest.register_version(
+          task[:task][:original_name],
+          relative_path,
+          task[:task][:params],
+          :default, # BuildTimeProcessor creates default versions
+          nil, # page_path (default versions aren't tied to specific pages)
+          nil, # file_digest
+          provider_name
+        )
+      end
+
+      Jekyll.logger.debug "📝 Registered #{completed_tasks.length} completed tasks in manifest"
+    end
+
+    # Find all original images in originals folder
+    def find_original_images
+      # Use PathResolver to get the originals directory path
+      originals_dir = File.join(@site.source, @config.originals)
+      input_formats = @config.input_formats
+
+      Dir.glob(File.join(originals_dir, "**", "*.{#{input_formats.join(',')}}"))
+    end
+
+    # Check if image needs processing
+    # @param path [String] Path to original image
+    # @return [Boolean] True if processing needed
+    def needs_processing?(path)
+      original_name = File.basename(path)
+
+      # If no versions exist, needs processing
+      return true unless @manifest.versions?(original_name)
+
+      # Check if original file was modified
+      versions = @manifest.get_versions(original_name)
+      default_versions = versions["default"] || []
+
+      return true if default_versions.empty?
+
+      # Get the most recent version creation time
+      latest_version = default_versions.max_by { |v| v["created_at"] || 0 }
+      version_time = latest_version["created_at"]
+
+      # If original is newer than versions, needs processing
+      return true unless version_time
+
+      original_mtime = File.mtime(path).to_i
+      return true if original_mtime > version_time
+
+      # No timestamp means we need to process
+
+      # Check if provider changed
+      current_provider = @registry.current_provider.class.provider_name
+      version_providers = default_versions.map { |v| v["provider"] }
+      version_providers.uniq!
+
+      return true unless version_providers.include?(current_provider)
+
+      # Up-to-date
+      false
+    end
+  end
+end