jackdaw 1.0.0

data/TODO.md

@@ -0,0 +1,167 @@
+# Jackdaw - Static Site Generator TODO
+
+## Project Overview
+A fast, incremental static site generator with folder-based organization using Thor for CLI.
+
+---
+
+## Phase 1: Project Setup
+- [x] Add Thor gem dependency to gemspec
+- [x] Add development dependencies (rspec, rubocop, etc.)
+- [x] Create basic CLI structure in exe/jackdaw
+- [x] Set up Thor CLI class structure
+- [x] Add any additional dependencies (for markdown parsing, file watching, etc.)
+
+## Phase 2: Core File System Structure
+- [x] Define standard project structure (site/ directory)
+- [x] Create file watcher/tracker for incremental builds
+- [x] Implement directory scanner
+- [x] Create file type registry (.page, .post, etc.)
+- [x] Build metadata extractor from file structure (path, name, dates)
+- [x] Extract metadata from first heading as title (H1)
+- [x] Implement content type handlers
+  - [x] Page handler
+  - [x] Post handler
+  - [x] Asset handler
+
+## Phase 3: Template System
+- [x] Design template engine integration (ERB)
+- [x] Implement template loader from templates/ directory
+- [x] Create template context builder
+- [x] Add layout/partial support
+- [x] Implement template caching for performance
+
+## Phase 4: Content Processing
+- [x] Implement markdown parser integration
+- [x] Create content preprocessor pipeline
+- [x] Extract metadata from filename (dates, slugs)
+- [x] Extract title from first H1 in content
+- [x] Derive metadata from folder structure and hierarchy
+- [x] Implement content transformation pipeline
+- [x] Add syntax highlighting support (Rouge)
+
+## Phase 5: Build System
+- [x] Design dependency graph for incremental builds
+- [x] Implement file change detection
+- [x] Create build cache system
+- [x] Build output directory management (public/)
+- [x] Implement parallel processing for speed
+- [x] Add build statistics/reporting
+- [x] Create clean build option
+
+## Phase 6: CLI Commands - Build
+- [x] Implement `jackdaw build` command
+  - [x] Full build option
+  - [x] Incremental build (default)
+  - [x] Watch mode option (deferred to serve command)
+  - [x] Verbose/debug output
+  - [x] Clean option
+- [x] Add build configuration loading (skipped - convention over configuration)
+- [x] Implement error handling and reporting
+
+## Phase 7: CLI Commands - Serve
+- [x] Implement `jackdaw serve` command
+- [x] Choose and integrate web server (Puma with Rack)
+- [x] Add live reload support (polling-based with JavaScript)
+- [x] Implement file watching for auto-rebuild (Listen gem)
+- [x] Configure port and host options
+- [x] Add middleware for development (StaticFileServer + LiveReloadMiddleware)
+- [x] Beautiful CLI output with rebuild stats
+- [x] Graceful shutdown handling (Ctrl+C)
+
+## Phase 8: CLI Commands - Create
+- [x] Implement `jackdaw create <template> <name>` command
+- [x] Create template scaffolding system
+- [x] Add default built-in templates:
+  - [x] `page` template (basic page.html.erb)
+  - [x] `blog` template (blog post with date handling)
+- [x] Support arbitrary user-defined templates in templates/ directory
+- [x] Discover available templates from templates/ folder
+- [x] Add date/timestamp handling for posts (auto-prefix for blog/post/article/news)
+- [x] Allow users to add custom templates by creating template files
+- [x] Implement `jackdaw template list` command
+- [x] Add --dated and --no-date flags for override control
+- [x] Support nested paths (e.g., `jackdaw create page company/about`)
+- [x] Validate template existence before creating files
+- [x] Add Builder warnings for missing templates
+
+## Phase 9: Configuration System
+- [x] Skipped - Maintaining config-less, convention-over-configuration approach
+
+## Phase 10: Asset Handling
+- [x] Implement asset copying
+- [x] Keeping minimal - assets copied as-is for speed and simplicity
+
+## Phase 11: Performance Optimization
+- [x] Profile build performance
+- [x] Implement multi-threading for file processing (Parallel gem)
+- [x] Optimize file I/O operations (incremental builds with mtime checks)
+- [ ] Add memory usage optimization
+- [x] Benchmark against common static site generators
+  - Small (30 files): 164 files/sec clean, 5821 files/sec incremental
+  - Medium (150 files): 428 files/sec clean, 12343 files/sec incremental
+  - Large (600 files): 693 files/sec clean, 16280 files/sec incremental
+  - Incremental builds: 6-18x faster than clean builds
+- [x] Cache template compilation (ERB caching in Renderer)
+- [x] Optimize dependency graph traversal (mtime-based change detection)
+
+## Phase 12: Documentation
+- [x] Write comprehensive README with quickstart guide
+- [x] Document CLI commands and options (in README)
+- [x] Create project structure guide (in README)
+- [x] Write template documentation (in README)
+- [x] Add configuration reference (N/A - convention-based, no config needed)
+- [x] Add troubleshooting guide (in README)
+- [x] Consolidate all require statements to jackdaw.rb
+- [x] Add comprehensive code comments to jackdaw.rb and project.rb
+- [ ] Add code comments to remaining core files
+- [ ] Create example site (basic example already created by `jackdaw new`)
+
+## Phase 13: Testing
+- [x] Unit tests for core components (Project, FileTypes, Scanner)
+- [x] Unit tests for Renderer (21 tests)
+- [x] Integration tests for Builder (18 tests)
+- [x] End-to-end build tests (10 comprehensive workflow tests)
+- [x] Test helper infrastructure with fixtures
+- [x] Performance benchmarks (completed in Phase 11)
+- **Total: 99 passing tests, 0 failures**
+
+## Phase 14: Polish & Release
+- [x] Code cleanup and refactoring (reduced complexity in Build command)
+- [x] RuboCop compliance (all files passing, 0 offenses)
+- [x] Error message improvements (beautiful CLI output with colors)
+- [x] Add progress indicators (build stats, rebuild notifications)
+- [x] Version 1.0.0 release preparation
+  - [x] Updated version to 1.0.0
+  - [x] Created comprehensive CHANGELOG.md
+  - [x] Updated gemspec with proper metadata
+  - [x] All 99 tests passing
+- [ ] Publish to RubyGems (deferred)
+
+## Phase 15: Essential Web Features
+- [ ] RSS/Atom feed generation (for blog posts)
+- [ ] Sitemap.xml generation (for SEO)
+- [ ] SEO meta tag helpers (Open Graph, Twitter Cards)
+
+---
+
+## Nice-to-Have Features (Future)
+- [ ] Search index generation (JSON for client-side search)
+- [ ] Git integration (simple deploy command wrapper)
+
+## Won't Implement (Against Project Philosophy)
+- ~~Multiple template engine support~~ - ERB is sufficient, adds complexity
+- ~~Image lazy loading helpers~~ - Can be done in templates/CSS
+- ~~i18n/l10n support~~ - Too complex, users can handle with folder structure
+- ~~CloudFlare/Netlify deployment helpers~~ - Too specific, manual deploy is simple enough
+
+---
+
+## Technical Decisions to Make
+- [x] Template engine: ERB (built-in, fast, zero dependencies)
+- [x] Markdown parser: Kramdown (pure Ruby, no compilation issues)
+- [x] Server: Puma (fast, industry standard)
+- [x] File watching: Listen gem (reliable, cross-platform)
+- [ ] Config format: YAML vs TOML vs Ruby DSL? (may skip - convention over configuration)
+- [x] Incremental build strategy: Timestamp-based (mtime checks)
+- [x] Metadata approach: Convention-based (NO frontmatter, derive from structure/content)

data/benchmark.rb

@@ -0,0 +1,235 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'bundler/setup'
+require 'benchmark'
+require 'fileutils'
+require_relative 'lib/jackdaw'
+
+# Benchmark Jackdaw performance
+class JackdawBenchmark
+  BENCHMARK_DIR = 'benchmark-site.site'
+  SIZES = {
+    small: { pages: 10, blog_posts: 20 },
+    medium: { pages: 50, blog_posts: 100 },
+    large: { pages: 100, blog_posts: 500 }
+  }.freeze
+
+  def initialize(size: :medium)
+    @size = size
+    @config = SIZES[size]
+    @project = Jackdaw::Project.new(BENCHMARK_DIR)
+  end
+
+  def run
+    puts "\n#{colorize('🚀 Jackdaw Performance Benchmark', :bold, :magenta)}"
+    puts colorize("Site size: #{@size} (#{@config[:pages]} pages, #{@config[:blog_posts]} blog posts)", :cyan)
+    puts colorize('=' * 70, :cyan)
+
+    cleanup
+    create_test_site
+    run_benchmarks
+    cleanup
+
+    puts "\n#{colorize('✓ Benchmark complete!', :green)}"
+  end
+
+  private
+
+  def create_test_site
+    print colorize('Creating test site...', :yellow)
+    @project.create!
+    create_templates
+    create_content
+    puts colorize(' Done!', :green)
+  end
+
+  def create_templates
+    # Layout
+    File.write(File.join(@project.templates_dir, 'layout.html.erb'), <<~ERB)
+      <!DOCTYPE html>
+      <html lang="en">
+      <head>
+        <meta charset="UTF-8">
+        <meta name="viewport" content="width=device-width, initial-scale=1.0">
+        <title><%= title %> - Benchmark Site</title>
+        <style>
+          body { max-width: 800px; margin: 0 auto; padding: 2rem; font-family: system-ui; line-height: 1.6; }
+          nav { margin-bottom: 2rem; }
+        </style>
+      </head>
+      <body>
+        <%= content %>
+      </body>
+      </html>
+    ERB
+
+    # Page template
+    File.write(File.join(@project.templates_dir, 'page.html.erb'), <<~ERB)
+      <main>
+        <%= content %>
+      </main>
+    ERB
+
+    # Blog template
+    File.write(File.join(@project.templates_dir, 'blog.html.erb'), <<~ERB)
+      <article>
+        <header>
+          <h1><%= title %></h1>
+          <time datetime="<%= date %>"><%= date.strftime('%B %d, %Y') %></time>
+        </header>
+        <%= content %>
+      </article>
+    ERB
+  end
+
+  def create_content
+    # Create pages
+    @config[:pages].times do |i|
+      File.write(File.join(@project.src_dir, "page-#{i}.page.md"), generate_page_content(i))
+    end
+
+    # Create blog posts
+    blog_dir = File.join(@project.src_dir, 'blog')
+    FileUtils.mkdir_p(blog_dir)
+    @config[:blog_posts].times do |i|
+      date = Date.today - i
+      File.write(File.join(blog_dir, "#{date}-post-#{i}.blog.md"), generate_blog_content(i))
+    end
+  end
+
+  def generate_page_content(index)
+    <<~MD
+      # Page #{index}
+
+      This is test page number #{index} for benchmarking Jackdaw performance.
+
+      ## Section 1
+
+      Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
+
+      ## Section 2
+
+      Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
+
+      ### Subsection
+
+      Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
+
+      ## Code Example
+
+      ```ruby
+      def hello_world
+        puts "Hello from page #{index}!"
+      end
+      ```
+
+      **Bold text** and *italic text* and `code snippets`.
+    MD
+  end
+
+  def generate_blog_content(index)
+    <<~MD
+      # Blog Post #{index}
+
+      This is test blog post number #{index} for benchmarking.
+
+      ## Introduction
+
+      Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer nec odio. Praesent libero.
+
+      ## Main Content
+
+      Sed cursus ante dapibus diam. Sed nisi. Nulla quis sem at nibh elementum imperdiet.
+
+      ```javascript
+      function testFunction() {
+        console.log('Test from post #{index}');
+        return true;
+      }
+      ```
+
+      ## Conclusion
+
+      Duis sagittis ipsum. Praesent mauris. Fusce nec tellus sed augue semper porta.
+
+      - List item 1
+      - List item 2
+      - List item 3
+    MD
+  end
+
+  def run_benchmarks
+    puts "\n#{colorize('Running benchmarks...', :bold)}\n"
+
+    # Clean build
+    clean_build_time = benchmark_build(clean: true)
+    total_files = @config[:pages] + @config[:blog_posts]
+
+    # Incremental build (no changes)
+    incremental_build_time = benchmark_build(clean: false)
+
+    # Incremental build (with change)
+    modify_single_file
+    incremental_change_time = benchmark_build(clean: false)
+
+    # Results
+    puts "\n#{colorize('Results:', :bold, :magenta)}"
+    puts colorize('-' * 70, :cyan)
+
+    puts "Total files:              #{colorize(total_files.to_s, :cyan)}"
+    puts "Clean build time:         #{colorize(format('%.3f seconds', clean_build_time), :green)}"
+    puts "  Files/second:           #{colorize(format('%.1f', total_files / clean_build_time), :cyan)}"
+    puts "Incremental (no change):  #{colorize(format('%.3f seconds', incremental_build_time), :green)}"
+    puts "  Files/second:           #{colorize(format('%.1f', total_files / incremental_build_time), :cyan)}"
+    puts "Incremental (1 change):   #{colorize(format('%.3f seconds', incremental_change_time), :green)}"
+    puts "  Speedup vs clean:       #{colorize(format('%.1fx', clean_build_time / incremental_change_time), :yellow)}"
+
+    puts colorize('-' * 70, :cyan)
+  end
+
+  def benchmark_build(clean: false)
+    builder = Jackdaw::Builder.new(@project, { clean: clean })
+
+    time = Benchmark.realtime do
+      builder.build
+    end
+
+    label = clean ? 'Clean build' : 'Incremental build'
+    puts "  #{colorize(label, :yellow)}: #{colorize(format('%.3fs', time), :green)}"
+
+    time
+  end
+
+  def modify_single_file
+    # Touch one file to trigger incremental rebuild
+    first_page = File.join(@project.src_dir, 'page-0.page.md')
+    content = File.read(first_page)
+    File.write(first_page, content + "\n\nUpdated content.")
+  end
+
+  def cleanup
+    FileUtils.rm_rf(BENCHMARK_DIR) if Dir.exist?(BENCHMARK_DIR)
+  end
+
+  def colorize(text, *colors)
+    codes = {
+      reset: "\e[0m",
+      bold: "\e[1m",
+      green: "\e[32m",
+      cyan: "\e[36m",
+      yellow: "\e[33m",
+      magenta: "\e[35m"
+    }
+
+    prefix = colors.map { |c| codes[c] }.join
+    "#{prefix}#{text}#{codes[:reset]}"
+  end
+end
+
+# Run benchmark
+if __FILE__ == $PROGRAM_NAME
+  size = ARGV[0]&.to_sym || :medium
+  benchmark = JackdawBenchmark.new(size: size)
+  benchmark.run
+end

data/exe/jackdaw

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'jackdaw'
+
+Jackdaw::CLI.start(ARGV)

data/lib/jackdaw/builder.rb

@@ -0,0 +1,206 @@
+# frozen_string_literal: true
+
+module Jackdaw
+  # Orchestrates the build process with incremental builds and parallel processing
+  class Builder
+    attr_reader :project, :scanner, :renderer, :stats
+
+    def initialize(project, options = {})
+      @project = project
+      @scanner = Scanner.new(project)
+      @renderer = Renderer.new(project)
+      @options = options
+      @stats = BuildStats.new
+    end
+
+    # Run full build
+    def build
+      start_time = Time.now
+
+      # Clean if requested
+      clean_output if @options[:clean]
+
+      # Ensure output directory exists
+      FileUtils.mkdir_p(project.output_dir)
+
+      # Build content files in parallel
+      build_content_files
+
+      # Copy assets in parallel
+      copy_assets
+
+      # Generate feeds and sitemap
+      generate_feeds_and_sitemap
+
+      @stats.total_time = Time.now - start_time
+      @stats
+    end
+
+    # Clean the output directory
+    def clean_output
+      return unless Dir.exist?(project.output_dir)
+
+      FileUtils.rm_rf(Dir.glob(File.join(project.output_dir, '*')))
+    end
+
+    private
+
+    def build_content_files
+      content_files = scanner.content_files
+
+      # Process files in parallel
+      results = Parallel.map(content_files, in_threads: Parallel.processor_count) do |content_file|
+        process_content_file(content_file)
+      end
+
+      # Aggregate results
+      results.each do |result|
+        case result[:status]
+        when :built
+          @stats.files_built += 1
+        when :skipped
+          @stats.files_skipped += 1
+        when :error
+          @stats.errors << result[:error]
+        end
+      end
+    end
+
+    def process_content_file(content_file)
+      # Check if template exists
+      template_file = File.join(project.templates_dir, "#{content_file.type}.html.erb")
+      unless File.exist?(template_file)
+        warning = "⚠️  Missing template '#{content_file.type}.html.erb' for #{content_file.path}"
+        puts "\e[33m#{warning}\e[0m" if @options[:verbose]
+        return { status: :error, file: content_file.path,
+                 error: StandardError.new("Missing template: #{content_file.type}.html.erb") }
+      end
+
+      # Check if we need to rebuild
+      return { status: :skipped, file: content_file.path } unless needs_rebuild?(content_file)
+
+      # Render and write
+      html = renderer.render_content(content_file)
+      output_file = content_file.output_file
+
+      FileUtils.mkdir_p(File.dirname(output_file))
+      File.write(output_file, html)
+
+      { status: :built, file: content_file.path }
+    rescue StandardError => e
+      { status: :error, file: content_file.path, error: e }
+    end
+
+    def copy_assets
+      asset_files = scanner.asset_files
+
+      results = Parallel.map(asset_files, in_threads: Parallel.processor_count) do |asset_file|
+        next { status: :skipped } unless needs_asset_copy?(asset_file)
+
+        asset_file.copy!
+        { status: :copied }
+      rescue StandardError => e
+        { status: :error, error: e }
+      end
+
+      results.each do |result|
+        case result[:status]
+        when :copied
+          @stats.assets_copied += 1
+        when :skipped
+          @stats.assets_skipped += 1
+        when :error
+          @stats.errors << result[:error]
+        end
+      end
+    end
+
+    def needs_rebuild?(content_file)
+      # Always rebuild if clean build
+      return true if @options[:clean]
+
+      output_file = content_file.output_file
+
+      # Rebuild if output doesn't exist
+      return true unless File.exist?(output_file)
+
+      output_mtime = File.mtime(output_file)
+
+      # Check if content file is newer
+      return true if content_file.mtime > output_mtime
+
+      # Check if template is newer
+      template_file = find_template_file(content_file.type)
+      return true if template_file && File.mtime(template_file) > output_mtime
+
+      # Check if layout is newer
+      layout_file = File.join(project.templates_dir, 'layout.html.erb')
+      return true if File.exist?(layout_file) && File.mtime(layout_file) > output_mtime
+
+      false
+    end
+
+    def needs_asset_copy?(asset_file)
+      # Always copy if clean build
+      return true if @options[:clean]
+
+      output_file = asset_file.output_file
+
+      # Copy if output doesn't exist
+      return true unless File.exist?(output_file)
+
+      # Copy if source is newer
+      asset_file.mtime > File.mtime(output_file)
+    end
+
+    def find_template_file(type)
+      template_path = File.join(project.templates_dir, "#{type}.html.erb")
+      File.exist?(template_path) ? template_path : nil
+    end
+
+    def generate_feeds_and_sitemap
+      # Generate RSS/Atom feeds only if there are blog posts
+      if has_blog_posts?
+        feed_generator = FeedGenerator.new(project)
+        feed_generator.generate_rss
+        feed_generator.generate_atom
+      end
+
+      # Always generate sitemap
+      sitemap_generator = SitemapGenerator.new(project)
+      sitemap_generator.generate
+    rescue StandardError => e
+      puts "⚠️  Warning: Failed to generate feeds/sitemap: #{e.message}" if @options[:verbose]
+    end
+
+    def has_blog_posts?
+      scanner.content_files.any? { |f| %w[blog post article news].include?(f.type) }
+    end
+  end
+
+  # Tracks build statistics
+  class BuildStats
+    attr_accessor :files_built, :files_skipped, :assets_copied, :assets_skipped, :errors, :total_time
+
+    def initialize
+      @files_built = 0
+      @files_skipped = 0
+      @assets_copied = 0
+      @assets_skipped = 0
+      @errors = []
+      @total_time = 0
+    end
+
+    def total_files
+      files_built + files_skipped
+    end
+
+    def total_assets
+      assets_copied + assets_skipped
+    end
+
+    def success?
+      errors.empty?
+    end
+  end
+end

data/lib/jackdaw/cli.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module Jackdaw
+  # Main CLI entry point
+  class CLI < Thor
+    include CLIHelpers
+
+    def self.exit_on_failure?
+      true
+    end
+
+    desc 'build', 'Build the static site'
+    method_option :clean, type: :boolean, aliases: '-c', desc: 'Clean output directory before building'
+    method_option :watch, type: :boolean, aliases: '-w', desc: 'Watch for changes and rebuild'
+    method_option :verbose, type: :boolean, aliases: '-v', desc: 'Verbose output'
+    def build
+      Commands::Build.new(Project.new, options).execute
+    end
+
+    desc 'serve', 'Start development server'
+    method_option :port, type: :numeric, aliases: '-p', default: 4000, desc: 'Port to run server on'
+    method_option :host, type: :string, aliases: '-h', default: 'localhost', desc: 'Host to bind to'
+    method_option :livereload, type: :boolean, aliases: '-l', default: true, desc: 'Enable live reload'
+    def serve
+      Commands::Serve.new(Project.new, options).execute
+    end
+
+    desc 'create TEMPLATE NAME', 'Create a new content file from template'
+    method_option :dated, type: :boolean, desc: 'Add date prefix to filename'
+    method_option :no_date, type: :boolean, desc: 'Skip date prefix (overrides default behavior)'
+    def create(template, name)
+      Commands::Create.new(Project.new, template, name, options).execute
+    end
+
+    desc 'new NAME', 'Create a new site project'
+    def new(name)
+      Commands::New.new(name).execute
+    end
+
+    desc 'template SUBCOMMAND', 'Manage templates'
+    subcommand 'template', Commands::Template
+
+    desc 'version', 'Show version'
+    def version
+      puts "\n#{colorize('Jackdaw', :bold)} #{colorize("v#{Jackdaw::VERSION}", :cyan)}"
+      puts colorize('Lightning-fast static site generator', :magenta)
+    end
+  end
+end

data/lib/jackdaw/cli_helpers.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Jackdaw
+  # Shared helper methods for CLI output formatting
+  module CLIHelpers
+    # ANSI color codes for beautiful output
+    COLORS = {
+      reset: "\e[0m",
+      bold: "\e[1m",
+      green: "\e[32m",
+      cyan: "\e[36m",
+      yellow: "\e[33m",
+      magenta: "\e[35m",
+      blue: "\e[34m"
+    }.freeze
+
+    def colorize(text, color)
+      "#{COLORS[color]}#{text}#{COLORS[:reset]}"
+    end
+
+    def success(text)
+      puts "#{colorize('✓', :green)} #{text}"
+    end
+
+    def info(text)
+      puts "#{colorize('→', :cyan)} #{text}"
+    end
+
+    def header(text)
+      puts "\n#{COLORS[:bold]}#{COLORS[:magenta]}#{text}#{COLORS[:reset]}"
+    end
+  end
+end