photish 0.3.3 → 0.3.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 837da75e8c5426ba38888369dc28eaa433e5162c
-  data.tar.gz: 8678a47a42f4b7e49ec3faae98130ada68e742e6
+  metadata.gz: 5ff58bf1d669264455034b1ea27a798ea979d2f0
+  data.tar.gz: 6f6bb07a9b0ecd54643b9090c27336b13c0ecb09
 SHA512:
-  metadata.gz: f5fb8362d0d569a7f2f7c13edcd86259112cc062714f64a30602d3d00aaf7e6efde8bf0f5753a72a7f92447702b67d823085c363e445defddee223d98341374d
-  data.tar.gz: e5309014399f150ee267288433e363206b6b9356c4f6c7fbe28d9a61f9cbdd00112d0cb01017a890b6e7c946bd8488d12f2f2e37a04f1ca5987cdd1d539f5bad
+  metadata.gz: ea5d2478db86db55aca53b8fec1abe53c65d6714e6fa12bd6d3c8cd85b5f4d145183e184b4e9a10bf1a8b9cd0fd4650426ae4416f2862b7c54bae49953881663
+  data.tar.gz: b86aa902da22f12807bdef9d26b46f7df180472268e56affc1ea9b63a7ee0c2f45fd15056a64c3a049bcb9a8acf05c6c99f54e7a3c185ce801f725f7bd72dfb6

data/.gitignore

@@ -3,6 +3,7 @@
 Gemfile.lock
 _yardoc
 coverage
+profile
 doc
 pkg
 spec/reports

data/README.md

@@ -34,9 +34,8 @@ It is strongly recommended to read through the [Installation](#installation)
 and [Usage](#usage) sections before seriously using Photish, however to get up
 and running:
 
-1. Ensure [ImageMagick](http://www.imagemagick.org/script/index.php),
-   [Exiftool](http://www.sno.phy.queensu.ca/~phil/exiftool/) and
-   [Nokogiri](http://www.nokogiri.org/tutorials/installing_nokogiri.html) are
+1. Ensure [ImageMagick](http://www.imagemagick.org/script/index.php) and
+   [Exiftool](http://www.sno.phy.queensu.ca/~phil/exiftool/) are
    installed (see [Dependencies](#dependencies))
 1. Install Photish `gem install photish`
 1. Create a base project with `photish init --example`
@@ -63,6 +62,8 @@ and running:
       - [Template Helpers](#template-helpers)
   - [Generate](#generate)
     - [Execution Order](#execution-order)
+    - [Workers and Threads](#workers-and-threads)
+    - [Caching](#caching)
   - [Host](#host)
   - [Rake Task](#rake-task)
   - [Plugins](#plugins)
@@ -191,8 +192,6 @@ Photish has dependencies on certain utilities:
   conversion
 - [Exiftool](http://www.sno.phy.queensu.ca/~phil/exiftool/) for image metadata
   retrieval
-- [Nokogiri](http://www.nokogiri.org/tutorials/installing_nokogiri.html) for
-  XML writing and parsing
 
 **On MacOSX, using [Brew](http://brew.sh/)**
 
@@ -293,6 +292,8 @@ logging:
 url:
   host: http://mydomain.com
   base: 'subdirectory'
+workers: 4
+threads: 2
 ```
 
 The meanings and purpose of each field is defined below:
@@ -315,6 +316,8 @@ Field                  | Purpose
 `url`                  | a listing of the various url options
 `url/host`             | if you would like URLs generated with a specific host prefix, you can define it here, otherwise leave it as '/' or do not set this configuration at all
 `url/base`             | if your website will be hosted in a sub folder and will not be accessible at the root of the host, you can specify the sub folder(s) here, this will also mean your website will be hosted in a sub folder when ran using `photish host`
+`workers`              | the number of workers to create, for computers with multiple processors, photish is configured by default to spawn a worker for each process, a worker is responsible for image generation and html generation, load balancing is done randomly via a simple round robin allocation
+`threads`              | the number of threads each worker should create to handle image magick transcoding
 
 #### Customizing Templates
 
@@ -468,6 +471,55 @@ The Generate command does the following:
 1. Converts all Photo(s) to the configured quality versions, writing various
    images to the `output` folder
 
+#### Workers and Threads
+
+In order to achieve maximum utilization of all processors on a CPU during
+generation, Photish has the ability to create multiple workers and threads.
+
+A worker is a spawned sub process created by the Generate command. The worker
+sub process is responsible for generating the HTML and Images for a sub set of
+the collection.
+
+Within each worker, threads are created when calling out to the Image Magick
+binary. During conversion, Image Magick often does not reach full processor
+utilization so rather then block the whole worker, it can be more performant to
+spawn multiple Image Magick processes at once.
+
+For collections with a large number of images and HTML pages, multiple workers
+and threads can be used to rapidly speed up generation. However if the
+collection is quite small and the images are of a small size, workers and
+threads will increase the generation time as loading a new ruby runtime and
+creating multiple threads may have a higher setup time then just generating in
+a single ruby process.
+
+The number of workers and threads is configurable in the [config
+file](#config-file-options) with the `workers` and `threads` options. By
+default, Photish will spawn a worker for each processor detected on the
+computer. It will then create 2 threads per worker. As each worker spawns it's
+own thread, for a computer with 4 processors, 4 workers will be created, each
+with 2 threads, which means in total Photish will manage 8 threads and
+potentially run 8 Image Magick processes concurrently. When tweaking the number
+of workers and threads it is important to consider IO bottlenecks as this will
+most likely be the limiting factor in performance.
+
+#### Caching
+
+Photish caches the generation of images to avoid regeneration when the
+Generate command is run or the generate event is triggered while hosting
+a local version of Photish with the Host command.
+
+The cache file is stored in the `output_dir` and is named `.changes.yml`.
+
+To do a full regeneration, simple run the Generate command with the `force`
+flag:
+
+    $ photish generate --force
+
+Images are regenerated when they are modified, renamed or moved.
+
+Changing the `qualities` option in the config file will also trigger a full
+regeneration of all images.
+
 ### Host
 
 To test and view your changes locally, the host command can be used to run a
@@ -521,8 +573,6 @@ called inside the template, it will render the message in bold wrapped in the
 
 **site/_plugins/shout.rb**
 ```ruby
-require 'nokogiri'
-
 module Photish::Plugin::Shout
 
   def self.is_for?(type)

data/TODO.md

@@ -2,6 +2,8 @@
 
 ## In Progress
 
+1. Look at error handling, exiting and cleanup (files, processes and threads)
+
 ## Backlog
 
 1. Plugin as a gem to deploy to github pages, netlify, amazon s3

data/exe/photish

@@ -1,10 +1,24 @@
 #!/usr/bin/env ruby
+
 if ENV['COVERAGE']
   require 'simplecov'
-
   SimpleCov.command_name "photish-binary-#{Process.pid}"
   SimpleCov.root(File.join(File.expand_path(File.dirname(__FILE__)), '..'))
 end
 
+if ENV['PROFILE']
+  require 'ruby-prof'
+  FileUtils.rm_rf('profile')
+  RubyProf.start
+end
+
 require 'photish'
 Photish::CLI::Interface.start
+
+if ENV['PROFILE']
+  profile_filename = "cmd-#{ARGV[0] || 'unknown'}-#{Process.pid}"
+  FileUtils.mkdir_p('profile')
+  RubyProf::MultiPrinter.new(RubyProf.stop)
+                        .print(profile: profile_filename,
+                               path: 'profile')
+end

data/lib/photish/assets/example/site/_plugins/footer_links.rb

@@ -1,5 +1,3 @@
-require 'nokogiri'
-
 module Photish::Plugin::FooterLinks
 
   def self.is_for?(type)
@@ -12,19 +10,16 @@ module Photish::Plugin::FooterLinks
   end
 
   def links_with_seperator(links, seperator)
-    doc = Nokogiri::HTML::DocumentFragment.parse("")
-    Nokogiri::HTML::Builder.with(doc) do |doc|
-      links.each_with_index do |link, index|
-        if index == (links.count - 1)
-          text = link[:text]
-        else
-          text = "#{link[:text]} #{seperator} "
-        end
-
-        doc.a(text, href: link[:url])
+    html = ''
+    links.each_with_index do |link, index|
+      if index == (links.count - 1)
+        text = link[:text]
+      else
+        text = "#{link[:text]} #{seperator} "
       end
+      html << "<a href=\"" << link[:url] << "\">" << text << "</a>"
     end
-    doc.to_html
+    html
   end
 end
 

data/lib/photish/assets/example/site/_plugins/yell_loud.rb

@@ -1,5 +1,3 @@
-require 'nokogiri'
-
 module Photish::Plugin::YellLoud
 
   def self.is_for?(type)
@@ -12,11 +10,7 @@ module Photish::Plugin::YellLoud
   end
 
   def yell_very_loud
-    doc = Nokogiri::HTML::DocumentFragment.parse("")
-    Nokogiri::HTML::Builder.with(doc) do |doc|
-      doc.span("Yelling \"#{name}\" from a plugin!",
-               style: 'font-weight:bold;color:red;font-size:200%;')
-    end
-    doc.to_html
+    text = "Yelling \"#{name}\" from a plugin!"
+    "<span style=\"font-weight:bold;color:red;font-size:200%;\">#{text}</span>"
   end
 end

data/lib/photish/assets/example/site/styles/basic.css

@@ -49,7 +49,6 @@ ul.breadcrumbs, ul.breadcrumbs li {
   padding:0;
 }
 ul.breadcrumbs {
-  border:1px solid #dedede;
   height:2.3em;
 }
 ul.breadcrumbs li {

data/lib/photish/cache/manifest.rb

@@ -0,0 +1,61 @@
+module Photish
+  module Cache
+    class Manifest
+      def initialize(output_dir, worker_index, version_hash)
+        @output_dir = output_dir
+        @worker_index = worker_index
+        @version_hash = version_hash
+        @cache = {}
+        @worker_db = {}
+      end
+
+      def record(key, file_path = nil)
+        checksum = checksum_of_file(file_path || key)
+        worker_db[key] = checksum
+        db[key] = checksum
+      end
+
+      def changed?(key, file_path = nil)
+        checksum = checksum_of_file(file_path || key)
+        worker_db[key] = checksum
+        checksum != db[key]
+      end
+
+      def flush_to_disk
+        File.open(worker_db_file, 'w') { |f| f.write(worker_db.to_yaml) }
+      end
+
+      def preload
+        db
+      end
+
+      private
+
+      attr_reader :output_dir,
+                  :cache,
+                  :version_hash,
+                  :worker_index,
+                  :worker_db
+
+      def checksum_of_file(file_path)
+        cache.fetch(file_path.hash) do |key|
+          cache[key] = version_hash.to_s +
+                       Digest::MD5.file(file_path).hexdigest
+        end
+      end
+
+      def db
+        return @db if @db
+        @db = File.exist?(db_file) ? YAML.load_file(db_file) : {}
+      end
+
+      def db_file
+        ManifestDbFile.db_file(output_dir)
+      end
+
+      def worker_db_file
+        ManifestDbFile.worker_db_file(output_dir, worker_index)
+      end
+    end
+  end
+end

data/lib/photish/cache/manifest_db_file.rb

@@ -0,0 +1,28 @@
+module Photish
+  module Cache
+    module ManifestDbFile
+      def concat_db_files(output_dir, workers)
+        changes = (1..workers).inject({}) do |accumulator, worker_index|
+          file = worker_db_file(output_dir, worker_index)
+          accumulator.merge!(YAML.load_file(file)) if File.exist?(file)
+          accumulator
+        end
+        File.open(db_file(output_dir), 'w') { |f| f.write(changes.to_yaml) }
+      end
+
+      def db_file(output_dir)
+        FileUtils.mkdir_p(output_dir)
+        File.join(output_dir, '.changes.yml')
+      end
+
+      def worker_db_file(output_dir, index)
+        FileUtils.mkdir_p(output_dir)
+        File.join(output_dir, ".changes.#{index}.yml")
+      end
+
+      module_function :concat_db_files,
+                      :db_file,
+                      :worker_db_file
+    end
+  end
+end

data/lib/photish/cli/interface.rb

@@ -4,10 +4,17 @@ module Photish
       package_name "Photish"
 
       desc "generate", "Generates the gallery static site"
+      option :worker_index, type: :numeric
       def generate
         Photish::Command::Generate.new(options).execute
       end
 
+      desc "worker", "A worker process that helps the generate command"
+      option :worker_index, type: :numeric
+      def worker
+        Photish::Command::Worker.new(options).execute
+      end
+
       desc "host", "Serves the HTML on a HTTP server"
       def host
         Photish::Command::Host.new(options).execute

data/lib/photish/command/base.rb

@@ -21,15 +21,18 @@ module Photish
       attr_reader :runtime_config,
                   :log
 
-      def config
-        @config ||= Photish::Config::AppSettings.new(runtime_config)
-                                                .config
-      end
+      delegate :config,
+               :version_hash,
+               to: :app_settings
 
       private
 
+      def app_settings
+        @app_settings ||= Config::AppSettings.new(runtime_config)
+      end
+
       def setup_logging
-        Photish::Log::Logger.instance.setup_logging(config)
+        Log::Logger.instance.setup_logging(config)
       end
     end
   end

data/lib/photish/command/deploy.rb

@@ -23,7 +23,7 @@ module Photish
       end
 
       def load_all_plugins
-        Photish::Plugin::Repository.reload(log, site_dir)
+        Plugin::Repository.reload(log, site_dir)
       end
 
       def engine_class
@@ -33,11 +33,11 @@ module Photish
       end
 
       def deploy_plugins
-        Photish::Plugin::Repository.plugins_for(deploy_plugin_type)
+        Plugin::Repository.plugins_for(deploy_plugin_type)
       end
 
       def deploy_plugin_type
-        Photish::Plugin::Type::Deploy
+        Plugin::Type::Deploy
       end
     end
   end

data/lib/photish/command/generate.rb

@@ -2,32 +2,57 @@ module Photish
   module Command
     class Generate < Base
       def run
+        log.info "Starting generation with #{workers} workers"
+
+        spawn_all_workers
         load_all_plugins
-        render_whole_site
-        log.info 'Site generation completed successfully'
+        wait_for_workers_to_complete
+        concat_db_files
+        perform_serial_generation
+
+        log.info "Generation completed successfully"
       end
 
       private
 
       delegate :output_dir,
-               :site_dir,
                :photo_dir,
-               :qualities,
-               :templates,
                :url,
-               :max_workers,
+               :site_dir,
+               :qualities,
+               :photish_executable,
+               :workers,
                to: :config
 
       def load_all_plugins
-        Photish::Plugin::Repository.reload(log, site_dir)
+        Plugin::Repository.reload(log, site_dir)
+      end
+
+      def spawn_all_workers
+        return single_worker if one_worker?
+        @spawned_processes ||= (1..workers).map do |index|
+          Process.spawn(ENV, worker_command(index))
+        end
+      end
+
+      def wait_for_workers_to_complete
+        return if one_worker?
+        @spawned_processes.map do |pid|
+          Process.waitpid(pid)
+        end
+      end
+
+      def one_worker?
+        workers == 1
+      end
+
+      def single_worker
+        Worker.new(runtime_config.merge(worker_index: 1)).execute
       end
 
-      def render_whole_site
-        Photish::Render::Site.new(templates,
-                                  site_dir,
-                                  output_dir,
-                                  max_workers)
-                             .all_for(collection)
+      def perform_serial_generation
+        Render::Site.new(config)
+                    .all_for(collection)
       end
 
       def collection
@@ -39,6 +64,16 @@ module Photish
       def qualities_mapped
         qualities.map { |quality| OpenStruct.new(quality) }
       end
+
+      def worker_command(worker_index)
+        [photish_executable,
+         'worker',
+         "--worker_index=#{worker_index}"].join(' ')
+      end
+
+      def concat_db_files
+        Cache::ManifestDbFile.concat_db_files(output_dir, workers)
+      end
     end
   end
 end

data/lib/photish/command/host.rb

@@ -8,6 +8,7 @@ module Photish
         log.info "Monitoring paths #{paths_to_monitor}"
 
         regenerate_entire_site
+        regenerate_thread
         start_http_server_with_listener
       end
 
@@ -17,14 +18,27 @@ module Photish
                :output_dir,
                :site_dir,
                :photo_dir,
+               :config_file_location,
                to: :config
 
       def start_http_server_with_listener
         trap 'INT' do server.shutdown end
         listener.start
         server.start
-        listener.stop
         log.info "Photish host has shutdown"
+      ensure
+        regenerate_thread.exit if @regenerate_thread
+        listener.stop if @listener
+      end
+
+      def regenerate_thread
+        @regenerate_thread ||= Thread.new do
+          loop do
+            queue.pop
+            queue.clear
+            regenerate_entire_site
+          end
+        end
       end
 
       def server
@@ -39,8 +53,7 @@ module Photish
           log.info "File was modified #{modified}" if modified.present?
           log.info "File was added #{added}" if added.present?
           log.info "File was removed #{removed}" if removed.present?
-
-          regenerate_entire_site
+          queue.push(modified || added || removed)
         end
       end
 
@@ -61,6 +74,10 @@ module Photish
         Photish::Command::Generate.new(runtime_config)
                                   .execute
       end
+
+      def queue
+        @queue ||= Queue.new
+      end
     end
   end
 end

data/lib/photish/command/worker.rb

@@ -0,0 +1,44 @@
+module Photish
+  module Command
+    class Worker < Base
+      def run
+        log.info "Worker ##{worker_index} starting"
+
+        load_all_plugins
+        render_whole_site
+
+        log.info "Site generation completed, by Worker ##{worker_index}"
+      end
+
+      private
+
+      delegate :site_dir,
+               :photo_dir,
+               :output_dir,
+               :qualities,
+               :url,
+               :worker_index,
+               to: :config
+
+      def load_all_plugins
+        return if Plugin::Repository.loaded?
+        Plugin::Repository.reload(log, site_dir)
+      end
+
+      def render_whole_site
+        Render::SiteWorker.new(config, version_hash)
+                          .all_for(collection)
+      end
+
+      def collection
+        @collection ||= Gallery::Collection.new(photo_dir,
+                                                qualities_mapped,
+                                                url)
+      end
+
+      def qualities_mapped
+        qualities.map { |quality| OpenStruct.new(quality) }
+      end
+    end
+  end
+end

data/lib/photish/config/app_settings.rb

@@ -9,6 +9,10 @@ module Photish
         @config ||= RecursiveOpenStruct.new(prioritized_config)
       end
 
+      def version_hash
+        @version_hash ||= Digest::MD5.hexdigest(sensitive_config.to_json)
+      end
+
       private
 
       attr_reader :runtime_config
@@ -17,10 +21,21 @@ module Photish
         {}.merge(default_config)
           .merge(file_config)
           .merge(runtime_config)
+          .merge(derived_config)
+      end
+
+      def sensitive_config
+        prioritized_config.slice('qualities')
+      end
+
+      def derived_config
+        {
+          config_file_location: config_file_location
+        }
       end
 
       def file_config
-        symbolize(FileConfig.new(file_config_location)
+        symbolize(FileConfig.new(config_file_location)
                                             .hash)
       end
 
@@ -28,9 +43,9 @@ module Photish
         symbolize(DefaultConfig.new.hash)
       end
 
-      def file_config_location
-        FileConfigLocation.new(runtime_config[:site_dir])
-                        .path
+      def config_file_location
+        FileConfigLocation.new(runtime_config[:config_dir])
+                          .path
       end
 
       def symbolize(hash)

data/lib/photish/config/default_config.rb

@@ -7,7 +7,10 @@ module Photish
           site_dir: File.join(Dir.pwd, 'site'),
           photo_dir: File.join(Dir.pwd, 'photos'),
           output_dir: File.join(Dir.pwd, 'output'),
-          max_workers: Facter.value('processors')['count'],
+          workers: workers,
+          threads: threads,
+          worker_index: 0,
+          photish_executable: photish_executable,
           qualities: [
             { name: 'Original',
               params: [] },
@@ -26,11 +29,34 @@ module Photish
             level: 'info'
           },
           url: {
-            host: '/',
+            host: '',
             base: nil
           }
         }
       end
+
+      private
+
+      def workers
+        processor_count / 2
+      end
+
+      def threads
+        2
+      end
+
+      def processor_count
+        Facter.value('processors')['count']
+      end
+
+      def photish_executable
+        File.join(File.dirname(__FILE__),
+                  '..',
+                  '..',
+                  '..',
+                  'exe',
+                  'photish')
+      end
     end
   end
 end