cloud-crowd 0.1.0

data/EPIGRAPHS

@@ -0,0 +1,17 @@
+The crowd, suddenly there where there was nothing before, is a mysterious and
+universal phenomenon. A few people may have been standing together -- five, ten
+or twelve, nor more; nothing has been announced, nothing is expected. Suddenly
+everywhere is black with people and more come streaming from all sides as though
+streets had only one direction. Most of them do not know what has happened and, 
+if questioned, have no answer; but they hurry to be there where most other
+people are. There is a determination in their movement which is quite different
+from the expression of ordinary curiosity. It seems as through the movement of 
+some of them transmits itself to all the others. But that is not all; they have
+a goal which is there before they can find words for it. -p 16
+
+Crowd crystals are the small, rigid groups of men, strictly delimited and of
+great constancy, which serve to precipitate crowds. Their structure is such 
+that they can be comprehended and taken in at a glance. Their unity is more
+important than their size. -p 73
+
+From Elias Canetti's "Crowds and Power" (1962).

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2009 Jeremy Ashkenas, DocumentCloud
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,93 @@
+=                                                                               
+           _  _                                                                
+          ( `   )_                                                             
+         (    )    `)                                                          
+       (_   (_ .  _) _)                                                        
+                                      _                                        
+                                     (  )                                      
+      _ .                         ( `  ) . )                                   
+    (  _ )_                      (_, _(  ,_)_)                                 
+  (_  _(_ ,)                                                                   
+                                                                               
+           _  _               ___ _             _  ___                   _     
+          ( `   )_           / __| |___ _  _ __| |/ __|_ _ _____ __ ____| |    
+         (    )    `)       | (__| / _ \ || / _` | (__| '_/ _ \ V  V / _` |    
+       (_   (_ .  _) _)      \___|_\___/\_,_\__,_|\___|_| \___/\_/\_/\__,_|    
+                                                                               
+                                                     _                         
+                                                    (  )                       
+                  _, _ .                         ( `  ) . )                    
+                 ( (  _ )_                      (_, _(  ,_)_)                  
+               (_(_  _(_ ,)                                                    
+                                                                               
+                                                                               
+                                                                               
+  ~ CloudCrowd ~
+
+    * Parallel processing for the rest of us
+    * Write your scripts in Ruby
+    * Built for Amazon EC2 and S3
+    * split -> process -> merge
+    * As easy as `gem install cloud-crowd`
+
+    Well-suited for:
+    
+    * Generating or resizing images.
+    * Encoding video.
+    * Running text extraction or OCR on PDFs.
+    * Migrating a large file set or database.
+    * Web scraping.
+    
+    
+  ~ Documentation ~
+  
+    Wiki: http://wiki.github.com/documentcloud/cloud-crowd
+    Rdoc: http://rdoc.info/projects/documentcloud/cloud-crowd
+  
+  
+  ~ Getting started ~
+  
+    # Install the gem.
+    
+      >> sudo gem install cloud-crowd
+    
+    # Install the CloudCrowd configuration files to a location of your choosing.
+    
+      >> crowd install ~/config/cloud-crowd
+    
+    # Now, you can use the full complement of `crowd` commands from inside of
+    # this configuration directory. To see the available commands:
+    
+      >> crowd --help
+    
+    # Edit the configuration files to your satisfaction, add AWS credentials, 
+    # and then load the CloudCrowd schema into your configured database.
+    
+      >> mate ~/config/cloud-crowd/config.yml
+      >> mate ~/config/cloud-crowd/database.yml
+      >> crowd load_schema
+    
+    # Write your actions, and install them into the 'actions' subdirectory.
+    # CloudCrowd comes with some default actions as an example.
+    
+    # To launch the central server (make sure that you include its location
+    # in config.yml), either:
+    
+      >> crowd server
+    
+    # or:
+    
+      >> thin -R config.ru --servers 3 -e production start
+    
+    # Any server that supports Rack should work with the rackup file.
+    
+    # Then, to spin up 10 workers:
+    
+      >> crowd workers start -n 10
+    
+    # To spin up workers remotely, install the 'cloud-crowd' gem, and copy over
+    # your configuration directory.
+    
+    # At this point you can visit your server console at localhost:9173 to 
+    # view all of your workers, ready for action.
+  

data/actions/graphics_magick.rb

@@ -0,0 +1,43 @@
+# The GraphicsMagick action, dependent on the `gm` command, is able to perform
+# any number of GraphicsMagick conversions on an image passed in as an input.
+# The options hash should specify the +name+ for the particular step (which is
+# appended to the resulting image filename) the +command+ (eg. convert, mogrify), 
+# the +options+ (to the command, eg. -shadow -blur), and the +extension+ which 
+# will determine the resulting image type. Optionally, you may also specify
+# +input+ as the name of a previous step; doing this will use the result of
+# that step as the source image, otherwise each step uses the original image
+# as its source.
+class GraphicsMagick < CloudCrowd::Action
+  
+  # Download the initial image, and run each of the specified GraphicsMagick
+  # commands against it, returning the aggregate output.
+  def process
+    options['steps'].inject({}) {|h, step| h[step['name']] = run_step(step); h }
+  end
+  
+  # Run an individual step (single GraphicsMagick command) in a shell-injection
+  # safe way, uploading the result to the AssetStore, and returning the public
+  # URL as the result.
+  # TODO: +system+ wasn't working, figure out some other way to escape.
+  def run_step(step)
+    cmd, opts = step['command'], step['options']
+    in_path, out_path = input_path_for(step), output_path_for(step)
+    `gm #{cmd} #{opts} #{in_path} #{out_path}`
+    save(out_path)    
+  end
+  
+  # Where should the starting image be located?
+  # If you pass in an optional step, returns the path to that step's output
+  # as input for further processing.
+  def input_path_for(step)
+    in_step = step && step['input'] && options['steps'].detect {|s| s['name'] == step['input']}
+    return input_path unless in_step
+    return output_path_for(in_step)
+  end
+  
+  # Where should resulting images be saved locally?
+  def output_path_for(step)
+    "#{work_directory}/#{file_name}_#{step['name']}.#{step['extension']}"
+  end
+  
+end

data/actions/process_pdfs.rb

@@ -0,0 +1,92 @@
+# Depends on working pdftk, gm (GraphicsMagick), and pdftotext (Poppler) commands.
+# Splits a pdf into batches of N pages, creates their thumbnails and icons,
+# as specified in the Job options, gets the text for every page, and merges 
+# it all back into a tar archive for convenient download.
+#
+# See <tt>examples/process_pdfs_example.rb</tt> for more information.
+class ProcessPdfs < CloudCrowd::Action
+  
+  # Split up a large pdf into single-page pdfs. Batch them into 'batch_size'
+  # chunks for processing. The double pdftk shuffle fixes the document xrefs.
+  def split
+    `pdftk #{input_path} burst output "#{file_name}_%05d.pdf_temp"`
+    FileUtils.rm input_path
+    pdfs = Dir["*.pdf_temp"]
+    pdfs.each {|pdf| `pdftk #{pdf} output #{File.basename(pdf, '.pdf_temp')}.pdf`}
+    pdfs = Dir["*.pdf"]
+    batch_size = options['batch_size']
+    batches = (pdfs.length / batch_size.to_f).ceil
+    batches.times do |batch_num|
+      tar_path = "#{sprintf('%05d', batch_num)}.tar"
+      batch_pdfs = pdfs[batch_num*batch_size...(batch_num + 1)*batch_size]
+      `tar -czf #{tar_path} #{batch_pdfs.join(' ')}`
+    end
+    Dir["*.tar"].map {|tar| save(tar) }.to_json
+  end
+
+  # Convert a pdf page into different-sized thumbnails. Grab the text.
+  def process
+    `tar -xzf #{input_path}`
+    FileUtils.rm input_path
+    cmds = []
+    generate_images_commands(cmds)
+    generate_text_commands(cmds)
+    system cmds.join(' && ')
+    FileUtils.rm Dir['*.pdf']
+    `tar -czf #{file_name}.tar *`
+    save("#{file_name}.tar")
+  end
+  
+  # Merge all of the resulting images, all of the resulting text files, and
+  # the concatenated merge of the full-text into a single tar archive, ready to
+  # for download.
+  def merge
+    input.each do |batch_url|
+      batch_path = File.basename(batch_url)
+      download(batch_url, batch_path)
+      `tar -xzf #{batch_path}`
+      FileUtils.rm batch_path
+    end
+    
+    names = Dir['*.txt'].map {|fn| fn.sub(/_\d+(_\w+)?\.txt\Z/, '') }.uniq
+    dirs = names.map {|n| ["#{n}/text/full", "#{n}/text/pages"] + options['images'].map {|i| "#{n}/images/#{i['name']}" } }.flatten
+    FileUtils.mkdir_p(dirs)
+    
+    Dir['*.*'].each do |file|
+      ext = File.extname(file)
+      name = file.sub(/_\d+(_\w+)?#{ext}\Z/, '')
+      if ext == '.txt'
+        FileUtils.mv(file, "#{name}/text/pages/#{file}")
+      else
+        suffix      = file.match(/_([^_]+)#{ext}\Z/)[1]
+        sans_suffix = file.sub(/_([^_]+)#{ext}\Z/, ext)
+        FileUtils.mv(file, "#{name}/images/#{suffix}/#{sans_suffix}")
+      end
+    end
+    
+    names.each {|n| `cat #{n}/text/pages/*.txt > #{n}/text/full/#{n}.txt` }
+    
+    `tar -czf processed_pdfs.tar *`
+    save("processed_pdfs.tar")
+  end
+  
+  
+  private
+  
+  def generate_images_commands(command_list)
+    Dir["*.pdf"].each do |pdf| 
+      name = File.basename(pdf, File.extname(pdf))
+      options['images'].each do |i|
+        command_list << "gm convert #{i['options']} #{pdf} #{name}_#{i['name']}.#{i['extension']}"
+      end
+    end
+  end
+  
+  def generate_text_commands(command_list)
+    Dir["*.pdf"].each do |pdf|
+      name = File.basename(pdf, File.extname(pdf))
+      command_list << "pdftotext -enc UTF-8 -layout -q #{pdf} #{name}.txt"
+    end
+  end
+
+end

data/actions/word_count.rb

@@ -0,0 +1,14 @@
+# A parallel WordCount. Depends on the 'wc' utility. 
+class WordCount < CloudCrowd::Action
+  
+  # Count the words in a single book.
+  def process
+    (`wc -w #{input_path}`).match(/\A\s*(\d+)/)[1].to_i
+  end
+  
+  # Sum the total word count.
+  def merge
+    input.inject(0) {|sum, count| sum + count }
+  end
+  
+end

data/bin/crowd

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "#{File.dirname(__FILE__)}/../lib/cloud_crowd/command_line"
+
+CloudCrowd::CommandLine.new

data/cloud-crowd.gemspec

@@ -0,0 +1,111 @@
+Gem::Specification.new do |s|
+  s.name      = 'cloud-crowd'
+  s.version   = '0.1.0'         # Keep version in sync with cloud-cloud.rb
+  s.date      = '2009-09-14'
+
+  s.homepage    = "http://wiki.github.com/documentcloud/cloud-crowd"
+  s.summary     = "Parallel Processing for the Rest of Us"
+  s.description = <<-EOS
+    The crowd, suddenly there where there was nothing before, is a mysterious and
+    universal phenomenon. A few people may have been standing together -- five, ten
+    or twelve, nor more; nothing has been announced, nothing is expected. Suddenly
+    everywhere is black with people and more come streaming from all sides as though
+    streets had only one direction.
+  EOS
+  
+  s.authors           = ['Jeremy Ashkenas']
+  s.email             = 'jeremy@documentcloud.org'
+  s.rubyforge_project = 'cloud-crowd'
+  
+  s.require_paths     = ['lib']
+  s.executables       = ['crowd']
+    
+  s.has_rdoc          = true
+  s.extra_rdoc_files  = ['README']
+  s.rdoc_options      << '--title'    << 'CloudCrowd | Parallel Processing for the Rest of Us' <<
+                         '--exclude'  << 'test' <<
+                         '--main'     << 'README' <<
+                         '--all'
+  
+  s.add_dependency 'sinatra',       ['>= 0.9.4']
+  s.add_dependency 'activerecord',  ['>= 2.3.3']
+  s.add_dependency 'json',          ['>= 1.1.7']
+  s.add_dependency 'rest-client',   ['>= 1.0.3']
+  s.add_dependency 'right_aws',     ['>= 1.10.0']
+  s.add_dependency 'daemons',       ['>= 1.0.10']
+
+  if s.respond_to?(:add_development_dependency)
+    s.add_development_dependency 'faker',               ['>= 0.3.1']
+    s.add_development_dependency 'thoughtbot-shoulda',  ['>= 2.10.2']
+    s.add_development_dependency 'notahat-machinist',   ['>= 1.0.3']
+    s.add_development_dependency 'rack-test',           ['>= 0.4.1']
+    s.add_development_dependency 'mocha',               ['>= 0.9.7']
+  end
+  
+  s.files = %w(
+actions/graphics_magick.rb
+actions/process_pdfs.rb
+actions/word_count.rb
+cloud-crowd.gemspec
+config/config.example.ru
+config/config.example.yml
+config/database.example.yml
+EPIGRAPHS
+examples/graphics_magick_example.rb
+examples/process_pdfs_example.rb
+examples/word_count_example.rb
+lib/cloud-crowd.rb
+lib/cloud_crowd/action.rb
+lib/cloud_crowd/app.rb
+lib/cloud_crowd/asset_store/filesystem_store.rb
+lib/cloud_crowd/asset_store/s3_store.rb
+lib/cloud_crowd/asset_store.rb
+lib/cloud_crowd/command_line.rb
+lib/cloud_crowd/daemon.rb
+lib/cloud_crowd/exceptions.rb
+lib/cloud_crowd/helpers/authorization.rb
+lib/cloud_crowd/helpers/resources.rb
+lib/cloud_crowd/helpers.rb
+lib/cloud_crowd/inflector.rb
+lib/cloud_crowd/models/job.rb
+lib/cloud_crowd/models/work_unit.rb
+lib/cloud_crowd/models/worker_record.rb
+lib/cloud_crowd/models.rb
+lib/cloud_crowd/runner.rb
+lib/cloud_crowd/schema.rb
+lib/cloud_crowd/worker.rb
+LICENSE
+public/css/admin_console.css
+public/css/reset.css
+public/images/bullet_green.png
+public/images/bullet_white.png
+public/images/cloud_hand.png
+public/images/header_back.png
+public/images/logo.png
+public/images/queue_fill.png
+public/images/server_error.png
+public/images/sidebar_bottom.png
+public/images/sidebar_top.png
+public/images/worker_info.png
+public/images/worker_info_loading.gif
+public/js/admin_console.js
+public/js/excanvas.js
+public/js/flot.js
+public/js/jquery.js
+README
+test/acceptance/test_app.rb
+test/acceptance/test_failing_work_units.rb
+test/acceptance/test_word_count.rb
+test/blueprints.rb
+test/config/config.ru
+test/config/config.yml
+test/config/database.yml
+test/config/actions/failure_testing.rb
+test/test_helper.rb
+test/unit/test_action.rb
+test/unit/test_configuration.rb
+test/unit/test_job.rb
+test/unit/test_work_unit.rb
+views/index.erb
+)
+end

data/config/config.example.ru

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+# This rackup script can be used to start the central CloudCrowd server
+# using any Rack-compliant server handler. For example, start up three servers 
+# with a specified port number, using Thin:
+#
+# thin start -R config.ru -p 9173 --servers 3
+
+require 'rubygems'
+require 'cloud-crowd'
+
+CloudCrowd.configure(File.dirname(__FILE__) + '/config.yml')
+CloudCrowd.configure_database(File.dirname(__FILE__) + '/database.yml')
+
+map '/' do
+  run CloudCrowd::App
+end

data/config/config.example.yml

@@ -0,0 +1,48 @@
+# The URL where you're planning on running the central server/queue/database.
+:central_server:          http://localhost:9173
+
+# The storage back-end that you'd like to use for intermediate and final results
+# of processing. 's3' and 'filesystem' are supported. 'filesystem' should only
+# be used in development, or on single-machine installations.
+:storage:                 s3
+
+# Please provide your AWS credentials for S3 storage of job output.
+:aws_access_key:          [your AWS access key]
+:aws_secret_key:          [your AWS secret access key]
+
+# Choose an S3 bucket to store all CloudCrowd output, and decide if you'd like
+# to keep all resulting files on S3 private. If so, you'll receive authenticated
+# S3 URLs as job output, good for 24 hours. If left public, you'll get the
+# straight URLs to the files on S3.
+:s3_bucket:               [your CloudCrowd bucket]
+:use_s3_authentication:   no
+
+# Use HTTP Basic Auth for all requests? (Includes all internal worker requests 
+# to the central server). If yes, specify the login and password that all 
+# requests must provide for authentication.
+:use_http_authentication: no
+:login:                   [your login name]
+:password:                [your password]
+
+# By default, CloudCrowd looks for installed actions inside the 'actions'
+# subdirectory of this configuration folder. 'actions_path' allows you to load
+# additional actions from a location of your choice.
+# :actions_path: /path/to/actions
+
+# Set the following numbers to tweak the configuration of your worker daemons. 
+# Optimum results will depend on proportion of the Memory/CPU/IO bottlenecks
+# in your actions, the number of central servers you have running, and your
+# desired balance between latency and traffic.
+  
+# The number of workers that `crowd workers start` spins up.
+:num_workers:             3
+
+# The minimum number of seconds a worker waits between checking the job queue.
+:min_worker_wait:         1
+
+# The maximum number of seconds a worker waits between checking the job queue.
+:max_worker_wait:         5
+
+# The number of separate attempts that will be made to process an individual
+# work unit, before marking it as having failed.
+:work_unit_retries:       3