mooktakim-cloud-crowd 0.3.4

data/EPIGRAPHS

@@ -0,0 +1,17 @@
+The crowd, suddenly there where there was nothing before, is a mysterious and
+universal phenomenon. A few people may have been standing together -- five, ten
+or twelve, nor more; nothing has been announced, nothing is expected. Suddenly
+everywhere is black with people and more come streaming from all sides as though
+streets had only one direction. Most of them do not know what has happened and, 
+if questioned, have no answer; but they hurry to be there where most other
+people are. There is a determination in their movement which is quite different
+from the expression of ordinary curiosity. It seems as through the movement of 
+some of them transmits itself to all the others. But that is not all; they have
+a goal which is there before they can find words for it. -p 16
+
+Crowd crystals are the small, rigid groups of men, strictly delimited and of
+great constancy, which serve to precipitate crowds. Their structure is such 
+that they can be comprehended and taken in at a glance. Their unity is more
+important than their size. -p 73
+
+From Elias Canetti's "Crowds and Power" (1962).

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2009 Jeremy Ashkenas, DocumentCloud
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,93 @@
+=                                                                               
+           _  _                                                                
+          ( `   )_                                                             
+         (    )    `)                                                          
+       (_   (_ .  _) _)                                                        
+                                      _                                        
+                                     (  )                                      
+      _ .                         ( `  ) . )                                   
+    (  _ )_                      (_, _(  ,_)_)                                 
+  (_  _(_ ,)                                                                   
+                                                                               
+           _  _               ___ _             _  ___                   _     
+          ( `   )_           / __| |___ _  _ __| |/ __|_ _ _____ __ ____| |    
+         (    )    `)       | (__| / _ \ || / _` | (__| '_/ _ \ V  V / _` |    
+       (_   (_ .  _) _)      \___|_\___/\_,_\__,_|\___|_| \___/\_/\_/\__,_|    
+                                                                               
+                                                     _                         
+                                                    (  )                       
+                  _, _ .                         ( `  ) . )                    
+                 ( (  _ )_                      (_, _(  ,_)_)                  
+               (_(_  _(_ ,)                                                    
+                                                                               
+                                                                               
+                                                                               
+  ~ CloudCrowd ~
+
+    * Parallel processing for the rest of us
+    * Write your scripts in Ruby
+    * Works with Amazon EC2 and S3
+    * split -> process -> merge
+    * As easy as `gem install cloud-crowd`
+
+    Well-suited for:
+    
+    * Generating or resizing images.
+    * Encoding video.
+    * Running text extraction or OCR on PDFs.
+    * Migrating a large file set or database.
+    * Web scraping.
+    
+    
+  ~ Documentation ~
+  
+    Wiki: http://wiki.github.com/documentcloud/cloud-crowd
+    Rdoc: http://rdoc.info/projects/documentcloud/cloud-crowd
+  
+  
+  ~ Getting started ~
+  
+    # Install the gem.
+    
+      >> sudo gem install cloud-crowd
+    
+    # Install the CloudCrowd configuration files to a location of your choosing.
+    
+      >> crowd install ~/config/cloud-crowd
+    
+    # Now, you can use the full complement of `crowd` commands from inside of
+    # this configuration directory. To see the available commands:
+    
+      >> crowd --help
+    
+    # Edit the configuration files to your satisfaction, add AWS credentials, 
+    # and then load the CloudCrowd schema into your configured database.
+    
+      >> cd ~/config/cloud-crowd
+      >> mate config.yml
+      >> mate database.yml
+      >> [create the database you just configured...]
+      >> crowd load_schema
+    
+    # Write your actions, and install them into the 'actions' subdirectory.
+    # CloudCrowd comes with a few default actions as an example.
+    
+    # To launch the central server (make sure that you include its location
+    # in config.yml):
+    
+      >> crowd server
+    
+    # The configuration folder also includes 'config.ru', which can be used by
+     # any Rack-compliant webserver to run your central server.
+    
+    # Then, to launch a node of workers:
+    
+      >> crowd node
+    
+    # To spin up remote nodes, install the 'cloud-crowd' gem and copy over
+    # your configuration directory. Run `crowd node`, and the remote machines
+    # will register with the central server, becoming available for processing.
+    
+    # At this point you can visit your Operations Center at localhost:9173 to 
+    # view all of your nodes, ready for action.
+  

data/actions/graphics_magick.rb

@@ -0,0 +1,43 @@
+# The GraphicsMagick action, dependent on the `gm` command, is able to perform
+# any number of GraphicsMagick conversions on an image passed in as an input.
+# The options hash should specify the +name+ for the particular step (which is
+# appended to the resulting image filename) the +command+ (eg. convert, mogrify), 
+# the +options+ (to the command, eg. -shadow -blur), and the +extension+ which 
+# will determine the resulting image type. Optionally, you may also specify
+# +input+ as the name of a previous step; doing this will use the result of
+# that step as the source image, otherwise each step uses the original image
+# as its source.
+class GraphicsMagick < CloudCrowd::Action
+  
+  # Download the initial image, and run each of the specified GraphicsMagick
+  # commands against it, returning the aggregate output.
+  def process
+    options['steps'].inject({}) {|h, step| h[step['name']] = run_step(step); h }
+  end
+  
+  # Run an individual step (single GraphicsMagick command) in a shell-injection
+  # safe way, uploading the result to the AssetStore, and returning the public
+  # URL as the result.
+  # TODO: +system+ wasn't working, figure out some other way to escape.
+  def run_step(step)
+    cmd, opts = step['command'], step['options']
+    in_path, out_path = input_path_for(step), output_path_for(step)
+    `gm #{cmd} #{opts} #{in_path} #{out_path}`
+    save(out_path)    
+  end
+  
+  # Where should the starting image be located?
+  # If you pass in an optional step, returns the path to that step's output
+  # as input for further processing.
+  def input_path_for(step)
+    in_step = step && step['input'] && options['steps'].detect {|s| s['name'] == step['input']}
+    return input_path unless in_step
+    return output_path_for(in_step)
+  end
+  
+  # Where should resulting images be saved locally?
+  def output_path_for(step)
+    "#{work_directory}/#{file_name}_#{step['name']}.#{step['extension']}"
+  end
+  
+end

data/actions/process_pdfs.rb

@@ -0,0 +1,92 @@
+# Depends on working pdftk, gm (GraphicsMagick), and pdftotext (Poppler) commands.
+# Splits a pdf into batches of N pages, creates their thumbnails and icons,
+# as specified in the Job options, gets the text for every page, and merges 
+# it all back into a tar archive for convenient download.
+#
+# See <tt>examples/process_pdfs_example.rb</tt> for more information.
+class ProcessPdfs < CloudCrowd::Action
+  
+  # Split up a large pdf into single-page pdfs. Batch them into 'batch_size'
+  # chunks for processing. The double pdftk shuffle fixes the document xrefs.
+  def split
+    `pdftk #{input_path} burst output "#{file_name}_%05d.pdf_temp"`
+    FileUtils.rm input_path
+    pdfs = Dir["*.pdf_temp"]
+    pdfs.each {|pdf| `pdftk #{pdf} output #{File.basename(pdf, '.pdf_temp')}.pdf`}
+    pdfs = Dir["*.pdf"]
+    batch_size = options['batch_size']
+    batches = (pdfs.length / batch_size.to_f).ceil
+    batches.times do |batch_num|
+      tar_path = "#{sprintf('%05d', batch_num)}.tar"
+      batch_pdfs = pdfs[batch_num*batch_size...(batch_num + 1)*batch_size]
+      `tar -czf #{tar_path} #{batch_pdfs.join(' ')}`
+    end
+    Dir["*.tar"].map {|tar| save(tar) }
+  end
+
+  # Convert a pdf page into different-sized thumbnails. Grab the text.
+  def process
+    `tar -xzf #{input_path}`
+    FileUtils.rm input_path
+    cmds = []
+    generate_images_commands(cmds)
+    generate_text_commands(cmds)
+    system cmds.join(' && ')
+    FileUtils.rm Dir['*.pdf']
+    `tar -czf #{file_name}.tar *`
+    save("#{file_name}.tar")
+  end
+  
+  # Merge all of the resulting images, all of the resulting text files, and
+  # the concatenated merge of the full-text into a single tar archive, ready to
+  # for download.
+  def merge
+    input.each do |batch_url|
+      batch_path = File.basename(batch_url)
+      download(batch_url, batch_path)
+      `tar -xzf #{batch_path}`
+      FileUtils.rm batch_path
+    end
+    
+    names = Dir['*.txt'].map {|fn| fn.sub(/_\d+(_\w+)?\.txt\Z/, '') }.uniq
+    dirs = names.map {|n| ["#{n}/text/full", "#{n}/text/pages"] + options['images'].map {|i| "#{n}/images/#{i['name']}" } }.flatten
+    FileUtils.mkdir_p(dirs)
+    
+    Dir['*.*'].each do |file|
+      ext = File.extname(file)
+      name = file.sub(/_\d+(_\w+)?#{ext}\Z/, '')
+      if ext == '.txt'
+        FileUtils.mv(file, "#{name}/text/pages/#{file}")
+      else
+        suffix      = file.match(/_([^_]+)#{ext}\Z/)[1]
+        sans_suffix = file.sub(/_([^_]+)#{ext}\Z/, ext)
+        FileUtils.mv(file, "#{name}/images/#{suffix}/#{sans_suffix}")
+      end
+    end
+    
+    names.each {|n| `cat #{n}/text/pages/*.txt > #{n}/text/full/#{n}.txt` }
+    
+    `tar -czf processed_pdfs.tar *`
+    save("processed_pdfs.tar")
+  end
+  
+  
+  private
+  
+  def generate_images_commands(command_list)
+    Dir["*.pdf"].each do |pdf| 
+      name = File.basename(pdf, File.extname(pdf))
+      options['images'].each do |i|
+        command_list << "gm convert #{i['options']} #{pdf} #{name}_#{i['name']}.#{i['extension']}"
+      end
+    end
+  end
+  
+  def generate_text_commands(command_list)
+    Dir["*.pdf"].each do |pdf|
+      name = File.basename(pdf, File.extname(pdf))
+      command_list << "pdftotext -enc UTF-8 -layout -q #{pdf} #{name}.txt"
+    end
+  end
+
+end

data/actions/word_count.rb

@@ -0,0 +1,16 @@
+# A parallel WordCount. Depends on the 'wc' utility.
+class WordCount < CloudCrowd::Action
+
+  # Count the words in a single book.
+  # Pretend that this takes longer than it really does, for demonstration purposes.
+  def process
+    sleep 5
+    (`wc -w #{input_path}`).match(/\A\s*(\d+)/)[1].to_i
+  end
+
+  # Sum the total word count.
+  def merge
+    input.inject(0) {|sum, count| sum + count }
+  end
+
+end

data/bin/crowd

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "#{File.dirname(__FILE__)}/../lib/cloud_crowd/command_line"
+
+CloudCrowd::CommandLine.new

data/config/config.example.ru

@@ -0,0 +1,23 @@
+#!/usr/bin/env ruby
+
+# This rackup script can be used to start the central CloudCrowd server
+# using any Rack-compliant server handler. For example, start up three servers 
+# with a specified port number, using Thin:
+#
+# thin start -R config.ru --servers 3
+#
+# Or a single server with Unicorn:
+#
+# unicorn config.ru
+#
+
+
+require 'rubygems'
+require 'cloud-crowd'
+
+CloudCrowd.configure(::File.dirname(__FILE__) + '/config.yml')
+CloudCrowd.configure_database(::File.dirname(__FILE__) + '/database.yml')
+
+map '/' do
+  run CloudCrowd::Server
+end

data/config/config.example.yml

@@ -0,0 +1,55 @@
+# The URL where you're planning on running the central server/queue/database.
+:central_server:      http://localhost:9173
+
+# The following settings allow you to control the number of workers that can run
+# on a given node, to prevent the node from becoming overloaded. 'max_workers' 
+# is a simple cap on the maximum number of workers a node is allowed to run
+# concurrently. 'max_load' is the maximum (one-minute) load average, above which
+# a node will refuse to take new work. 'min_free_memory' is the minimum amount
+# of free RAM (in megabytes) a node is allowed to have, below which no new 
+# workers are run. These settings may be used in any combination.
+:max_workers:         5
+# :max_load:            5.0
+# :min_free_memory:     150
+
+# The storage back-end that you'd like to use for intermediate and final results
+# of processing. 's3' and 'filesystem' are supported. 'filesystem' should only
+# be used in development, on single-machine installations, or networked drives.
+# If you *are* developing an action, filesystem is certainly faster and easier.
+:storage:             s3
+
+# Please provide your AWS credentials for S3 storage of job output.
+:aws_access_key:      [your AWS access key]
+:aws_secret_key:      [your AWS secret access key]
+
+# Choose an S3 bucket to store all CloudCrowd output, and decide if you'd like
+# to keep all resulting files on S3 private. If so, you'll receive authenticated
+# S3 URLs as job output, good for 24 hours. If left public, you'll get the
+# straight URLs to the files on S3.
+:s3_bucket:           [your CloudCrowd bucket]
+:s3_authentication:   no
+
+# The following settings configure local paths. 'local_storage_path' is the 
+# directory in which all files will be saved if you're using the 'filesystem'
+# storage. 'log_path' and 'pid_path' are the directories in which daemonized 
+# servers and nodes will store their process ids and log files. The default 
+# values are listed.
+# :local_storage_path:  /tmp/cloud_crowd_storage
+# :log_path:            log
+# :pid_path:            tmp/pids
+
+# Use HTTP Basic Auth for all requests? (Includes all internal worker requests 
+# to the central server). If yes, specify the login and password that all 
+# requests must provide for authentication.
+:http_authentication: no
+:login:               [your login name]
+:password:            [your password]
+
+# By default, CloudCrowd looks for installed actions inside the 'actions'
+# subdirectory of this configuration folder. 'actions_path' allows you to load
+# additional actions from a location of your choice.
+# :actions_path: /path/to/actions
+
+# The number of separate attempts that will be made to process an individual
+# work unit, before marking it as having failed.
+:work_unit_retries:   3

data/config/database.example.yml

@@ -0,0 +1,16 @@
+# This is a standard ActiveRecord database.yml file. You can configure it 
+# to use any database that ActiveRecord supports. Only the central server needs
+# this file to be configured -- nodes never connect directly to the database.
+
+:adapter:  mysql
+:encoding: utf8
+:username: root
+:password:
+:socket:   /tmp/mysql.sock
+:database: cloud_crowd
+
+# If you'd prefer to use an SQLite database instead, the following configuration
+# will do nicely:
+#
+# :adapter:  sqlite3
+# :database: cloud_crowd.db

data/examples/graphics_magick_example.rb

@@ -0,0 +1,44 @@
+#!/usr/bin/env ruby -rubygems
+
+require 'restclient'
+require 'json'
+
+# This example demonstrates the GraphicsMagick action by taking in a list of
+# five images, and producing annotated, blurred, and black and white versions
+# of each image. See actions/graphics_magick.rb
+
+RestClient.post('http://localhost:9173/jobs', 
+  {:job => {
+  
+    'action' => 'graphics_magick',
+    
+    'inputs' => [
+      'http://www.sci-fi-o-rama.com/wp-content/uploads/2008/10/dan_mcpharlin_the_land_of_sleeping_things.jpg',
+      'http://www.sci-fi-o-rama.com/wp-content/uploads/2009/07/dan_mcpharlin_wired_spread01.jpg',
+      'http://www.sci-fi-o-rama.com/wp-content/uploads/2009/07/dan_mcpharlin_wired_spread03.jpg',
+      'http://www.sci-fi-o-rama.com/wp-content/uploads/2009/07/dan_mcpharlin_wired_spread02.jpg',
+      'http://www.sci-fi-o-rama.com/wp-content/uploads/2009/02/dan_mcpharlin_untitled.jpg'
+    ],
+    
+    'options' => {
+      'steps' => [{
+        'name'      => 'annotated',
+        'command'   => 'convert',
+        'options'   => '-font helvetica -fill red -draw "font-size 35; text 75,75 CloudCrowd!"',
+        'extension' => 'jpg'
+      },{
+        'name'      => 'blurred',
+        'command'   => 'convert',
+        'options'   => '-blur 10x5',
+        'extension' => 'png'
+      },{
+        'name'      => 'bw', 
+        'input'     => 'blurred',
+        'command'   => 'convert', 
+        'options'   => '-monochrome', 
+        'extension' => 'jpg'
+      }]
+    }
+    
+  }.to_json}
+)

data/examples/process_pdfs_example.rb

@@ -0,0 +1,40 @@
+#!/usr/bin/env ruby -rubygems
+
+require 'restclient'
+require 'json'
+
+# This example demonstrates a fairly complicated PDF-processing action, designed
+# to extract the PDF's text, and produce GIF versions of each page. The action
+# (actions/process_pdfs.rb) shows an example of using all three steps,
+# split, process, and merge.
+
+RestClient.post('http://localhost:9173/jobs',
+  {:job => {
+  
+    'action' => 'process_pdfs',
+    
+    'inputs' => [
+      'http://tigger.uic.edu/~victor/personal/futurism.pdf',
+      'http://www.jonasmekas.com/Catalog_excerpt/The%20Avant-Garde%20From%20Futurism%20to%20Fluxus.pdf',
+      'http://www.dzignism.com/articles/Futurist.Manifesto.pdf',
+      'http://www.pitt.edu/~slavic/sisc/SISC4/dadswell.pdf'
+    ],
+    
+    'options' => {
+      
+      'batch_size' => 7,
+      
+      'images' => [{
+        'name'      => '700',
+        'options'   => '-resize 700x -density 220 -depth 4 -unsharp 0.5x0.5+0.5+0.03',
+        'extension' => 'gif'
+      },{
+        'name'      => '1000',
+        'options'   => '-resize 1000x -density 220 -depth 4 -unsharp 0.5x0.5+0.5+0.03',
+        'extension' => 'gif'
+      }]
+      
+    }
+    
+  }.to_json}
+)

data/examples/word_count_example.rb

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby -rubygems
+
+require 'restclient'
+require 'json'
+
+# Let's count all the words in Shakespeare.
+
+RestClient.post('http://localhost:9173/jobs', 
+  {:job => {
+  
+    'action' => 'word_count',
+    
+    'inputs' => [
+      'http://www.gutenberg.org/dirs/etext97/1ws3010.txt',  # All's Well That Ends Well
+      'http://www.gutenberg.org/dirs/etext99/1ws3511.txt',  # Anthony and Cleopatra
+      'http://www.gutenberg.org/dirs/etext97/1ws2510.txt',  # As You Like It
+      'http://www.gutenberg.org/dirs/etext97/1ws0610.txt',  # The Comedy of Errors
+      'http://www.gutenberg.org/dirs/etext99/1ws3911.txt',  # Cymbeline
+      'http://www.gutenberg.org/dirs/etext00/0ws2610.txt',  # Hamlet
+      'http://www.gutenberg.org/dirs/etext00/0ws1910.txt',  # Henry IV
+      'http://www.gutenberg.org/dirs/etext99/1ws2411.txt',  # Julius Caesar
+      'http://www.gutenberg.org/dirs/etext98/2ws3310.txt',  # King Lear
+      'http://www.gutenberg.org/dirs/etext99/1ws1211j.txt', # Love's Labour's Lost
+      'http://www.gutenberg.org/dirs/etext98/2ws3410.txt',  # Macbeth
+      'http://www.gutenberg.org/dirs/etext98/2ws1810.txt',  # The Merchant of Venice
+      'http://www.gutenberg.org/dirs/etext99/1ws1711.txt',  # Midsummer Night's Dream
+      'http://www.gutenberg.org/dirs/etext98/3ws2210.txt',  # Much Ado About Nothing
+      'http://www.gutenberg.org/dirs/etext00/0ws3210.txt',  # Othello
+      'http://www.gutenberg.org/dirs/etext98/2ws1610.txt',  # Romeo and Juliet
+      'http://www.gutenberg.org/dirs/etext98/2ws1010.txt',  # The Taming of the Shrew
+      'http://www.gutenberg.org/dirs/etext99/1ws4111.txt',  # The Tempest
+      'http://www.gutenberg.org/dirs/etext00/0ws0910.txt',  # Titus Andronicus
+      'http://www.gutenberg.org/dirs/etext99/1ws2911.txt',  # Troilus and Cressida
+      'http://www.gutenberg.org/dirs/etext98/3ws2810.txt',  # Twelfth Night
+      'http://www.gutenberg.org/files/1539/1539.txt'        # The Winter's Tale
+    ]
+    
+  }.to_json}
+)
+
+# With 23 Workers running, and over Wifi, it counted all the words in 5.5 secs.
+# On a fast internet connection, you may not even see this job show up.