mapredus 0.0.1 → 0.0.2

data/README.md

@@ -15,7 +15,7 @@ Goals:
 * simple M/R-style programming for existing Ruby projects
 * low cost of entry (no need for a dedicated cluster)
 
-If you are looking for a high-performance MapReduce implementation
+if you are looking for a high-performance MapReduce implementation
 that can meet your big data needs, try Hadoop.
 
 
@@ -25,26 +25,31 @@ Using MapRedus
 MapRedus uses Resque to handle the processes that it runs, and redis
 to keep a store for the values/data produced.
 
-Workers for a MapRedus process, are Resque workers.  Refer to the
+Workers for a MapRedus process are Resque workers.  Refer to the
 Resque worker documentation to see how to load the necessary
 environment for your worker to be able to run mapreduce processs.  An
 example is also located in the tests.
 
 ### Attaching a mapreduce process to a class
-Often times you'll want to define a mapreduce process that does
-operation on data within a class.  Here is how this looks.  There is
-also an example of this in the tests.
+
+You will often want to define a mapreduce process that does some
+operations on data within a class.  The process should have an
+inputter, mapper, reducer, finalizer, and outputter defined. By
+default a process will have the specifications shown below.  There is
+also an example of how to do this in the tests.
+
     class GetWordCount < MapRedus::Process
-      def self.specification
-        {
-          :inputter => WordStream,
-          :mapper => WordCounter,
-          :reducer => Adder,
-          :finalizer => ToRedisHash,
-          :outputter => MapRedus::RedisHasher,
-          :ordered => false
-        }
-      end
+      inputter MapRedus::WordStream
+      mapper MapRedus::WordCounter
+      reducer MapRedus::Adder
+      finalizer MapRedus::ToRedisHash
+      outputter MapRedus::RedisHasher
+      ordered false
+    end
+
+    class GetCharCount < MapRedus::Process
+      inputter MapRedus::CharStream
+      mapper MapRedus::CharCounter
     end
 
     class Job
@@ -91,10 +96,10 @@ example:
       end
     end
 
-In this example, the inputt stream calls yield to output a mapredus
+In this example, the input stream calls yield to output a mapredus
 file number and a the value that is saved to file (in redis).  The
-mapper's map function calls yield to emit the key value pair for
-storage in redis.  The reducer's reduce function acts similarly.
+mapper's `map` function calls yield to emit the key value pair for
+storage in redis.  The reducer's `reduce` function acts similarly.
 
 The finalizer runs whatever needs to be run when a process completes,
 an example:
@@ -102,7 +107,7 @@ an example:
     class Finalizer < MapRedus::Finalizer
       def self.finalize(process)
         process.each_key_reduced_value do |key, value|
-          process.outputter.encode(process.keyname, key, value)
+          process.outputter.encode(process.result_key, key, value)
         end
         ...
         < set off a new mapredus process to use this stored data >
@@ -127,6 +132,16 @@ hash.
 The default Outputter makes no changes to original result, and tries
 to store that directly into redis as a string.
 
+Working Locally
+---------------
+
+MapRedus uses Bundler to manage dependencies. With Bundler installed:
+
+    bundle install
+
+You should now be able to run tests and do all other tasks with
+`rake`.
+
 Running Tests
 -------------
 
@@ -136,15 +151,15 @@ tests (you'll need to have bundler installed)
 
 Requirements
 ------------
-Bundler (this will install all the requirements below)
-Redis
-RedisSupport
-Resque
-Resque-scheduler
+* Bundler (this will install all the requirements below)
+* Redis
+* RedisSupport
+* Resque
+* Resque-scheduler
 
 ### Notes
-    Instead of calling "emit_intermediate"/"emit" in your map/reduce
-    to produce a key value pair/value you call yield, which will call
+    Instead of calling `emit_intermediate`/`emit` in your map/reduce
+    to produce a key value pair/value you call `yield`, which will call
     emit_intermediate/emit for you.  This gives flexibility in using
     Mapper/Reducer classes especially in testing.
 
@@ -198,17 +213,17 @@ not necessarily in the given order
 
 * think about the following logic
 
-    if a reducer starts working on a key after all maps have finished
+    + if a reducer starts working on a key after all maps have finished
     then when it is done the work on that key is finished forerver
     
-    this would imply a process finishes when all map tasks have
+    + this would imply a process finishes when all map tasks have
     finished and all reduce tasks that start after the map tasks have
     finished
     
-    if a reducer started before all map tasks were finished, then load
+    + if a reducer started before all map tasks were finished, then load
     its reduced result back onto the value list
     
-    if the reducer started after all map tasks finished, then emit the
+    + if the reducer started after all map tasks finished, then emit the
     result
 
 Note on Patches/Pull Requests

data/lib/mapredus.rb

@@ -95,7 +95,6 @@ module MapRedus
 end
 
 require 'mapredus/keys'
-require 'mapredus/process'
 require 'mapredus/filesystem'
 require 'mapredus/master'
 require 'mapredus/mapper'
@@ -104,3 +103,5 @@ require 'mapredus/finalizer'
 require 'mapredus/support'
 require 'mapredus/outputter'
 require 'mapredus/inputter'
+require 'mapredus/default_classes'
+require 'mapredus/process'

data/lib/mapredus/default_classes.rb

@@ -0,0 +1,54 @@
+module MapRedus
+  class WordStream < InputStream
+    def self.scan(data_object)
+      #
+      # The data_object should be a reference to an object that is
+      # stored on your system.  The scanner is used to break up what you
+      # need from the object into manageable pieces for the mapper.  In
+      # this example, the data object is a reference to a redis string.
+      #
+      test_string = FileSystem.get(data_object)
+      
+      test_string.split.each_slice(10).each_with_index do |word_set, i|
+        yield(i, word_set.join(" "))
+      end
+    end
+  end
+
+  class WordCounter < Mapper
+    def self.map(map_data)
+      map_data.split(/\W/).each do |word|
+        next if word.empty?
+        yield(word.downcase, 1)
+      end
+    end
+  end
+
+  class Adder < Reducer
+    def self.reduce(value_list)
+      yield( value_list.reduce(0) { |r, v| r += v.to_i } )
+    end
+  end
+
+  class ToRedisHash < Finalizer
+    def self.finalize(process)
+      process.each_key_reduced_value do |key, value|
+        process.outputter.encode(process.result_key, key, value)
+      end
+    end
+  end
+
+  class RedisHasher < Outputter
+    def self.keys(result_key)
+      FileSystem.hkeys(result_key)
+    end
+
+    def self.encode(result_key, k, v)
+      FileSystem.hset(result_key, k, v)
+    end
+
+    def self.decode(result_key, k)
+      FileSystem.hget(result_key, k)
+    end
+  end
+end

data/lib/mapredus/filesystem.rb

@@ -24,20 +24,20 @@ module MapRedus
     # Setup locks on results using RedisSupport lock functionality
     #
     # Examples
-    #   FileSystem::has_lock?(keyname)
+    #   FileSystem::has_lock?(key)
     #   # => true or false 
     #
     # Returns true if there's a lock
-    def self.has_lock?(keyname)
-      MapRedus.has_redis_lock?( RedisKey.result_cache(keyname) ) 
+    def self.has_lock?(key)
+      MapRedus.has_redis_lock?( RedisKey.result_cache(key) ) 
     end
     
-    def self.acquire_lock(keyname)
-      MapRedus.acquire_redis_lock_nonblock( RedisKey.result_cache(keyname), 60 * 60 )
+    def self.acquire_lock(key)
+      MapRedus.acquire_redis_lock_nonblock( RedisKey.result_cache(key), 60 * 60 )
     end
     
-    def self.release_lock(keyname)
-      MapRedus.release_redis_lock( RedisKey.result_cache(keyname) )
+    def self.release_lock(key)
+      MapRedus.release_redis_lock( RedisKey.result_cache(key) )
     end
   end
 end

data/lib/mapredus/keys.rb

@@ -36,13 +36,6 @@ module MapRedus
   #
   redis_key :temp, "mapredus:process:PID:temp_reduce_key:HASHED_KEY:UNIQUE_REDUCE_HOSTNAME:UNIQUE_REDUCE_PROCESS_ID"
 
-  # If we want to hold on to our final data we have a key to put that data in
-  # In normal map reduce we would just be outputting files
-  #
-  redis_key :result, "mapredus:process:PID:result"
-  redis_key :result_cache, "mapredus:result:KEYNAME"
-
-
   #### USED WITHIN master.rb ####
   
   # Keeps track of the current slaves (by appending "1" to a redis list)

data/lib/mapredus/outputter.rb

@@ -29,14 +29,4 @@ module MapRedus
       FileSystem.set(result_key, Helper.encode(o))
     end
   end
-
-  class RedisHasher < Outputter
-    def self.encode(result_key, k, v)
-      FileSystem.hset(result_key, k, v)
-    end
-
-    def self.decode(result_key, k)
-      FileSystem.hget(result_key, k)
-    end
-  end
 end

data/lib/mapredus/process.rb

@@ -6,16 +6,19 @@ module MapRedus
   # the value of the redis object is a json object which contains: 
   # 
   #   {
+  #     inputter : inputstreamclass,
   #     mapper : mapclass,
   #     reducer : reduceclass,
   #     finalizer : finalizerclass,
+  #     outputter : outputterclass,
   #     partitioner : <not supported>,
   #     combiner : <not supported>,
   #     ordered : true_or_false   ## ensures ordering keys from the map output --> [ order, key, value ],
   #     synchronous : true_or_false   ## runs the process synchronously or not (generally used for testing)
   #     result_timeout : lenght of time a result is saved ## 3600 * 24
-  #     keyname : the location to the save the result of the process (cache location)
+  #     key_args : arguments to be added to the key location of the result save (cache location)
   #     state : the current state of the process (shouldn't be set by the process and starts off as nil)
+  #     type : the original process class ( currently this is needed so we can have namespaces for the result_cache keys )
   #   }
   #
   # The user has the ability in subclassing this class to create extra features if needed
@@ -24,7 +27,7 @@ module MapRedus
     # Public: Keep track of information that may show up as the redis json value
     #         This is so we know exactly what might show up in the json hash
     READERS = [:pid]
-    ATTRS = [:inputter, :mapper, :reducer, :finalizer, :outputter, :ordered, :synchronous, :result_timeout, :keyname, :state]
+    ATTRS = [:inputter, :mapper, :reducer, :finalizer, :outputter, :ordered, :synchronous, :result_timeout, :key_args, :state, :type]
     READERS.each { |r| attr_reader r }
     ATTRS.each { |a| attr_accessor a }
 
@@ -42,10 +45,11 @@ module MapRedus
       @ordered = json_helper(json_info, :ordered)
       @synchronous = json_helper(json_info, :synchronous)
       @result_timeout = json_helper(json_info, :result_timeout) || DEFAULT_TIME
-      @keyname = json_helper(json_info, :keyname)
+      @key_args = json_helper(json_info, :key_args) || []
       @state = json_helper(json_info, :state) || NOT_STARTED
       @outputter = json_helper(json_info, :outputter)
       @outputter = @outputter ? Helper.class_get(@outputter) : MapRedus::Outputter
+      @type = Helper.class_get(json_helper(json_info, :type) || Process)
     end
 
     def json_helper(json_info, key)
@@ -174,7 +178,7 @@ module MapRedus
     #
     # Examples
     #   emit_intermediate(key, value)
-    #   # =>
+    #   # => if an ordering is required
     #   emit_intermediate(rank, key, value)
     #
     # Returns the true on success.
@@ -197,6 +201,16 @@ module MapRedus
       true
     end
 
+    # The emission associated with a reduce.  Currently all reduced
+    # values are pushed onto a redis list.  It may be the case that we
+    # want to directly use a different redis type given the kind of
+    # reduce we are doing.  Often a reduce only returns one value, so
+    # instead of a rpush, we should do a set.
+    # 
+    # Examples
+    #   emit(key, reduced_value)
+    #
+    # Returns "OK" on success.
     def emit(key, reduce_val)
       hashed_key = Helper.hash(key)
       FileSystem.rpush( ProcessInfo.reduce(@pid, hashed_key), reduce_val )
@@ -207,32 +221,6 @@ module MapRedus
             FileSystem.get( ProcessInfo.hash_to_key(@pid, hashed_key) ) == key.to_s )
     end
 
-    # Saves the result to the specified keyname, using the specified outputter
-    #
-    # Example
-    #   (mapreduce:process:result:KEYNAME)
-    # OR
-    #   process:pid:result
-    #
-    # The client must ensure the the result will not be affected when to_s is applied
-    # since redis stores all values as strings
-    #
-    # Returns true on success.
-    def save_result(result)
-      res = @outputter.encode(result)
-      FileSystem.save(ProcessInfo.result(@pid), res)
-      FileSystem.save(ProcessInfo.result_cache(@keyname), res, @result_timeout) if @keyname
-      true
-    end
-
-    def get_saved_result
-      @outputter.decode(Process.get_saved_result(@keyname))
-    end
-
-    def delete_saved_result
-      Process.delete_saved_result(@keyname)
-    end
-
     # Keys that the map operation produced
     #
     # Examples
@@ -248,11 +236,6 @@ module MapRedus
       end
     end
 
-    def num_values(key)
-      hashed_key = Helper.hash(key)
-      FileSystem.llen( ProcessInfo.map(@pid, hashed_key) )
-    end
-
     # values that the map operation produced, for a key
     #
     # Examples
@@ -265,6 +248,10 @@ module MapRedus
       FileSystem.lrange( ProcessInfo.map(@pid, hashed_key), 0, -1 )
     end
 
+    def num_values(key)
+      hashed_key = Helper.hash(key)
+      FileSystem.llen( ProcessInfo.map(@pid, hashed_key) )
+    end
 
     # values that the reduce operation produced, for a key
     #
@@ -278,37 +265,104 @@ module MapRedus
       FileSystem.lrange( ProcessInfo.reduce(@pid, hashed_key), 0, -1 )
     end
 
-    # Map and Reduce are strings naming the Mapper and Reducer
-    # classes we want to run our map reduce with.
-    # 
-    # For instance
-    #   Mapper = "Mapper"
-    #   Reducer = "Reducer"
-    # 
-    # Default finalizer
-    #   "MapRedus::Finalizer"
+    def result_key(*args)
+      Helper.class_get(@type).result_key(*[@key_args, args].flatten)
+    end
+
+    def self.result_key(*args)
+      ProcessInfo.send( "#{self.to_s.gsub(/\W/,"_")}_result_cache", *args )
+    end
+
+    def self.set_result_key(key_struct)
+      MapRedus.redefine_redis_key( "#{self.to_s.gsub(/\W/,"_")}_result_cache", key_struct )
+    end
+
+    # Create sets up a process to be run with the given specification.
+    # It saves the information in the FileSystem and returns an
+    # instance of the process that run should be called on when
+    # running is desired.
     # 
-    # Returns the new process id.
-    def self.create( *args )
+    # Example
+    #   process = MapRedus::Process.create
+    #   process.run
+    #   
+    # Returns an instance of the process
+    def self.create
       new_pid = get_available_pid
-      
-      spec = specification(*args)
-      return nil unless spec
-
-      Process.new(new_pid, spec).save
+      specification = ATTRS.inject({}) do |ret, attr|
+        ret[attr] = send(attr)
+        ret
+      end
+      specification[:type] = self
+      self.new(new_pid, specification).save
     end
-    
-    def self.specification(*args)
-      raise ProcessSpecificationError
+
+    # This defines the attributes to be associated with a MapRedus process
+    # This will allow us to subclass a Process, creating a new specification
+    # by specifying what say the inputter should equal
+    #
+    # Example
+    #   class AnswerDistribution < MapRedus::Process
+    #     inputter = JudgmentStream
+    #     mapper = ResponseFrequencyMap
+    #     reducer = Adder
+    #     finalizer = AnswerCount
+    #     outputter = MapRedus::RedisHasher
+    #   end
+    class << self; attr_reader *ATTRS; end
+
+    # Setter/Getter method definitions to set/get the attribute for
+    # the class. In the getter if it is not defined (nil) then return
+    # the default attribute defined in MapRedus::Process.
+    #
+    # Example
+    #   class AnswerDistribution < MapRedus::Process
+    #     inputter JudgmentStream
+    #     mapper ResponseFrequency
+    #   end
+    #   AnswerDistribution.reducer.should == Adder
+    ATTRS.each do |attr|
+      (class << self; self; end).send(:define_method, attr) do |*one_arg|
+        attribute = "@#{attr}"
+        case one_arg.size
+        when 0
+          instance_variable_get(attribute) || MapRedus::Process.instance_variable_get(attribute)
+        when 1
+          instance_variable_set(attribute, one_arg.first)
+        else
+          raise ArgumentError.new("wrong number of arguments (#{one_arg.size}) when zero or one arguments were expected")
+        end
+      end
     end
 
+    # Default attributes for the process class.  All other attributes
+    # are nil by default.
+    inputter WordStream
+    mapper WordCounter
+    reducer Adder
+    finalizer ToRedisHash
+    outputter RedisHasher
+    type Process
+    
+    # This function returns all the redis keys produced associated
+    # with a process's process id.
+    #
+    # Example
+    #   Process.info(17)
+    #
+    # Returns an array of keys associated with the process id.
     def self.info(pid)
       FileSystem.keys(ProcessInfo.pid(pid) + "*")
     end
     
+    # Returns an instance of the process class given the process id.
+    # If no such process id exists returns nil.
+    #
+    # Example
+    #   process = Process.open(17)
     def self.open(pid)
       spec = Helper.decode( FileSystem.get(ProcessInfo.pid(pid)) )
-      spec && Process.new( pid, spec )
+      spec && self.new( pid, spec )
     end
 
     # Find out what map reduce processes are out there
@@ -331,12 +385,13 @@ module MapRedus
       FileSystem.incrby(ProcessInfo.processes_count, 1 + rand(20)) 
     end
 
-    # Given a result keyname, delete the result
+    # Given a arguments for a result key, delete the result from the
+    # filesystem.
     #
     # Examples
     #   Process.delete_saved_result(key)
-    def self.delete_saved_result(keyname)
-      FileSystem.del( ProcessInfo.result_cache(keyname) )
+    def self.delete_saved_result(*key_args)
+      FileSystem.del( result_key(*key_args) )
     end
     
     # Remove redis keys associated with this process if the Master isn't working.

data/lib/mapredus/support.rb

@@ -4,7 +4,6 @@ module MapRedus
     class DuplicateProcessDefinitionError < MapRedusRunnerError ; end
 
     class Runner
-      attr_reader :process
       def initialize(class_name)
         @class = class_name
       end
@@ -14,7 +13,7 @@ module MapRedus
         if self.respond_to?(mr_process)
           self.send(mr_process, *args, &block)
         else
-          super(method, *args, &block)
+          super(method, *args, &block)  
         end
       end
     end
@@ -24,7 +23,7 @@ module MapRedus
     end
 
     module ClassMethods
-      def mapreduce_process( process_name, mapredus_process_class, result_store, opts = {})
+      def mapreduce_process( process_name, mapredus_process_class, result_store )
         runner_self = Runner
         class_name = self.to_s.gsub(/\W/,"_")
 
@@ -34,17 +33,18 @@ module MapRedus
           raise DuplicateProcessDefintionError
         end
         
-        keyname = "mapredus_key_#{global_process_name}"
-        RedisSupport.redis_key( keyname, result_store )
+        mapredus_process_class.set_result_key( result_store )
 
-        runner_self.send( :define_method, global_process_name ) do |data, *var|
-          @process = mapredus_process_class.create
-          @process.update(:keyname => RedisSupport::Keys.send( keyname, *var ))
-          @process.run(data)
+        runner_self.send( :define_method, global_process_name ) do |data, key_arguments|
+          process = mapredus_process_class.create
+          process.update(:key_args => key_arguments)
+          process.run(data)
+          process
         end
 
-        runner_self.send( :define_method, "#{global_process_name}_result" ) do |*outputter_args|
-          @process.outputter.decode(@process.keyname, *outputter_args)
+        runner_self.send( :define_method, "#{global_process_name}_result" ) do |key_arguments, *outputter_args|
+          key = mapredus_process_class.result_key( *key_arguments )
+          mapredus_process_class.outputter.decode( key, *outputter_args)
         end
       end
     end

data/spec/helper_classes.rb

@@ -1,49 +1,3 @@
-class GetCharCount < MapRedus::Process
-  EXPECTED_ANSWER = {"k"=>2, "v"=>1, " "=>54, ","=>3, "w"=>7, "a"=>17, "l"=>12, "b"=>2, "m"=>4, "c"=>3, "."=>2, "y"=>3, "n"=>18, "D"=>1, "d"=>15, "o"=>13, "p"=>14, "e"=>34, "f"=>6, "r"=>13, "g"=>6, "S"=>1, "s"=>12, "h"=>19, "H"=>1, "t"=>20, "i"=>16, "u"=>5, "j"=>1}
-  def self.specification
-    {
-      :inputter => CharStream,
-      :mapper => CharCounter,
-      :reducer => Adder,
-      :finalizer => ToRedisHash,
-      :outputter => MapRedus::RedisHasher,
-      :ordered => false
-    }
-  end
-end
-
-class GetWordCount < MapRedus::Process
-  TEST = "He pointed his finger in friendly jest and went over to the parapet laughing to himself. Stephen Dedalus stepped up, followed him wearily halfway and sat down on the edge of the gunrest, watching him still as he propped his mirror on the parapet, dipped the brush in the bowl and lathered cheeks and neck."
-  EXPECTED_ANSWER = {"gunrest"=>1, "over"=>1, "still"=>1, "of"=>1, "him"=>2, "and"=>4, "bowl"=>1, "himself"=>1, "went"=>1, "friendly"=>1, "finger"=>1, "propped"=>1, "cheeks"=>1, "dipped"=>1, "down"=>1, "wearily"=>1, "up"=>1, "stepped"=>1, "dedalus"=>1, "to"=>2, "in"=>2, "sat"=>1, "the"=>6, "pointed"=>1, "as"=>1, "followed"=>1, "stephen"=>1, "laughing"=>1, "his"=>2, "he"=>2, "brush"=>1, "jest"=>1, "neck"=>1, "mirror"=>1, "edge"=>1, "on"=>2, "parapet"=>2, "lathered"=>1, "watching"=>1, "halfway"=>1}
-  def self.specification
-    {
-      :inputter => WordStream,
-      :mapper => WordCounter,
-      :reducer => Adder,
-      :finalizer => ToRedisHash,
-      :outputter => MapRedus::RedisHasher,
-      :ordered => false,
-      :keyname => "test:result"
-    }
-  end
-end
-
-class WordStream < MapRedus::InputStream
-  def self.scan(data_object)
-    #
-    # The data_object should be a reference to an object that is
-    # stored on your system.  The scanner is used to break up what you
-    # need from the object into manageable pieces for the mapper.  In
-    # this example, the data object is a reference to a redis string.
-    #
-    test_string = MapRedus::FileSystem.get(data_object)
-    
-    test_string.split.each_slice(10).each_with_index do |word_set, i|
-      yield(i, word_set.join(" "))
-    end
-  end
-end
-
 class CharStream < MapRedus::InputStream
   def self.scan(data_object)
     test_string = MapRedus::FileSystem.get(data_object)
@@ -56,15 +10,6 @@ class CharStream < MapRedus::InputStream
   end
 end
 
-class WordCounter < MapRedus::Mapper
-  def self.map(map_data)
-    map_data.split(/\W/).each do |word|
-      next if word.empty?
-      yield(word.downcase, 1)
-    end
-  end
-end
-
 class CharCounter < MapRedus::Mapper
   def self.map(map_data)
     map_data.each_char do |char|
@@ -73,18 +18,16 @@ class CharCounter < MapRedus::Mapper
   end
 end
 
-class Adder < MapRedus::Reducer
-  def self.reduce(value_list)
-    yield( value_list.reduce(0) { |r, v| r += v.to_i } )
-  end
+class GetCharCount < MapRedus::Process
+  EXPECTED_ANSWER = {"k"=>2, "v"=>1, " "=>54, ","=>3, "w"=>7, "a"=>17, "l"=>12, "b"=>2, "m"=>4, "c"=>3, "."=>2, "y"=>3, "n"=>18, "D"=>1, "d"=>15, "o"=>13, "p"=>14, "e"=>34, "f"=>6, "r"=>13, "g"=>6, "S"=>1, "s"=>12, "h"=>19, "H"=>1, "t"=>20, "i"=>16, "u"=>5, "j"=>1}
+  inputter CharStream
+  mapper CharCounter
 end
 
-class ToRedisHash < MapRedus::Finalizer
-  def self.finalize(process)
-    process.each_key_reduced_value do |key, value|
-      process.outputter.encode(process.keyname, key, value)
-    end
-  end
+class GetWordCount < MapRedus::Process
+  TEST = "He pointed his finger in friendly jest and went over to the parapet laughing to himself. Stephen Dedalus stepped up, followed him wearily halfway and sat down on the edge of the gunrest, watching him still as he propped his mirror on the parapet, dipped the brush in the bowl and lathered cheeks and neck."
+  EXPECTED_ANSWER = {"gunrest"=>1, "over"=>1, "still"=>1, "of"=>1, "him"=>2, "and"=>4, "bowl"=>1, "himself"=>1, "went"=>1, "friendly"=>1, "finger"=>1, "propped"=>1, "cheeks"=>1, "dipped"=>1, "down"=>1, "wearily"=>1, "up"=>1, "stepped"=>1, "dedalus"=>1, "to"=>2, "in"=>2, "sat"=>1, "the"=>6, "pointed"=>1, "as"=>1, "followed"=>1, "stephen"=>1, "laughing"=>1, "his"=>2, "he"=>2, "brush"=>1, "jest"=>1, "neck"=>1, "mirror"=>1, "edge"=>1, "on"=>2, "parapet"=>2, "lathered"=>1, "watching"=>1, "halfway"=>1}
+  set_result_key "test:result"
 end
 
 class Document

data/spec/mapredus_spec.rb

@@ -8,13 +8,41 @@ describe "MapRedus" do
     MapRedus::FileSystem.setnx("wordstream:test", GetWordCount::TEST)
   end
 
+  it "has sets up the correct default classes" do
+    MapRedus::Process.inputter.should == MapRedus::WordStream
+    MapRedus::Process.mapper.should == MapRedus::WordCounter
+    MapRedus::Process.reducer.should == MapRedus::Adder
+    MapRedus::Process.finalizer.should == MapRedus::ToRedisHash
+    MapRedus::Process.outputter.should == MapRedus::RedisHasher
+
+    GetWordCount.result_key.should == "test:result"
+    GetWordCount.inputter.should == MapRedus::WordStream
+    GetWordCount.mapper.should == MapRedus::WordCounter
+    GetWordCount.reducer.should == MapRedus::Adder
+    GetWordCount.finalizer.should == MapRedus::ToRedisHash
+    GetWordCount.outputter.should == MapRedus::RedisHasher
+    
+    GetCharCount.inputter.should == CharStream
+    GetCharCount.mapper.should == CharCounter
+    GetCharCount.reducer.should == MapRedus::Adder
+    GetCharCount.finalizer.should == MapRedus::ToRedisHash
+    GetCharCount.outputter.should == MapRedus::RedisHasher
+  end
+
   it "creates a process successfully" do
     process = GetWordCount.open(@process.pid)
     
-    process.inputter.should == WordStream
-    process.mapper.should == WordCounter
-    process.reducer.should == Adder
-    process.finalizer.should == ToRedisHash
+    process.inputter.should == MapRedus::WordStream
+    process.mapper.should == MapRedus::WordCounter
+    process.reducer.should == MapRedus::Adder
+    process.finalizer.should == MapRedus::ToRedisHash
+    process.outputter.should == MapRedus::RedisHasher
+
+    process = GetCharCount.create
+    process.inputter.should == CharStream
+    process.mapper.should == CharCounter
+    process.reducer.should == MapRedus::Adder
+    process.finalizer.should == MapRedus::ToRedisHash
     process.outputter.should == MapRedus::RedisHasher
   end
 
@@ -22,6 +50,7 @@ describe "MapRedus" do
     ##
     ## In general map reduce shouldn't be running operations synchronously
     ##
+    @process.class.should == GetWordCount
     @process.run("wordstream:test", synchronously = true)
     @process.map_keys.size.should == GetWordCount::EXPECTED_ANSWER.size
 
@@ -31,7 +60,7 @@ describe "MapRedus" do
     end
 
     @process.each_key_reduced_value do |key, value|
-      @process.outputter.decode(@process.keyname, key).to_i.should == GetWordCount::EXPECTED_ANSWER[key]
+      @process.outputter.decode(@process.result_key, key).to_i.should == GetWordCount::EXPECTED_ANSWER[key]
     end
   end
 
@@ -46,7 +75,7 @@ describe "MapRedus" do
     end
 
     @process.each_key_reduced_value do |key, value|
-      @process.outputter.decode(@process.keyname, key).to_i.should == GetWordCount::EXPECTED_ANSWER[key]
+      @process.outputter.decode(@process.result_key, key).to_i.should == GetWordCount::EXPECTED_ANSWER[key]
     end
   end
 end
@@ -158,8 +187,8 @@ describe "MapRedus Process" do
 
   it "emit_intermediate on an ordered process" do
     @process.update(:ordered => true)
-    @process.emit_intermediate(1, "number", "one")
     @process.emit_intermediate(2, "place", "two")
+    @process.emit_intermediate(1, "number", "one")
     res = []
     @process.each_key_nonreduced_value do |key, value|
       res << [key, value]
@@ -207,14 +236,14 @@ describe "MapRedus Master" do
   end
 
   it "handles slaves (enslaving) correctly" do
-    MapRedus::Master.enslave(@process, WordCounter, @process.pid, "test")
-    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid, "test"], "class"=>"WordCounter"}]
+    MapRedus::Master.enslave(@process, MapRedus::WordCounter, @process.pid, "test")
+    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid, "test"], "class"=>"MapRedus::WordCounter"}]
     MapRedus::Master.slaves(@process.pid).should == ["1"]
   end
 
   it "handles slaves (freeing) correctly" do
-    MapRedus::Master.enslave(@process, WordCounter, @process.pid, "test")
-    MapRedus::Master.enslave(@process, WordCounter, @process.pid, "test")
+    MapRedus::Master.enslave(@process, MapRedus::WordCounter, @process.pid, "test")
+    MapRedus::Master.enslave(@process, MapRedus::WordCounter, @process.pid, "test")
 
     MapRedus::Master.slaves(@process.pid).should == ["1", "1"]
 
@@ -237,12 +266,12 @@ describe "MapRedus Mapper/Reducer/Finalizer" do
     @process.update(:state => MapRedus::INPUT_MAP_IN_PROGRESS)
     @process.state.should == MapRedus::INPUT_MAP_IN_PROGRESS
     @process.inputter.perform(@process.pid, "wordstream:test")
-    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid, 0], "class"=>"WordCounter"}]
+    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid, 0], "class"=>"MapRedus::WordCounter"}]
     Resque.pop(:mapredus)
     @process.mapper.perform(@process.pid, 0)
     @process.reload
     @process.state.should == MapRedus::REDUCE_IN_PROGRESS
-    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid, "data"], "class"=>"Adder"}]
+    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid, "data"], "class"=>"MapRedus::Adder"}]
   end
 
   it "runs a reduce correctly proceeding to the correct next state" do
@@ -252,7 +281,7 @@ describe "MapRedus Mapper/Reducer/Finalizer" do
     @process.reducer.perform(@process.pid, "data")
     @process.reload
     @process.state.should == MapRedus::FINALIZER_IN_PROGRESS
-    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid], "class"=>"ToRedisHash"}]
+    Resque.peek(:mapredus, 0, -1).should == [{"args"=>[@process.pid], "class"=>"MapRedus::ToRedisHash"}]
   end
 
   it "should test that the finalizer correctly saves" do
@@ -268,7 +297,7 @@ describe "MapRedus Mapper/Reducer/Finalizer" do
   end
 end
 
-describe "MapReduce Support" do
+describe "MapRedus Support" do
   before(:each) do
     MapRedus::FileSystem.flushall
     @doc = Document.new(10)
@@ -285,11 +314,11 @@ describe "MapReduce Support" do
     work_off
 
     GetCharCount::EXPECTED_ANSWER.keys.each do |char|
-      @doc.mapreduce.char_count_result(char).should == GetCharCount::EXPECTED_ANSWER[char].to_s
+      @doc.mapreduce.char_count_result([@doc.id], char).should == GetCharCount::EXPECTED_ANSWER[char].to_s
     end
     
     other_answer.keys.each do |char|
-      @other_doc.mapreduce.char_count_result(char).should == other_answer[char].to_s
+      @other_doc.mapreduce.char_count_result([@other_doc.id], char).should == other_answer[char].to_s
     end
   end
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: mapredus
 version: !ruby/object:Gem::Version 
-  hash: 29
+  hash: 27
   prerelease: false
   segments: 
   - 0
   - 0
-  - 1
-  version: 0.0.1
+  - 2
+  version: 0.0.2
 platform: ruby
 authors: 
 - John Le
@@ -16,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-07-06 00:00:00 -07:00
+date: 2010-07-09 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -89,6 +89,7 @@ extra_rdoc_files:
 - README.md
 files: 
 - lib/mapredus.rb
+- lib/mapredus/default_classes.rb
 - lib/mapredus/filesystem.rb
 - lib/mapredus/finalizer.rb
 - lib/mapredus/inputter.rb