map_reduce 0.0.1.alpha4 → 0.0.1.alpha5

data/README.md

@@ -18,9 +18,183 @@ Or install it yourself as:
 
     $ gem install mapreduce
 
+## Introduction
+
+MapReduce has got three entities:
+
+* Master
+* Mapper
+* Reducer
+
+Perhaps later Manager will be presented to synchronyse Reducers.
+
+### Master
+
+Master is a process who accepts emmited by Mappers data, sorts it and sends grouped data to Reducers. One Master can serve multiple tasks (multiple Mappers clusters).
+
+To run Master you could specify following options
+
+* TCP/IPC/Unix (TCP if you need to work over Network) socket address to bind; Workers will connect to this address (default is `tcp://127.0.0.1:5555`)
+* Logs folder to store temprorary logs with received data (default is `/tmp/map_reduce`); be sure to add read/write access for proccess to this folder
+* Delimeter for key, value (default is `\t`); sometimes you want to set your own delimeter if TAB could be found in your key
+
+Also you could define some blocks of code. It could be useful for getting some stats from Master.
+
+* after_map - this block will be executed after Master received emmited data
+* after_reduce - this block will be executed after Master sended data to Reducer
+
+All blocks recieves `|key, value, task_name|`
+
+Simple Master
+
+```ruby
+require 'map_reduce'
+# Default params
+master = MapReduce::Master.new
+# Same as
+master = MapReduce::Master.new socket: "tcp://127.0.0.1:555",
+                               log_folder: "/tmp/map_reduce",
+                               delimiter: "\t"
+
+# Define some logging after map and reduce
+master.after_map do |key, value, task|
+  puts "Task: #{task}, received key: #{key}"
+end
+
+master.after_reduce do |key, values, task|
+  puts "Task: #{task}, for key: #{key} was sended #{values.size} items"
+end
+
+# Run Master
+master.run
+```
+
+### Mapper
+
+Mapper emmits data to masters. It could read log, database, or answer to phone calls. What should Mapper know is how to connect to Masters and it is ready to go. Also you could choose mode in which you want to work. Worker works asynchronously, but you could choose if you want to write callbacks (pure EventMachine) or you prefer to wrap it in Fibers (em-synchrony, for example). Also you could specify task name to worker if Masters serve many tasks.
+
+* masters - is an array of all available Masters' sockets
+* type - `:em` or `:sync` (`:em` is default)
+* task - task name, default is `nil`
+
+For example, we have got some Web application (shop) and you want to explore which goods people look with each other.
+
+(Let's suppose that web server is running under EventMachine and each request is spawned in Fiber)
+
+```ruby
+# Define somewhere your worker
+require 'map_reduce'
+
+@worker = MapReduce::Worker.new type: :sync, 
+            masters: ["tcp://192.168.1.1:5555", "tcp://192.168.1.2:5555"],
+            task: "goods"
+
+# And use it in your Web App
+get "/good/:id" do
+  @good = Good.find(params[:id])
+  # Send current user's id and good's id
+  @worker.map(current_user.id, @good.id)
+  haml :good
+end
+```
+
+Also Mapper has got `wait_for_all` method. If you are mapping data not permanently and you need to know when all mappers finished mapping data you should call this method.
+
+```ruby
+rand(1000000).times do |i|
+  mapper.map(i, 1)
+end
+mapper.wait_for_all
+```
+
+So you will be blocked till all servers will finish mapping data. Then you could start reducing data, for example.
+
+### Reducer
+
+Reducer is a guy who receives grouped data from Masters. In our previous example with shop Reducer will recieve all goods that current user visited for every user. So now you can use some ML algorithms, or append data to existing GoodsGraph or whatever science.
+
+As a Worker, Reducer should know masters' sockets addresses, type of connection and task name if needed (if Mapper emits data with named task, Reducer should specify it as well).
+
+```ruby
+require 'em-synchrony'
+require 'map_reduce'
+# initialize one
+reducer = MapReduce::Reducer.new type: :sync, 
+            masters: ["tcp://192.168.1.1:5555", "tcp://192.168.1.2:5555"],
+            task: "goods"
+
+# Let's give masters to collect some data between each reduce and sleep for a while
+EM.synchrony do
+  while true
+    reducer.reduce do |key, values|
+      # You can do some magick here
+      puts "User: #{key}, visited #{values} today"
+    end
+    EM::Synchrony.sleep(60 * 60 * 3)
+  end
+end
+```
+
 ## Usage
 
-TODO
+So. Generally you need to specify two thigs:
+
+* What to map
+* How to reduce
+
+And implement it with given primitives. 
+
+Maybe the simplest example should be count of page visits (video views, tracks listens) for each article. In the case you have got millions of visits incrementing your data for each visit in RDBMS could be very expensive operation. So updating one/two times per day in some cases is a good choice. So we have got bunch of logs `article_id, user_id, timestamp` on each frontend and we need to count visits for each article and increment it in database.
+
+
+So on each server you could run Master, Mapper and Reducer.
+
+You could even combine Mapper and Reducer in one process, becuse you need to fire Reducer right after you have finished your map phase.
+
+```ruby
+# master.rb
+require 'map_reduce'
+
+MapReduce::Master.new(socket: "#{current_ip}:5555")
+```
+
+```ruby
+# map_reducer.rb
+require 'map_reduce'
+require 'em-synchrony'
+
+@mapper = MapReduce::Mapper.new masters: [ ... ], type: :sync
+@reducer = MapReduce::Reducer.new masters: [ ... ], type: :sync
+
+EM.synchrony do
+  # Run process each 12 hours
+  EM::Synchrony.add_periodic_timer(60*60*12) do
+    File.open("/path/to/log").each do |line|
+      article_id, user_id, timestamp = line.chomp.split(", ")
+      @mapper.map(article_id, 1)
+    end
+
+    @mapper.wait_for_all
+
+    @reducer.reduce do |key, values|
+      # How many time article was visited
+      count = values.size
+      # Let's increment this value
+      Article.increment(id: key, visits: count)
+    end
+  end
+end
+```
+
+And run them
+
+    $ ruby master.rb
+    $ ruby map_reducer.rb
+
+
+## Summary
+
+It is pretty simple implementation of map reduce and it doesn't solve synchronizing, loosing connectivity, master/worker/reducer failing problems. They are totally up to developers. And there is Hadoop for really big map reduce problems.
 
 ## Contributing
 
@@ -28,4 +202,4 @@ TODO
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+5. Create new Pull Request

data/lib/map_reduce.rb

@@ -17,5 +17,11 @@ module MapReduce
   end
 end
 
+require File.expand_path("../map_reduce/exceptions", __FILE__)
+require File.expand_path("../map_reduce/socket/req_fiber", __FILE__)
+require File.expand_path("../map_reduce/map_log", __FILE__)
+require File.expand_path("../map_reduce/reduce_log", __FILE__)
+require File.expand_path("../map_reduce/socket/master", __FILE__)
 require File.expand_path("../map_reduce/master", __FILE__)
-require File.expand_path("../map_reduce/worker", __FILE__)
+require File.expand_path("../map_reduce/mapper", __FILE__)
+require File.expand_path("../map_reduce/reducer", __FILE__)

data/lib/map_reduce/exceptions.rb

@@ -0,0 +1,4 @@
+module MapReduce::Exceptions
+  class BlankKey < StandardError; end
+  class BlankMasters < StandardError; end
+end

data/lib/map_reduce/map_log.rb

@@ -0,0 +1,45 @@
+module MapReduce
+  class MapLog
+    MAX_BUFFER_SIZE = 2 ** 20
+
+    def initialize(log_folder, task)
+      @log_folder = log_folder
+      @task = task
+      @log = ""
+      @log_size = 0
+    end
+
+    def <<(str)
+      @log_size += str.size
+      @log << str << "\n"
+      flush  if @log_size >= MAX_BUFFER_SIZE
+    end
+
+    def flush
+      unless @log.empty?
+        log_file << @log
+        log_file.flush
+      end
+    end
+
+    def reset
+      flush
+      if @log_file
+        fn = File.path(@log_file)
+        @log_file.close
+        @log_file = nil
+        fn
+      end
+    end
+
+    def log_file
+      @log_file ||= begin
+        begin
+          fn = File.join(@log_folder, "map_#{@task}_#{Time.now.to_i}_#{rand(1000)}.log")
+        end while File.exist?(fn)
+        FileUtils.mkdir_p(@log_folder)
+        File.open(fn, "a")
+      end
+    end
+  end
+end

data/lib/map_reduce/mapper.rb

@@ -0,0 +1,72 @@
+module MapReduce
+  class Mapper
+    def initialize(opts = {})
+      @masters         = opts[:masters] || [::MapReduce::DEFAULT_SOCKET]
+      @connection_type = opts[:type]    || :em
+      @task_name       = opts[:task]
+    end
+
+    def emit(key, value, &blk)
+      raise MapReduce::Exceptions::BlankKey, "Key can't be nil"  if key.nil?
+
+      sock = pick_master(key)
+      sock.send_request(["map", key, value, @task_name], &blk)
+    end
+    alias :map :emit
+
+    def wait_for_all(&blk)
+      finished = Hash[socket.map{ |s| [s, false] }]
+      sockets.each do |sock|
+        sock.send_request(["map_finished", @task_name]) do |message|
+          finished[sock] = message[0] == "ok"
+          if finished.all?{ |k,v| v }
+            if block_given?
+              blk.call
+            else
+              return
+            end
+          else
+            after(1) do
+              wait_for_all(&blk)
+            end
+          end
+        end
+      end
+    end
+
+    private
+
+    def after(sec)
+      klass = if @connection_type == :sync
+        EM::Synchrony
+      else
+        EM
+      end
+
+      klass.add_timer(sec) do
+        yield
+      end
+    end
+
+    def pick_master(key)
+      num = Digest::MD5.hexdigest(key.to_s).to_i(16) % sockets.size
+      sockets[num]
+    end
+
+    def sockets
+      @sockets ||= begin
+        klass = if @connection_type == :sync
+          EM::Protocols::Zmq2::ReqFiber
+        else
+          EM::Protocols::Zmq2::ReqCb
+        end
+
+        @masters.map do |sock|
+          s = klass.new
+          s.connect(sock)
+          s
+        end
+      end
+    end
+  end
+end

data/lib/map_reduce/master.rb

@@ -2,179 +2,121 @@ require File.expand_path("../socket/master", __FILE__)
 
 module MapReduce
   class Master
-    # How often data will be flushed to disk
-    FLUSH_TIMEOUT = 1
-    # How many lines should be parsed by one iteration of grouping
-    GROUP_LINES = 100
-    # How many seconds should we sleep if grouping is going faster then reducing
-    GROUP_TIMEOUT = 1
-    # How many keys should be stored before timeout happend
-    GROUP_MAX = 10_000
-
-    # Valid options:
-    #   * socket - socket address to bind
-    #       default is 'ipc:///dev/shm/master.sock'
-    #   * log_folder - folder to store recieved MAP data
-    #       default is '/tmp/mapreduce/'
-    #   * workers - count of workers that will emit data.
-    #       default is :auto, 
-    #       but in small jobs it is better to define in explicitly,
-    #       because if one worker will stop before others start
-    #       master will decide that map job is done and will start reducing
-    #   * delimiter - master log stores data like "key{delimiter}values"
-    #       so to prevent collisions you can specify your own uniq delimiter
-    #       default is a pipe "|"
-    #
     def initialize(opts = {})
-      # Socket addr to bind
-      @socket_addr = opts[:socket] || ::MapReduce::DEFAULT_SOCKET
-      # Folder to write logs
-      @log_folder = opts[:log_folder] || "/tmp/mapreduce/"
-      # How many MapReduce workers will emit data
-      @workers = opts[:workers] || 1
-      # Delimiter to store key/value pairs in log
-      @delimiter = opts[:delimiter] || "|"
-
-      @log = []
-      @data = []
-      @workers_envelopes = {}
-      @log_filename = File.join(@log_folder, "master-#{Process.pid}.log")
-      @sorted_log_filename = File.join(@log_folder, "master-#{Process.pid}_sorted.log")
-
-      FileUtils.mkdir_p(@log_folder)
-      FileUtils.touch(@log_filename)
-    end
-
-    # Start Eventloop
-    #
+      @socket_addr = opts[:socket]     || ::MapReduce::DEFAULT_SOCKET
+      @log_folder  = opts[:log_folder] || "/tmp/map_reduce"
+      @delimiter   = opts[:delimiter]  || "\t"
+
+      @tasks = {}
+    end
+
     def run
       EM.run do
-        # Init socket
-        master_socket
-
-        # Init flushing timer
-        flush
+        socket
       end
     end
 
-    # Stop Eventloop
-    #
     def stop
       EM.stop
     end
 
-    # Store data in log array till flush
-    #
-    def map(key, message)
-      @log << "#{key}#{@delimiter}#{message}"
-    end
-
-    # Send data back to worker.
-    # Last item in data is last unfinished session,
-    #   so till the end of file reading we don't send it
-    #
-    def reduce(envelope)
-      if @data.size >= 2 
-        data = @data.shift
-        data = data.flatten
-        master_socket.send_reply(data, envelope)
-      elsif @reduce_stop
-        data = @data.shift
-        data = data.flatten  if data
-        master_socket.send_reply(data, envelope)
+    def after_map(&blk)
+      @after_map = blk
+    end
+
+    def after_reduce(&blk)
+      @after_reduce = blk
+    end
+
+    def recieve_msg(message, envelope)
+      mtype = case message[0]
+      when "map"
+        store_map(message, envelope)
+      when "map_finished"
+        all_finished?(message, envelope)
+      when "reduce"
+        send_reduce(message, envelope)
       else
-        EM.add_timer(1) do
-          reduce(envelope)
-        end
+        MapReduce.logger.error("Wrong message type: #{mtype}")
       end
     end
 
-    # Openning log file for read/write
-    #
-    def log_file
-      @log_file ||= begin
-        File.open(@log_filename, "w+")
-      end
+    private
+
+    def store_map(message, envelope)
+      status, key, value, task = message
+      map_log(task) << "#{key}#{@delimiter}#{value}"
+      ok(envelope)
+      register(task, envelope, "mapper", status)
+
+      @after_map.call(key, value, task)  if @after_map
     end
 
-    # Openning sorted log for reading
-    #
-    def sorted_log_file
-      @sorted_log_file ||= begin
-        File.open(@sorted_log_filename, "r")
+    def send_reduce(message, envelope)
+      status, task = message
+
+      data = if @tasks.fetch(task, {}).fetch("reducer", {}).fetch(envelope[0], nil) == "reduce"
+        reduce_log(task).get_data
+      else
+        reduce_log(task, true).get_data
       end
-    end
 
-    # Flushing data to disk once per FLUSH_TIMEOUT seconds
-    #
-    def flush
-      if @log.any?
-        log_file << @log*"\n" << "\n"
-        log_file.flush
-        @log.clear
+      reply(data, envelope)
+
+      if data
+        register(task, envelope, "reducer", status)
+      else
+        register(task, envelope, "reducer", "reduce_finished")
       end
 
-      EM.add_timer(FLUSH_TIMEOUT) do
-        flush
+      @after_reduce.call(data[0], data[1], task)  if data && @after_reduce
+    end
+
+    def all_finished?(message, envelope)
+      status, task = message
+      register(task, envelope, "mapper", status)
+      if @tasks[task]["mapper"].all?{ |k,v| v == status }
+        ok(envelope)
+      else
+        no(envelope)
       end
     end
 
-    # Sorting log.
-    # Linux sort is the fastest way to sort big file.
-    # Deleting original log after sort.
-    #
-    def sort
-      `sort #{@log_filename} -o #{@sorted_log_filename}`
-      FileUtils.rm(@log_filename)
-      @log_file = nil
+    def map_log(task)
+      @map_log ||= {}
+      @map_log[task] ||= MapReduce::MapLog.new(@log_folder, task)
     end
 
-    # Start reducing part.
-    # First, flushing rest of log to disk.
-    # Then sort data.
-    # Then start to read/group data
-    #
-    def reduce!
-      flush
-      sort
+    def reduce_log(task, force = false)
+      @reduce_log ||= {}
+      log = @reduce_log[task] ||= MapReduce::ReduceLog.new(map_log(task), @delimiter)
+      @reduce_log[task].force  if force
+      log
+    end
 
-      iter = sorted_log_file.each_line
-      group iter
+    def ok(envelope)
+      reply(["ok"], envelope)
     end
 
-    # Reading sorted data and grouping by key.
-    # If queue (@data) is growing faster then workers grad data we pause reading file.
-    #
-    def group(iter)
-      if @data.size >= GROUP_MAX
-        EM.add_timer(GROUP_TIMEOUT){ group(iter) }
-      else
-        GROUP_LINES.times do
-          line = iter.next.chomp
-          key, msg = line.split(@delimiter)
-
-          last = @data.last
-          if last && last[0] == key
-            last[1] << msg
-          else
-            @data << [key, [msg]]
-          end
-        end
-
-        EM.next_tick{ group(iter) }
-      end
-    rescue StopIteration => e
-      FileUtils.rm(@sorted_log_filename)
-      @sorted_log_file = nil
-      @reduce_stop = true
-    end
-
-    # Initializing and binding socket
-    #
-    def master_socket
-      @master_socket ||= begin
-        sock = MapReduce::Socket::Master.new self, @workers
-        sock.bind @socket_addr
+    def np(envelope)
+      reply(["not ok"], envelope)
+    end
+
+    def reply(resp, envelope)
+      socket.send_reply(resp, envelope)
+    end
+
+    def register(task, envelope, type, status)
+      @tasks[task] ||= {}
+      @tasks[task][type] ||= {}
+      @tasks[task][type][envelope[0]] = status
+    end
+
+    def socket
+      @socket ||= begin
+        master = self
+        sock = MapReduce::Socket::Master.new(self)
+        sock.bind(@socket_addr)
         sock
       end
     end