moped 1.0.0.alpha → 1.0.0.beta

data/README.md

@@ -15,6 +15,21 @@ session[:artists].find(name: "Syd Vicious").
   )
 ```
 
+## Features
+
+* Automated replica set node discovery and failover.
+* No C or Java extensions
+* No external dependencies
+* Simple, stable, public API.
+
+### Unsupported Features
+
+* GridFS
+* Map/Reduce
+
+These features are possible to implement, but outside the scope of Moped's
+goals. Consider them perfect opportunities to write a companion gem!
+
 # Project Breakdown
 
 Moped is composed of three parts: an implementation of the [BSON
@@ -43,6 +58,31 @@ id.generation_time # => 2012-04-11 13:14:29 UTC
 id == Moped::BSON::ObjectId.from_string(id.to_s) # => true
 ```
 
+<table><tbody>
+
+<tr><th>new</th>
+<td>Creates a new object id.</td></tr>
+
+<tr><th>from_string</th>
+<td>Creates a new object id from an object id string.
+<br>
+<code>Moped::BSON::ObjectId.from_string("4f8d8c66e5a4e45396000009")</code>
+</td></tr>
+
+<tr><th>from_time</th>
+<td>Creates a new object id from a time.
+<br>
+<code>Moped::BSON::ObjectId.from_time(Time.new)</code>
+</td></tr>
+
+<tr><th>legal?</th>
+<td>Validates an object id string.
+<br>
+<code>Moped::BSON::ObjectId.legal?("4f8d8c66e5a4e45396000009")</code>
+</td></tr>
+
+</tbody></table>
+
 ### Moped::BSON::Code
 
 The `Code` class is used for working with javascript on the server.
@@ -299,6 +339,126 @@ scope.one # nil
 
 </tbody></table>
 
+# Exceptions
+
+Here's a list of the exceptions generated by Moped.
+
+<table><tbody>
+
+<tr><th>Moped::Errors::ConnectionFailure</th>
+<td>Raised when a node cannot be reached or a connection is lost.
+<br>
+<strong>Note:</strong> this exception is only raised if Moped could not
+reconnect, so you shouldn't attempt to rescue this.</td></tr>
+
+<tr><th>Moped::Errors::OperationFailure</th>
+<td>Raised when a command fails or is invalid, such as when an insert fails in
+safe mode.</td></tr>
+
+<tr><th>Moped::Errors::QueryFailure</th>
+<td>Raised when an invalid query was sent to the database.</td></tr>
+
+<tr><th>Moped::Errors::AuthenticationFailure</th>
+<td>Raised when invalid credentials were passed to `session.login`.</td></tr>
+
+<tr><th>Moped::Errors::SocketError</th>
+<td>Not a real exception, but a module used to tag unhandled exceptions inside
+of a node's networking code. Allows you to `rescue Moped::SocketError` which
+preserving the real exception.</td></tr>
+
+</tbody></table>
+
+Other exceptions are possible while running commands, such as IO Errors around
+failed connections. Moped tries to be smart about managing its connections,
+such as checking if they're dead before executing a command; but those checks
+aren't foolproof, and Moped is conservative about handling unexpected errors on
+its connections. Namely, Moped will *not* retry a command if an unexpected
+exception is raised. Why? Because it's impossible to know whether the command
+was actually received by the remote Mongo instance, and without domain
+knowledge it cannot be safely retried.
+
+Take for example this case:
+
+```ruby
+session.with(safe: true)["users"].insert(name: "John")
+```
+
+It's entirely possible that the insert command will be sent to Mongo, but the
+connection gets closed before we read the result for `getLastError`. In this
+case, there's no way to know whether the insert was actually successful!
+
+If, however, you want to gracefully handle this in your own application, you
+could do something like:
+
+```ruby
+document = { _id: Moped::BSON::ObjectId.new, name: "John" }
+
+begin
+  session["users"].insert(document)
+rescue Moped::Errors::SocketError
+  session["users"].find(_id: document[:_id]).upsert(document)
+end
+```
+
+# Replica Sets
+
+Moped has full support for replica sets including automatic failover and node
+discovery.
+
+## Automatic Failover
+
+Moped will automatically retry lost connections and attempt to detect dead
+connections before sending an operation. Note, that it will *not* retry
+individual operations! For example, these cases will work and not raise any
+exceptions:
+
+```ruby
+session[:users].insert(name: "John")
+# kill primary node and promote secondary
+session[:users].insert(name: "John")
+session[:users].find.count # => 2.0
+
+# primary node drops our connection
+session[:users].insert(name: "John")
+```
+
+However, you'll get an operation error in a case like:
+
+```ruby
+# primary node goes down while reading the reply
+session.with(safe: true)[:users].insert(name: "John")
+```
+
+And you'll get a connection error in a case like:
+
+```ruby
+# primary node goes down, no new primary available yet
+session[:users].insert(name: "John")
+```
+
+If your session is running with eventual consistency, read operations will
+never raise connection errors as long as any secondary or primary node is
+running. The only case where you'll see a connection failure is if a node goes
+down while attempting to retrieve more results from a cursor, because cursors
+are tied to individual nodes.
+
+When two attempts to connect to a node fail, it will be marked as down. This
+removes it from the list of available nodes for `:down_interval` (default 30
+seconds). Note that the `:down_interval` only applies to normal operations;
+that is, if you ask for a primary node and none is available, all nodes will be
+retried. Likewise, if you ask for a secondary node, and no secondary or primary
+node is available, all nodes will be retreied.
+
+## Node Discovery
+
+The addresses you pass into your session are used as seeds for setting up
+replica set connections. After connection, each seed node will return a list of
+other known nodes which will be added to the set.
+
+This information is cached according to the `:refresh_interval` option (default:
+5 minutes). That means, e.g., that if you add a new node to your replica set,
+it should be represented in Moped within 5 minutes.
+
 # Thread-Safety
 
 Moped is thread-safe -- depending on your definition of thread-safe. For Moped,

data/lib/moped.rb

@@ -6,14 +6,16 @@ require "forwardable"
 require "moped/bson"
 require "moped/cluster"
 require "moped/collection"
+require "moped/connection"
 require "moped/cursor"
 require "moped/database"
 require "moped/errors"
 require "moped/indexes"
 require "moped/logging"
+require "moped/node"
 require "moped/protocol"
 require "moped/query"
-require "moped/server"
 require "moped/session"
-require "moped/socket"
+require "moped/session/context"
+require "moped/threaded"
 require "moped/version"

data/lib/moped/bson/object_id.rb

@@ -8,30 +8,33 @@ module Moped
       # Formatting string for outputting an ObjectId.
       @@string_format = ("%02x" * 12).freeze
 
-      attr_reader :data
-
       class << self
         def from_string(string)
+          raise Errors::InvalidObjectId.new(string) unless legal?(string)
           data = []
           12.times { |i| data << string[i*2, 2].to_i(16) }
-          new data
+          from_data data.pack("C12")
+        end
+
+        def from_time(time)
+          from_data @@generator.generate(time.to_i)
         end
 
         def legal?(str)
-          !!str.match(/^[0-9a-f]{24}$/i)
+          !!str.match(/\A\h{24}\Z/i)
         end
-      end
 
-      def initialize(data = nil, time = nil)
-        if data
-          @data = data
-        elsif time
-          @data = @@generator.generate(time.to_i)
-        else
-          @data = @@generator.next
+        def from_data(data)
+          id = allocate
+          id.instance_variable_set :@data, data
+          id
         end
       end
 
+      def data
+        @data ||= @@generator.next
+      end
+
       def ==(other)
         BSON::ObjectId === other && data == other.data
       end
@@ -42,28 +45,27 @@ module Moped
       end
 
       def to_s
-        @@string_format % data
+        @@string_format % data.unpack("C12")
       end
 
       # Return the UTC time at which this ObjectId was generated. This may
       # be used instread of a created_at timestamp since this information
       # is always encoded in the object id.
       def generation_time
-        Time.at(@data.pack("C4").unpack("N")[0]).utc
+        Time.at(data.unpack("N")[0]).utc
       end
 
       class << self
         def __bson_load__(io)
-          new io.read(12).unpack('C*')
+          from_data(io.read(12))
         end
-
       end
 
       def __bson_dump__(io, key)
         io << Types::OBJECT_ID
         io << key
         io << NULL_BYTE
-        io << data.pack('C12')
+        io << data
       end
 
       # @api private
@@ -71,49 +73,28 @@ module Moped
         def initialize
           # Generate and cache 3 bytes of identifying information from the current
           # machine.
-          @machine_id = Digest::MD5.digest(Socket.gethostname).unpack("C3")
+          @machine_id = Digest::MD5.digest(Socket.gethostname).unpack("N")[0]
 
           @mutex = Mutex.new
-          @last_timestamp = nil
           @counter = 0
         end
 
-        # Return object id data based on the current time, incrementing a
-        # counter for object ids generated in the same second.
+        # Return object id data based on the current time, incrementing the
+        # object id counter.
         def next
-          now = Time.new.to_i
-
-          counter = @mutex.synchronize do
-            last_timestamp, @last_timestamp = @last_timestamp, now
-
-            if last_timestamp == now
-              @counter += 1
-            else
-              @counter = 0
-            end
+          @mutex.lock
+          begin
+            counter = @counter = (@counter + 1) % 0xFFFFFF
+          ensure
+            @mutex.unlock rescue nil
           end
 
-          generate(now, counter)
+          generate(Time.new.to_i, counter)
         end
 
-        # Generate object id data for a given time using the provided +inc+.
-        def generate(time, inc = 0)
-          pid = Process.pid % 0xFFFF
-
-          [
-            time >> 24 & 0xFF, # 4 bytes time (network order)
-            time >> 16 & 0xFF,
-            time >> 8  & 0xFF,
-            time       & 0xFF,
-            @machine_id[0],   # 3 bytes machine
-            @machine_id[1],
-            @machine_id[2],
-            pid  >> 8  & 0xFF, # 2 bytes process id
-            pid        & 0xFF,
-            inc  >> 16 & 0xFF, # 3 bytes increment
-            inc  >> 8  & 0xFF,
-            inc        & 0xFF,
-          ]
+        # Generate object id data for a given time using the provided +counter+.
+        def generate(time, counter = 0)
+          [time, @machine_id, Process.pid, counter << 8].pack("N NX lXX NX")
         end
       end
 

data/lib/moped/cluster.rb

@@ -1,173 +1,133 @@
 module Moped
 
-  # @api private
-  #
-  # The internal class managing connections to both a single node and replica
-  # sets.
-  #
-  # @note Though the socket class itself *is* threadsafe, the cluster presently
-  #   is not. This means that in the course of normal operations sessions can be
-  #   shared across threads, but in failure modes (when a resync is required),
-  #   things can possibly go wrong.
   class Cluster
 
-    # @return [Array] the user supplied seeds
+    # @return [Array<String>] the seeds the replica set was initialized with
     attr_reader :seeds
 
-    # @return [Boolean] whether this is a direct connection
-    attr_reader :direct
-
-    # @return [Array] all available nodes
-    attr_reader :servers
-
-    # @return [Array] seeds gathered from cluster discovery
-    attr_reader :dynamic_seeds
-
-    # @param [Array] seeds an array of host:port pairs
-    # @param [Boolean] direct (false) whether to connect directly to the hosts
-    # provided or to find additional available nodes.
-    def initialize(seeds, direct = false)
-      @seeds  = seeds
-      @direct = direct
-
-      @servers = []
-      @dynamic_seeds = []
-    end
-
-    # @return [Array] available secondary nodes
-    def secondaries
-      servers.select(&:secondary?)
-    end
-
-    # @return [Array] available primary nodes
-    def primaries
-      servers.select(&:primary?)
-    end
-
-    # @return [Array] all known addresses from user supplied seeds, dynamically
-    # discovered seeds, and active servers.
-    def known_addresses
-      [].tap do |addresses|
-        addresses.concat seeds
-        addresses.concat dynamic_seeds
-        addresses.concat servers.map { |server| server.address }
-      end.uniq
-    end
-
-    def remove(server)
-      servers.delete(server)
-    end
-
-    def reconnect
-      @servers = servers.map { |server| Server.new(server.address) }
+    # @option options :down_interval number of seconds to wait before attempting
+    # to reconnect to a down node. (30)
+    #
+    # @option options :refresh_interval number of seconds to cache information
+    # about a node. (300)
+    def initialize(hosts, options)
+      @options = {
+        down_interval: 30,
+        refresh_interval: 300
+      }.merge(options)
+
+      @seeds = hosts
+      @nodes = hosts.map { |host| Node.new(host) }
     end
 
-    def sync
-      known = known_addresses.shuffle
-      seen  = {}
-
-      sync_seed = ->(seed) do
-        server = Server.new seed
-
-        unless seen[server.resolved_address]
-          seen[server.resolved_address] = true
-
-          hosts = sync_server(server)
-
-          hosts.each do |host|
-            sync_seed[host]
+    # Refreshes information for each of the nodes provided. The node list
+    # defaults to the list of all known nodes.
+    #
+    # If a node is successfully refreshed, any newly discovered peers will also
+    # be refreshed.
+    #
+    # @return [Array<Node>] the available nodes
+    def refresh(nodes_to_refresh = @nodes)
+      refreshed_nodes = []
+      seen = {}
+
+      # Set up a recursive lambda function for refreshing a node and it's peers.
+      refresh_node = ->(node) do
+        unless seen[node]
+          seen[node] = true
+
+          # Add the node to the global list of known nodes.
+          @nodes << node unless @nodes.include?(node)
+
+          begin
+            node.refresh
+
+            # This node is good, so add it to the list of nodes to return.
+            refreshed_nodes << node unless refreshed_nodes.include?(node)
+
+            # Now refresh any newly discovered peer nodes.
+            (node.peers - @nodes).each &refresh_node
+          rescue Errors::ConnectionFailure
+            # We couldn't connect to the node, so don't do anything with it.
           end
         end
       end
 
-      known.each do |seed|
-        sync_seed[seed]
-      end
+      nodes_to_refresh.each &refresh_node
 
-      unless servers.empty?
-        @dynamic_seeds = servers.map(&:address)
-      end
-
-      true
+      refreshed_nodes.to_a
     end
 
-    def sync_server(server)
-      [].tap do |hosts|
-        socket = server.socket
-
-        if socket.connect
-          info = socket.simple_query Protocol::Command.new(:admin, ismaster: 1)
-
-          if info["ismaster"]
-            server.primary = true
-          end
-
-          if info["secondary"]
-            server.secondary = true
-          end
+    # Returns the list of available nodes, refreshing 1) any nodes which were
+    # down and ready to be checked again and 2) any nodes whose information is
+    # out of date.
+    #
+    # @return [Array<Node>] the list of available nodes.
+    def nodes
+      # Find the nodes that were down but are ready to be refreshed, or those
+      # with stale connection information.
+      needs_refresh, available = @nodes.partition do |node|
+        (node.down? && node.down_at < (Time.new - @options[:down_interval])) ||
+          node.needs_refresh?(Time.new - @options[:refresh_interval])
+      end
 
-          if info["primary"]
-            hosts.push info["primary"]
-          end
+      # Refresh those nodes.
+      available.concat refresh(needs_refresh)
 
-          if info["hosts"]
-            hosts.concat info["hosts"]
-          end
+      # Now return all the nodes that are available.
+      available.reject &:down?
+    end
 
-          if info["passives"]
-            hosts.concat info["passives"]
+    # Yields the replica set's primary node to the provided block. This method
+    # will retry the block in case of connection errors or replica set
+    # reconfiguration.
+    #
+    # @raises ConnectionFailure when no primary node can be found
+    def with_primary(retry_on_failure = true, &block)
+      if node = nodes.find(&:primary?)
+        begin
+          node.ensure_primary do
+            return yield node.apply_auth(auth)
           end
-
-          merge(server)
-
+        rescue Errors::ConnectionFailure, Errors::ReplicaSetReconfigured
+          # Fall through to the code below if our connection was dropped or the
+          # node is no longer the primary.
         end
-      end.uniq
-    end
-
-    def merge(server)
-      previous = servers.find { |other| other == server }
-      primary = server.primary?
-      secondary = server.secondary?
+      end
 
-      if previous
-        previous.merge(server)
+      if retry_on_failure
+        # We couldn't find a primary node, so refresh the list and try again.
+        refresh
+        with_primary(false, &block)
       else
-        servers << server
+        raise Errors::ConnectionFailure, "Could not connect to a primary node for replica set #{inspect}"
       end
     end
 
-    # @param [:read, :write] mode the type of socket to return
-    # @return [Socket] a socket valid for +mode+ operations
-    def socket_for(mode)
-      sync unless primaries.any? || (secondaries.any? && mode == :read)
-
-      server = nil
-      while primaries.any? || (secondaries.any? && mode == :read)
-        if mode == :write || secondaries.empty?
-          server = primaries.sample
-        else
-          server = secondaries.sample
-        end
-
-        if server
-          socket = server.socket
-          socket.connect unless socket.connection
-
-          if socket.alive?
-            break server
-          else
-            remove server
-          end
+    # Yields a secondary node if available, otherwise the primary node. This
+    # method will retry the block in case of connection errors.
+    #
+    # @raises ConnectionError when no secondary or primary node can be found
+    def with_secondary(retry_on_failure = true, &block)
+      available_nodes = nodes.shuffle!.partition(&:secondary?).flatten
+
+      while node = available_nodes.shift
+        begin
+          return yield node.apply_auth(auth)
+        rescue Errors::ConnectionFailure
+          # That node's no good, so let's try the next one.
+          next
         end
       end
 
-      unless server
-        raise Errors::ConnectionFailure.new("Could not connect to any primary or secondary servers")
+      if retry_on_failure
+        # We couldn't find a secondary or primary node, so refresh the list and
+        # try again.
+        refresh
+        with_secondary(false, &block)
+      else
+        raise Errors::ConnectionFailure, "Could not connect to any secondary or primary nodes for replica set #{inspect}"
       end
-
-      socket = server.socket
-      socket.apply_auth auth
-      socket
     end
 
     # @return [Hash] the cached authentication credentials for this cluster.
@@ -175,19 +135,11 @@ module Moped
       @auth ||= {}
     end
 
-    # Log in to +database+ with +username+ and +password+. Does not perform the
-    # actual log in, but saves the credentials for later authentication on a
-    # socket.
-    def login(database, username, password)
-      auth[database.to_s] = [username, password]
-    end
+    private
 
-    # Log out of +database+. Does not perform the actual log out, but will log
-    # out when the socket is used next.
-    def logout(database)
-      auth.delete(database.to_s)
+    def initialize_copy(_)
+      @nodes = @nodes.map &:dup
     end
 
   end
-
 end