mongo 1.4.0 → 1.5.0.rc0

data/docs/HISTORY.md

@@ -1,5 +1,20 @@
 # MongoDB Ruby Driver History
 
+### 1.5.0.rc0
+2011-11-18
+
+Fix bugs associated with replica set refresh.
+
+### 1.4.1
+2011-10-17
+
+If you're using 1.4.0, this is a necessary upgrade.
+
+* Simplified replica set refresh.
+* Fix bugs associated with replica set refresh.
+* Make cursor smart enough to continue functioning
+even if a refresh is triggered.
+
 ### 1.4.0
 2011-9-19
 

data/docs/REPLICA_SETS.md

@@ -28,31 +28,43 @@ Imagine that either the master node or one of the read nodes goes offline. How w
 
 If any read operation fails, the driver will raise a *ConnectionFailure* exception. It then becomes the client's responsibility to decide how to handle this.
 
-If the client decides to retry, it's not guaranteed that another member of the replica set will have been promoted to master right away, so it's still possible that the driver will raise another *ConnectionFailure*. However, once a member has been promoted to master, typically within a few seconds, subsequent operations will succeed.
+If the client decides to retry, it's not guaranteed that another member of the replica set will have been promoted to master right away, so it's still possible that the driver will raise another *ConnectionFailure*. However, once a member has been promoted to master, typically within a few seconds, subsequent operations will succeed. *Note that this does not prevent
+exception in the event of a primary failover.*
 
 The driver will essentially cycle through all known seed addresses until a node identifies itself as master.
 
 ### Refresh mode
 
 You can now specify a refresh mode and refresh interval for a replica set connection. This will help to ensure that
-changes to a replica set's configuration are quickly reflected on the driver side. Refresh mode is
-enabled in synchronous mode by default. Here's how to specify this explicitly:
+changes to a replica set's configuration are quickly reflected on the driver side. In particular, if you change
+the state of any secondary node, the automated refresh will ensure that this state is recorded on the client side.
+If you add a secondary that responds to pings much faster than the existing nodes, then the new secondary will
+be used for reads.
 
-    @connection = ReplSetConnection.new(['n1.mydb.net', 27017], :refresh_mode => :sync)
+Refresh mode is disabled by default.
 
-If you want to refresh to happen via a background thread, use the `:async` mode:
+However, if you expect to make live changes to your secondaries, and you want this to be reflected without
+having to manually restart your app server, then you should enable it. You can enable this mode
+synchronously, which will refresh the replica set data in a synchronous fashion (which may
+ocassionally slow down your queries):
 
-    @connection = ReplSetConnection.new(['n1.mydb.net', 27017], :refresh_mode => :async)
+    @connection = ReplSetConnection.new(['n1.mydb.net', 27017], :refresh_mode => :sync)
 
 If you want to change the default refresh interval of 90 seconds, you can do so like this:
 
-    @connection = ReplSetConnection.new(['n1.mydb.net', 27017], :refresh_mode => :async,
+    @connection = ReplSetConnection.new(['n1.mydb.net', 27017], :refresh_mode => :sync,
         :refresh_interval => 60)
 
+Do not set this value to anything lower than 30, or you may start to experience performance issues.
+
 You can also disable refresh mode altogether:
 
     @connection = ReplSetConnection.new(['n1.mydb.net', 27017], :refresh_mode => false)
 
+And you can call `refresh` manually on any replica set connection:
+
+    @connection.refresh
+
 ### Recovery
 
 Driver users may wish to wrap their database calls with failure recovery code. Here's one possibility, which will attempt to connection

data/lib/mongo.rb

@@ -65,6 +65,7 @@ require 'mongo/util/ssl_socket'
 require 'mongo/util/uri_parser'
 
 require 'mongo/collection'
+require 'mongo/networking'
 require 'mongo/connection'
 require 'mongo/repl_set_connection'
 require 'mongo/cursor'

data/lib/mongo/collection.rb

@@ -217,7 +217,7 @@ module Mongo
       max_scan   = opts.delete(:max_scan)
       return_key = opts.delete(:return_key)
       transformer = opts.delete(:transformer)
-      show_disk_loc = opts.delete(:max_scan)
+      show_disk_loc = opts.delete(:show_disk_loc)
       read          = opts.delete(:read) || @read_preference
 
       if timeout == false && !block_given?

data/lib/mongo/connection.rb

@@ -19,26 +19,26 @@
 require 'set'
 require 'socket'
 require 'thread'
-
 module Mongo
 
-  # Instantiates and manages connections to MongoDB.
+  # Instantiates and manages self.connections to MongoDB.
   class Connection
     include Mongo::Logging
+    include Mongo::Networking
 
     TCPSocket = ::TCPSocket
     Mutex = ::Mutex
     ConditionVariable = ::ConditionVariable
 
+    Thread.abort_on_exception = true
+
     DEFAULT_PORT = 27017
-    STANDARD_HEADER_SIZE = 16
-    RESPONSE_HEADER_SIZE = 20
 
-    attr_reader :logger, :size, :auths, :primary, :safe, :host_to_try,
-      :pool_size, :connect_timeout, :primary_pool, :socket_class
+    mongo_thread_local_accessor :connections
 
-    # Counter for generating unique request ids.
-    @@current_request_id = 0
+    attr_reader :logger, :size, :auths, :primary, :safe, :host_to_try,
+      :pool_size, :connect_timeout, :pool_timeout,
+      :primary_pool, :socket_class
 
     # Create a connection to single MongoDB instance.
     #
@@ -63,9 +63,9 @@ module Mongo
     #   to a single, slave node.
     # @option opts [Logger, #debug] :logger (nil) A Logger instance for debugging driver ops. Note that
     #   logging negatively impacts performance; therefore, it should not be used for high-performance apps.
-    # @option opts [Integer] :pool_size (1) The maximum number of socket connections allowed per
+    # @option opts [Integer] :pool_size (1) The maximum number of socket self.connections allowed per
     #   connection pool. Note: this setting is relevant only for multi-threaded applications.
-    # @option opts [Float] :pool_timeout (5.0) When all of the connections a pool are checked out,
+    # @option opts [Float] :pool_timeout (5.0) When all of the self.connections a pool are checked out,
     #   this is the number of seconds to wait for a new connection to be released before throwing an exception.
     #   Note: this setting is relevant only for multi-threaded applications (which in Ruby are rare).
     # @option opts [Float] :op_timeout (nil) The number of seconds to wait for a read operation to time out.
@@ -80,7 +80,7 @@ module Mongo
     # @example localhost, 27017
     #   Connection.new("localhost")
     #
-    # @example localhost, 3000, max 5 connections, with max 5 seconds of wait time.
+    # @example localhost, 3000, max 5 self.connections, with max 5 seconds of wait time.
     #   Connection.new("localhost", 3000, :pool_size => 5, :timeout => 5)
     #
     # @example localhost, 3000, where this node may be a slave
@@ -91,7 +91,7 @@ module Mongo
     # @raise [ReplicaSetConnectionError] This is raised if a replica set name is specified and the
     #   driver fails to connect to a replica set with that name.
     #
-    # @core connections
+    # @core self.connections
     def initialize(host=nil, port=nil, opts={})
       @host_to_try = format_pair(host, port)
 
@@ -379,125 +379,12 @@ module Mongo
       @slave_ok
     end
 
-    # Send a message to MongoDB, adding the necessary headers.
-    #
-    # @param [Integer] operation a MongoDB opcode.
-    # @param [BSON::ByteBuffer] message a message to send to the database.
-    #
-    # @option opts [Symbol] :connection (:writer) The connection to which
-    #   this message should be sent. Valid options are :writer and :reader.
-    #
-    # @return [Integer] number of bytes sent
-    def send_message(operation, message, opts={})
-      if opts.is_a?(String)
-        warn "Connection#send_message no longer takes a string log message. " +
-          "Logging is now handled within the Collection and Cursor classes."
-        opts = {}
-      end
-
-      connection = opts.fetch(:connection, :writer)
-
-      begin
-        add_message_headers(message, operation)
-        packed_message = message.to_s
-
-        if connection == :writer
-          socket = checkout_writer
-        else
-          socket = checkout_reader
-        end
-
-        send_message_on_socket(packed_message, socket)
-      ensure
-        checkin(socket)
-      end
-    end
-
-    # Sends a message to the database, waits for a response, and raises
-    # an exception if the operation has failed.
-    #
-    # @param [Integer] operation a MongoDB opcode.
-    # @param [BSON::ByteBuffer] message a message to send to the database.
-    # @param [String] db_name the name of the database. used on call to get_last_error.
-    # @param [Hash] last_error_params parameters to be sent to getLastError. See DB#error for
-    #   available options.
-    #
-    # @see DB#get_last_error for valid last error params.
-    #
-    # @return [Hash] The document returned by the call to getlasterror.
-    def send_message_with_safe_check(operation, message, db_name, log_message=nil, last_error_params=false)
-      docs = num_received = cursor_id = ''
-      add_message_headers(message, operation)
-
-      last_error_message = BSON::ByteBuffer.new
-      build_last_error_message(last_error_message, db_name, last_error_params)
-      last_error_id = add_message_headers(last_error_message, Mongo::Constants::OP_QUERY)
-
-      packed_message = message.append!(last_error_message).to_s
-      begin
-        sock = checkout_writer
-        @safe_mutexes[sock].synchronize do
-          send_message_on_socket(packed_message, sock)
-          docs, num_received, cursor_id = receive(sock, last_error_id)
-        end
-      ensure
-        checkin(sock)
-      end
-
-      if num_received == 1 && (error = docs[0]['err'] || docs[0]['errmsg'])
-        close if error == "not master"
-        error = "wtimeout" if error == "timeout"
-        raise OperationFailure.new(docs[0]['code'].to_s + ': ' + error, docs[0]['code'], docs[0])
-      end
-
-      docs[0]
-    end
-
-    # Sends a message to the database and waits for the response.
-    #
-    # @param [Integer] operation a MongoDB opcode.
-    # @param [BSON::ByteBuffer] message a message to send to the database.
-    # @param [String] log_message this is currently a no-op and will be removed.
-    # @param [Socket] socket a socket to use in lieu of checking out a new one.
-    # @param [Boolean] command (false) indicate whether this is a command. If this is a command,
-    #   the message will be sent to the primary node.
-    # @param [Boolean] command (false) indicate whether the cursor should be exhausted. Set
-    #   this to true only when the OP_QUERY_EXHAUST flag is set.
-    #
-    # @return [Array]
-    #   An array whose indexes include [0] documents returned, [1] number of document received,
-    #   and [3] a cursor_id.
-    def receive_message(operation, message, log_message=nil, socket=nil, command=false, read=:primary, exhaust=false)
-      request_id = add_message_headers(message, operation)
-      packed_message = message.to_s
-      begin
-        if socket
-          sock = socket
-          should_checkin = false
-        else
-          if command
-            sock = checkout_writer
-          elsif read == :primary
-            sock = checkout_writer
-          elsif read == :secondary
-            sock = checkout_reader
-          else
-            sock = checkout_tagged(read)
-          end
-          should_checkin = true
-        end
-
-        result = ''
-        @safe_mutexes[sock].synchronize do
-          send_message_on_socket(packed_message, sock)
-          result = receive(sock, request_id, exhaust)
-        end
-      ensure
-        if should_checkin
-          checkin(sock)
-        end
-      end
-      result
+    def get_socket_from_thread_local
+      Thread.current[:socket_map] ||= {}
+      Thread.current[:socket_map][self] ||= {}
+      Thread.current[:socket_map][self][:writer] ||= checkout_writer
+      Thread.current[:socket_map][self][:reader] = 
+        Thread.current[:socket_map][self][:writer]
     end
 
     # Create a new socket and attempt to connect to master.
@@ -536,7 +423,7 @@ module Mongo
     # NOTE: Do check if this needs to be more stringent.
     # Probably not since if any node raises a connection failure, all nodes will be closed.
     def connected?
-      @primary_pool && @primary_pool.host && @primary_pool.port
+      @primary_pool && !@primary_pool.closed?
     end
 
     # Determine if the connection is active. In a normal case the *server_info* operation
@@ -564,6 +451,13 @@ module Mongo
     end
     alias :primary? :read_primary?
 
+    # The socket pool that this connection reads from.
+    #
+    # @return [Mongo::Pool]
+    def read_pool
+      @primary_pool
+    end
+
     # The value of the read preference. Because
     # this is a single-node connection, the value
     # is +:primary+, and the connection will read
@@ -604,16 +498,12 @@ module Mongo
     # Checkin a socket used for reading.
     # Note: this is overridden in ReplSetConnection.
     def checkin_reader(socket)
-      warn "Connection#checkin_writer is not deprecated and will be removed " +
-        "in driver v2.0. Use Connection#checkin instead."
       checkin(socket)
     end
 
     # Checkin a socket used for writing.
     # Note: this is overridden in ReplSetConnection.
     def checkin_writer(socket)
-      warn "Connection#checkin_writer is not deprecated and will be removed " +
-        "in driver v2.0. Use Connection#checkin instead."
       checkin(socket)
     end
 
@@ -651,7 +541,7 @@ module Mongo
         warn "The :timeout option has been deprecated " +
           "and will be removed in the 2.0 release. Use :pool_timeout instead."
       end
-      @timeout = opts[:pool_timeout] || opts[:timeout] || 5.0
+      @pool_timeout = opts[:pool_timeout] || opts[:timeout] || 5.0
 
       # Timeout on socket read operation.
       @op_timeout = opts[:op_timeout] || nil
@@ -666,10 +556,6 @@ module Mongo
       # Global safe option. This is false by default.
       @safe = opts[:safe] || false
 
-            # Create a mutex when a new key, in this case a socket,
-      # is added to the hash.
-      @safe_mutexes = Hash.new { |h, k| h[k] = Mutex.new }
-
       # Condition variable for signal and wait
       @queue = ConditionVariable.new
 
@@ -680,8 +566,7 @@ module Mongo
       @logger = opts[:logger] || nil
 
       if @logger
-        @logger.debug("MongoDB logging. Please note that logging negatively impacts performance " +
-        "and should be disabled for high-performance production apps.")
+        write_logging_startup_message
       end
 
       should_connect = opts.fetch(:connect, true)
@@ -743,214 +628,7 @@ module Mongo
     def set_primary(node)
       host, port = *node
       @primary = [host, port]
-      @primary_pool = Pool.new(self, host, port, :size => @pool_size, :timeout => @timeout)
-    end
-
-    ## Low-level connection methods.
-
-    def receive(sock, cursor_id, exhaust=false)
-      begin
-        if exhaust
-          docs = []
-          num_received = 0
-
-          while(cursor_id != 0) do
-            receive_header(sock, cursor_id, exhaust)
-            number_received, cursor_id = receive_response_header(sock)
-            new_docs, n = read_documents(number_received, sock)
-            docs += new_docs
-            num_received += n
-          end
-
-          return [docs, num_received, cursor_id]
-        else
-          receive_header(sock, cursor_id, exhaust)
-          number_received, cursor_id = receive_response_header(sock)
-          docs, num_received = read_documents(number_received, sock)
-
-          return [docs, num_received, cursor_id]
-        end
-      rescue Mongo::ConnectionFailure => ex
-        close
-        raise ex
-      end
-    end
-
-    def receive_header(sock, expected_response, exhaust=false)
-      header = receive_message_on_socket(16, sock)
-      size, request_id, response_to = header.unpack('VVV')
-      if !exhaust && expected_response != response_to
-        raise Mongo::ConnectionFailure, "Expected response #{expected_response} but got #{response_to}"
-      end
-
-      unless header.size == STANDARD_HEADER_SIZE
-        raise "Short read for DB response header: " +
-          "expected #{STANDARD_HEADER_SIZE} bytes, saw #{header.size}"
-      end
-      nil
-    end
-
-    def receive_response_header(sock)
-      header_buf = receive_message_on_socket(RESPONSE_HEADER_SIZE, sock)
-      if header_buf.length != RESPONSE_HEADER_SIZE
-        raise "Short read for DB response header; " +
-          "expected #{RESPONSE_HEADER_SIZE} bytes, saw #{header_buf.length}"
-      end
-      flags, cursor_id_a, cursor_id_b, starting_from, number_remaining = header_buf.unpack('VVVVV')
-      check_response_flags(flags)
-      cursor_id = (cursor_id_b << 32) + cursor_id_a
-      [number_remaining, cursor_id]
-    end
-
-    def check_response_flags(flags)
-      if flags & Mongo::Constants::REPLY_CURSOR_NOT_FOUND != 0
-        raise Mongo::OperationFailure, "Query response returned CURSOR_NOT_FOUND. " +
-          "Either an invalid cursor was specified, or the cursor may have timed out on the server."
-      elsif flags & Mongo::Constants::REPLY_QUERY_FAILURE != 0
-        # Getting odd failures when a exception is raised here.
-      end
-    end
-
-    def read_documents(number_received, sock)
-      docs = []
-      number_remaining = number_received
-      while number_remaining > 0 do
-        buf = receive_message_on_socket(4, sock)
-        size = buf.unpack('V')[0]
-        buf << receive_message_on_socket(size - 4, sock)
-        number_remaining -= 1
-        docs << BSON::BSON_CODER.deserialize(buf)
-      end
-      [docs, number_received]
-    end
-
-    # Constructs a getlasterror message. This method is used exclusively by
-    # Connection#send_message_with_safe_check.
-    #
-    # Because it modifies message by reference, we don't need to return it.
-    def build_last_error_message(message, db_name, opts)
-      message.put_int(0)
-      BSON::BSON_RUBY.serialize_cstr(message, "#{db_name}.$cmd")
-      message.put_int(0)
-      message.put_int(-1)
-      cmd = BSON::OrderedHash.new
-      cmd[:getlasterror] = 1
-      if opts.is_a?(Hash)
-        opts.assert_valid_keys(:w, :wtimeout, :fsync)
-        cmd.merge!(opts)
-      end
-      message.put_binary(BSON::BSON_CODER.serialize(cmd, false).to_s)
-      nil
-    end
-
-    # Prepares a message for transmission to MongoDB by
-    # constructing a valid message header.
-    #
-    # Note: this method modifies message by reference.
-    #
-    # @return [Integer] the request id used in the header
-    def add_message_headers(message, operation)
-      headers = [
-        # Message size.
-        16 + message.size,
-
-        # Unique request id.
-        request_id = get_request_id,
-
-        # Response id.
-        0,
-
-        # Opcode.
-        operation
-      ].pack('VVVV')
-
-      message.prepend!(headers)
-
-      request_id
-    end
-
-    # Increment and return the next available request id.
-    #
-    # return [Integer]
-    def get_request_id
-      request_id = ''
-      @id_lock.synchronize do
-        request_id = @@current_request_id += 1
-      end
-      request_id
-    end
-
-    # Low-level method for sending a message on a socket.
-    # Requires a packed message and an available socket,
-    #
-    # @return [Integer] number of bytes sent
-    def send_message_on_socket(packed_message, socket)
-      begin
-      total_bytes_sent = socket.send(packed_message, 0)
-      if total_bytes_sent != packed_message.size
-        packed_message.slice!(0, total_bytes_sent)
-        while packed_message.size > 0
-          byte_sent = socket.send(packed_message, 0)
-          total_bytes_sent += byte_sent
-          packed_message.slice!(0, byte_sent)
-        end
-      end
-      total_bytes_sent
-      rescue => ex
-        close
-        raise ConnectionFailure, "Operation failed with the following exception: #{ex}"
-      end
-    end
-
-    # Low-level method for receiving data from socket.
-    # Requires length and an available socket.
-    def receive_message_on_socket(length, socket)
-      begin
-        if @op_timeout
-          message = nil
-          Mongo::TimeoutHandler.timeout(@op_timeout, OperationTimeout) do
-            message = receive_data(length, socket)
-          end
-        else
-          message = receive_data(length, socket)
-        end
-        rescue => ex
-          close
-
-          if ex.class == OperationTimeout
-            raise OperationTimeout, "Timed out waiting on socket read."
-          else
-            raise ConnectionFailure, "Operation failed with the following exception: #{ex}"
-          end
-      end
-      message
-    end
-
-    def receive_data(length, socket)
-      message = new_binary_string
-      socket.read(length, message)
-      raise ConnectionFailure, "connection closed" unless message && message.length > 0
-      if message.length < length
-        chunk = new_binary_string
-        while message.length < length
-          socket.read(length - message.length, chunk)
-          raise ConnectionFailure, "connection closed" unless chunk.length > 0
-          message << chunk
-        end
-      end
-      message
-    end
-
-    if defined?(Encoding)
-      BINARY_ENCODING = Encoding.find("binary")
-
-      def new_binary_string
-        "".force_encoding(BINARY_ENCODING)
-      end
-    else
-      def new_binary_string
-        ""
-      end
+      @primary_pool = Pool.new(self, host, port, :size => @pool_size, :timeout => @pool_timeout)
     end
   end
 end