cql-rb 1.1.3 → 1.2.0.pre0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 77764255905873f7b923f9fca746dbef0496eb15
-  data.tar.gz: c1c39af44330064cb4d9dd2693d81384dc69f09f
+  metadata.gz: 5c27b19f205bb59c5f8fe37faa6259064d85c761
+  data.tar.gz: 74615d268c49624381e8c8088fb173834e3c0313
 SHA512:
-  metadata.gz: 14452316f397f8f2bdc37f764aaf7c070625be977c6b579c4eb881d3e9736a33eb720da296f370c7dc27de7d4fe1917f192267f1a884ae3381afd870378edbd7
-  data.tar.gz: fa4f95a86ab0ae162ccc64c9eaed8f3c99c7bc7cf217750617e3339cfba62d31badc1609f63ffc65026818777f9bc69bacdcaa21f88d32d07b730c7f5f007ef2
+  metadata.gz: 7e8bd6f3e31b0b6e201baca992267b12502e5b78b71ef64378d4f52b791a8a2036c049650c5c4e7f17a0a1b4814d942a423c802742e5173071430d59d1b811ab
+  data.tar.gz: 86539fa464d45531f30aea8198d70a726d20d6bf3247d2b25fe76d1e49a37b568c465772059af4adfd637a7371406f7410389ae824d592cd6b18c43bafcfff56

data/README.md

@@ -162,6 +162,19 @@ The default consistency level unless you've set it yourself is `:quorum`.
 
 Consistency is ignored for `USE`, `TRUNCATE`, `CREATE` and `ALTER` statements, and some (like `:any`) aren't allowed in all situations.
 
+## Compression
+
+The CQL protocol supports frame compression, which can give you a performance boost if your requests or responses are big. To enable it you can pass a compressor object when you connect.
+
+Cassandra currently supports two compression algorithms: Snappy and LZ4. Support for Snappy compression ships with cql-rb, but in order to use it you will have to install the [snappy](http://rubygems.org/gems/snappy) gem separately. Once it's installed you can enable compression like this:
+
+```ruby
+require 'cql/compression/snappy_compressor'
+
+compressor = Cql::Compression::SnappyCompressor.new
+client = Cql::Client.connect(hosts: %w[localhost], compressor: compressor)
+```
+
 # CQL3
 
 This is just a driver for the Cassandra native CQL protocol, it doesn't really know anything about CQL. You can run any CQL3 statement and the driver will return whatever Cassandra replies with.
@@ -206,6 +219,10 @@ end
 
 If your process does not fork and you still encounter deadlock errors, it might also be a bug. All IO is done is a dedicated thread, and if something happens that makes that thread shut down, Ruby will detect that the locks that the client code is waiting on can't be unlocked.
 
+## I get "Bad file descriptor"
+
+If you're using cql-rb on Windows there's an [experimental branch with Windows support](https://github.com/iconara/cql-rb/tree/windows_support). The problem is that Windows does not support non blocking reads on IO objects other than sockets, and the fix is very small. Unfortunately I have no way of properly testing things in Windows, so therefore the "experimental" label.
+
 ## I'm not getting all elements back from my list/set/map
 
 There's a known issue with collections that get too big. The protocol uses a short for the size of collections, but there is no way for Cassandra to stop you from creating a collection bigger than 65536 elements, so when you do the size field overflows with strange results. The data is there, you just can't get it back.
@@ -232,9 +249,8 @@ Prereleases will be stable, in the sense that they will have finished and proper
 
 # Known bugs & limitations
 
-* No support for compression.
-* No support for request tracing.
 * JRuby 1.6 is not officially supported, although 1.6.8 should work, if you're stuck in JRuby 1.6.8 try and see if it works for you.
+* Windows is not supported (there is experimental support in the [`windows` branch](https://github.com/iconara/cql-rb/tree/windows_support)).
 * Large results are buffered in memory until the whole response has been loaded, the protocol makes it possible to start to deliver rows to the client code as soon as the metadata is loaded, but this is not supported yet.
 * There is no cluster introspection utilities (like the `DESCRIBE` commands in `cqlsh`).
 * New features in v2 of the protocol are not supported
@@ -243,15 +259,7 @@ Also check out the [issues](https://github.com/iconara/cql-rb/issues) for open b
 
 # How to contribute
 
-Fork the repository, make your changes in a topic branch that branches off from the right place in the history (HEAD isn't necessarily always right), make your changes and finally submit a pull request.
-
-Follow the style of the existing code, make sure that existing tests pass, and that everything new has good test coverage. Put some effort into writing clear and concise commit messages, and write a good pull request description.
-
-It takes time to understand other people's code, and even more time to understand a patch, so do as much as you can to make the maintainers' work easier. Be prepared for rejection, many times a feature is already planned, or the proposed design would be in the way of other planned features, or the maintainers' just feel that it will be faster to implement the features themselves than to try to integrate your patch.
-
-Feel free to open a pull request before the feature is finished, that way you can have a conversation with the maintainers' during the development, and you can make adjustments to the design as you go along instead of having your whole feature rejected because of reasons such as those above. If you do, please make it clear that the pull request is a work in progress, or a request for comment.
-
-Always remember that the maintainers' work on this project in their free time and that they don't work for you, or for your benefit. They have no obligation to do what you think is right -- but if you're nice they might anyway.
+[See CONTRIBUTING.md](CONTRIBUTING.md)
 
 # Copyright
 

data/lib/cql.rb

@@ -9,5 +9,6 @@ require 'cql/time_uuid'
 require 'cql/byte_buffer'
 require 'cql/future'
 require 'cql/io'
+require 'cql/compression'
 require 'cql/protocol'
 require 'cql/client'

data/lib/cql/client.rb

@@ -6,12 +6,13 @@ module Cql
   # if any. `message` contains the human readable error message sent by the
   # server.
   class QueryError < CqlError
-    attr_reader :code, :cql
+    attr_reader :code, :cql, :details
 
-    def initialize(code, message, cql=nil)
+    def initialize(code, message, cql=nil, details=nil)
       super(message)
       @code = code
       @cql = cql
+      @details = details
     end
   end
 
@@ -19,6 +20,7 @@ module Cql
   TimeoutError = Class.new(CqlError)
   ClientError = Class.new(CqlError)
   AuthenticationError = Class.new(ClientError)
+  IncompleteTraceError = Class.new(ClientError)
 
   # A CQL client manages connections to one or more Cassandra nodes and you use
   # it run queries, insert and update data.
@@ -88,6 +90,11 @@ module Cql
     # @option options [Integer] :default_consistency (:quorum) The consistency
     #   to use unless otherwise specified. Consistency can also be specified on
     #   a per-request basis.
+    # @option options [Cql::Compression::Compressor] :compressor An object that
+    #   can compress and decompress frames. By specifying this option frame
+    #   compression will be enabled. If the server does not support compression
+    #   or the specific compression algorithm specified by the compressor,
+    #   compression will not be enabled and a warning will be logged.
     # @option options [Integer] :logger If you want the client to log
     #   significant events pass an object implementing the standard Ruby logger
     #   interface (e.g. quacks like `Logger` from the standard library) with
@@ -142,7 +149,23 @@ module Cql
 
       # @!method execute(cql, options_or_consistency=nil)
       #
-      # Execute a CQL statement
+      # Execute a CQL statement.
+      #
+      # @example A simple CQL query
+      #   result = client.execute("SELECT * FROM users WHERE user_name = 'sue'")
+      #   result.each do |row|
+      #     p row
+      #   end
+      #
+      # @example Specifying the consistency as a symbol
+      #   client.execute("UPDATE users SET full_name = 'Sue S. Smith' WHERE user_name = 'sue'", consistency: :one)
+      #
+      # @example Specifying the consistency and other options
+      #   client.execute("SELECT * FROM users", consistency: :all, timeout: 1.5)
+      #
+      # @example Activating tracing for a query
+      #   result = client.execute("SELECT * FROM users", tracing: true)
+      #   p result.trace_id
       #
       # @param [String] cql
       # @param [Hash] options_or_consistency Either a consistency as a symbol
@@ -153,14 +176,19 @@ module Cql
       # @option options_or_consistency [Symbol] :timeout (nil) How long to wait
       #   for a response. If this timeout expires a {Cql::TimeoutError} will
       #   be raised.
+      # @option options_or_consistency [Symbol] :trace (false) Request tracing
+      #   for this request. See {Cql::Client::QueryResult} and
+      #   {Cql::Client::VoidResult} for how to retrieve the tracing data.
       # @raise [Cql::NotConnectedError] raised when the client is not connected
       # @raise [Cql::TimeoutError] raised when a timeout was specified and no
       #   response was received within the timeout.
       # @raise [Cql::QueryError] raised when the CQL has syntax errors or for
       #   other situations when the server complains.
-      # @return [nil, Cql::Client::QueryResult] Most queries have no result and
-      #   return `nil`, but `SELECT` statements return an `Enumerable` of rows
-      #   (see {Cql::Client::QueryResult}).
+      # @return [nil, Cql::Client::QueryResult, Cql::Client::VoidResult] Some
+      #   queries have no result and return `nil`, but `SELECT` statements
+      #   return an `Enumerable` of rows (see {Cql::Client::QueryResult}), and
+      #   `INSERT` and `UPDATE` return a similar type
+      #   (see {Cql::Client::VoidResult}).
 
       # @!method prepare(cql)
       #
@@ -206,9 +234,11 @@ module Cql
       # @raise [Cql::Io::IoError] raised when there is an IO error, for example
       #   if the server suddenly closes the connection
       # @raise [Cql::QueryError] raised when there is an error on the server side
-      # @return [nil, Cql::Client::QueryResult] Most statements have no result
-      #    and return `nil`, but `SELECT` statements return an `Enumerable` of
-      #   rows (see {Cql::Client::QueryResult}).
+      # @return [nil, Cql::Client::QueryResult, Cql::Client::VoidResult] Some
+      #   queries have no result and return `nil`, but `SELECT` statements
+      #   return an `Enumerable` of rows (see {Cql::Client::QueryResult}), and
+      #   `INSERT` and `UPDATE` return a similar type
+      #   (see {Cql::Client::VoidResult}).
       def execute(*args)
       end
     end
@@ -221,6 +251,8 @@ require 'cql/client/null_logger'
 require 'cql/client/column_metadata'
 require 'cql/client/result_metadata'
 require 'cql/client/query_result'
+require 'cql/client/void_result'
+require 'cql/client/query_trace'
 require 'cql/client/execute_options_decoder'
 require 'cql/client/keyspace_changer'
 require 'cql/client/asynchronous_client'

data/lib/cql/client/asynchronous_client.rb

@@ -5,8 +5,9 @@ module Cql
     # @private
     class AsynchronousClient < Client
       def initialize(options={})
+        compressor = options[:compressor]
         @logger = options[:logger] || NullLogger.new
-        @io_reactor = options[:io_reactor] || Io::IoReactor.new(Protocol::CqlProtocolHandler)
+        @io_reactor = options[:io_reactor] || Io::IoReactor.new(protocol_handler_factory(compressor))
         @hosts = extract_hosts(options)
         @initial_keyspace = options[:keyspace]
         @lock = Mutex.new
@@ -22,7 +23,7 @@ module Cql
         connection_timeout = options[:connection_timeout] || DEFAULT_CONNECTION_TIMEOUT
         default_consistency = options[:default_consistency] || DEFAULT_CONSISTENCY
         @execute_options_decoder = ExecuteOptionsDecoder.new(default_consistency)
-        @connection_helper = ConnectionHelper.new(@io_reactor, port, credentials, connections_per_node, connection_timeout, @logger)
+        @connection_helper = ConnectionHelper.new(@io_reactor, port, credentials, connections_per_node, connection_timeout, compressor, @logger)
       end
 
       def connect
@@ -85,8 +86,8 @@ module Cql
 
       def execute(cql, options_or_consistency=nil)
         with_failure_handler do
-          consistency, timeout = @execute_options_decoder.decode_options(options_or_consistency)
-          execute_request(Protocol::QueryRequest.new(cql, consistency), timeout)
+          consistency, timeout, trace = @execute_options_decoder.decode_options(options_or_consistency)
+          execute_request(Protocol::QueryRequest.new(cql, consistency, trace), timeout)
         end
       end
 
@@ -103,6 +104,10 @@ module Cql
       DEFAULT_CONNECTION_TIMEOUT = 10
       MAX_RECONNECTION_ATTEMPTS = 5
 
+      def protocol_handler_factory(compressor)
+        lambda { |connection, timeout| Protocol::CqlProtocolHandler.new(connection, timeout, compressor) }
+      end
+
       def extract_hosts(options)
         if options[:hosts]
           options[:hosts].uniq

data/lib/cql/client/asynchronous_prepared_statement.rb

@@ -60,9 +60,9 @@ module Cql
         unless bound_args.size == @raw_metadata.size && args.size <= 1
           raise ArgumentError, "Expected #{@raw_metadata.size} arguments, got #{bound_args.size}"
         end
-        consistency, timeout = @execute_options_decoder.decode_options(args.last)
+        consistency, timeout, trace = @execute_options_decoder.decode_options(args.last)
         statement_id = connection[self]
-        request = Protocol::ExecuteRequest.new(statement_id, @raw_metadata, bound_args, consistency)
+        request = Protocol::ExecuteRequest.new(statement_id, @raw_metadata, bound_args, consistency, trace)
         @request_runner.execute(connection, request, timeout)
       end
     end

data/lib/cql/client/connection_helper.rb

@@ -4,7 +4,7 @@ module Cql
   module Client
     # @private
     class ConnectionHelper
-      def initialize(io_reactor, port, credentials, connections_per_node, connection_timeout, logger)
+      def initialize(io_reactor, port, credentials, connections_per_node, connection_timeout, compressor, logger)
         @io_reactor = io_reactor
         @port = port
         @credentials = credentials
@@ -13,6 +13,7 @@ module Cql
         @logger = logger
         @request_runner = RequestRunner.new
         @keyspace_changer = KeyspaceChanger.new
+        @compressor = compressor
       end
 
       def connect(hosts, initial_keyspace)
@@ -100,18 +101,50 @@ module Cql
       end
 
       def connect_to_host(host, keyspace)
-        @logger.debug('Connecting to node at %s:%d' % [host, @port])
+        if @compressor
+          @logger.debug('Connecting to node at %s:%d using "%s" compression' % [host, @port, @compressor.algorithm])
+        else
+          @logger.debug('Connecting to node at %s:%d' % [host, @port])
+        end
         connected = @io_reactor.connect(host, @port, @connection_timeout)
         connected.flat_map do |connection|
-          initialize_connection(connection, keyspace)
+          f = cache_supported_options(connection)
+          f = f.flat_map { send_startup_request(connection) }
+          f = f.flat_map { |response| maybe_authenticate(response, connection) }
+          f = f.flat_map { identify_node(connection) }
+          f = f.flat_map { change_keyspace(keyspace, connection) }
+          f
         end
       end
 
-      def initialize_connection(connection, keyspace)
-        started = @request_runner.execute(connection, Protocol::StartupRequest.new)
-        authenticated = started.flat_map { |response| maybe_authenticate(response, connection) }
-        identified = authenticated.flat_map { identify_node(connection) }
-        identified.flat_map { @keyspace_changer.use_keyspace(keyspace, connection) }
+      def cache_supported_options(connection)
+        f = @request_runner.execute(connection, Protocol::OptionsRequest.new)
+        f.on_value do |supported_options|
+          connection[:cql_version] = supported_options['CQL_VERSION']
+          connection[:compression] = supported_options['COMPRESSION']
+        end
+        f
+      end
+
+      def send_startup_request(connection)
+        compression = @compressor && @compressor.algorithm
+        if @compressor && !connection[:compression].include?(@compressor.algorithm)
+          @logger.warn(%[Compression algorithm "#{@compressor.algorithm}" not supported (server supports "#{connection[:compression].join('", "')}")])
+          compression = nil
+        end
+        request = Protocol::StartupRequest.new(nil, compression)
+        @request_runner.execute(connection, request)
+      end
+
+      def maybe_authenticate(response, connection)
+        return Future.resolved(connection) unless response.is_a?(AuthenticationRequired)
+        return Future.failed(AuthenticationError.new('Server requested authentication, but no credentials given')) unless @credentials
+        send_credentials(@credentials, connection)
+      end
+
+      def send_credentials(credentials, connection)
+        credentials_request = Protocol::CredentialsRequest.new(credentials)
+        @request_runner.execute(connection, credentials_request)
       end
 
       def identify_node(connection)
@@ -126,18 +159,8 @@ module Cql
         f
       end
 
-      def maybe_authenticate(response, connection)
-        case response
-        when AuthenticationRequired
-          if @credentials
-            credentials_request = Protocol::CredentialsRequest.new(@credentials)
-            @request_runner.execute(connection, credentials_request).map { connection }
-          else
-            Future.failed(AuthenticationError.new('Server requested authentication, but no credentials given'))
-          end
-        else
-          Future.resolved(connection)
-        end
+      def change_keyspace(keyspace, connection)
+        @keyspace_changer.use_keyspace(keyspace, connection)
       end
 
       class FailedConnection

data/lib/cql/client/execute_options_decoder.rb

@@ -2,6 +2,7 @@
 
 module Cql
   module Client
+    # @private
     class ExecuteOptionsDecoder
       def initialize(default_consistency)
         @default_consistency = default_consistency
@@ -10,14 +11,16 @@ module Cql
       def decode_options(options_or_consistency)
         consistency = @default_consistency
         timeout = nil
+        trace = false
         case options_or_consistency
         when Symbol
           consistency = options_or_consistency
         when Hash
           consistency = options_or_consistency[:consistency] || consistency
           timeout = options_or_consistency[:timeout]
+          trace = options_or_consistency[:trace]
         end
-        return consistency, timeout
+        return consistency, timeout, trace
       end
     end
   end

data/lib/cql/client/query_result.rb

@@ -8,14 +8,19 @@ module Cql
       # @return [ResultMetadata]
       attr_reader :metadata
 
+      # The ID of the query trace associated with the query, if any.
+      #
+      # @return [Cql::Uuid]
+      attr_reader :trace_id
+
       # @private
-      def initialize(metadata, rows)
+      def initialize(metadata, rows, trace_id)
         @metadata = ResultMetadata.new(metadata)
         @rows = rows
+        @trace_id = trace_id
       end
 
       # Returns whether or not there are any rows in this result set
-      #
       def empty?
         @rows.empty?
       end
@@ -24,7 +29,6 @@ module Cql
       #
       # @yieldparam [Hash] row each row in the result set as a hash
       # @return [Enumerable<Hash>]
-      #
       def each(&block)
         @rows.each(&block)
       end

data/lib/cql/client/query_trace.rb

@@ -0,0 +1,46 @@
+# encoding: utf-8
+
+module Cql
+  module Client
+    # @private
+    class QueryTrace
+      attr_reader :coordinator, :cql, :started_at, :duration, :events
+
+      # @private
+      def initialize(session, events)
+        if session
+          raise IncompleteTraceError, 'Trace incomplete, try loading it again' unless session['duration']
+          @coordinator = session['coordinator']
+          @cql = (parameters = session['parameters']) && parameters['query']
+          @started_at = session['started_at']
+          @duration = session['duration']/1_000_000.0
+          if events
+            @events = events.map { |e| TraceEvent.new(e) }.freeze
+          end
+        else
+          @events = [].freeze
+        end
+      end
+    end
+
+    # @private
+    class TraceEvent
+      attr_reader :activity, :source, :source_elapsed, :time
+
+      # @private
+      def initialize(event)
+        @activity = event['activity']
+        @source = event['source']
+        @source_elapsed = event['source_elapsed']/1_000_000.0
+        @time = event['event_id'].to_time
+      end
+    end
+
+    # @private
+    class NullQueryTrace < QueryTrace
+      def initialize
+        super(nil, nil)
+      end
+    end
+  end
+end

data/lib/cql/client/request_runner.rb

@@ -8,14 +8,19 @@ module Cql
         connection.send_request(request, timeout).map do |response|
           case response
           when Protocol::RowsResultResponse
-            QueryResult.new(response.metadata, response.rows)
+            QueryResult.new(response.metadata, response.rows, response.trace_id)
+          when Protocol::VoidResultResponse
+            response.trace_id ? VoidResult.new(response.trace_id) : VoidResult::INSTANCE
           when Protocol::ErrorResponse
             cql = request.is_a?(Protocol::QueryRequest) ? request.cql : nil
-            raise QueryError.new(response.code, response.message, cql)
+            details = response.respond_to?(:details) ? response.details : nil
+            raise QueryError.new(response.code, response.message, cql, details)
           when Protocol::SetKeyspaceResultResponse
             KeyspaceChanged.new(response.keyspace)
           when Protocol::AuthenticateResponse
             AuthenticationRequired.new(response.authentication_class)
+          when Protocol::SupportedResponse
+            response.options
           else
             if block_given?
               yield response