rbhive 0.6.0 → 1.0.0.pre

data/README.md

@@ -1,7 +1,7 @@
-# RBHive -- Ruby thrift lib for executing Hive queries
+# RBHive - A Ruby Thrift client for Apache Hive
 
 RBHive is a simple Ruby gem to communicate with the [Apache Hive](http://hive.apache.org)
-Thrift server.
+Thrift servers.
 
 It supports:
 * Hiveserver (the original Thrift service shipped with Hive since early releases)
@@ -13,6 +13,10 @@ It is capable of using the following Thrift transports:
 * SaslClientTransport ([SASL-enabled](http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer) transport)
 * HTTPClientTransport (tunnels Thrift over HTTP)
 
+As of version 1.0, it supports asynchronous execution of queries. This allows you to submit
+a query, disconnect, then reconnect later to check the status and retrieve the results.
+This frees systems of the need to keep a persistent TCP connection. 
+
 ## About Thrift services and transports
 
 ### Hiveserver
@@ -31,6 +35,12 @@ supported; starting with Hive 0.12, HTTPClientTransport is also supported.
 Each of the versions after Hive 0.10 has a slightly different Thrift interface; when
 connecting, you must specify the Hive version or you may get an exception.
 
+Hiveserver2 supports (in versions later than 0.12) asynchronous query execution. This
+works by submitting a query and retrieving a handle to the execution process; you can
+then reconnect at a later time and retrieve the results using this handle.
+Using the asynchronous methods has some caveats - please read the Asynchronous Execution
+section of the documentation thoroughly before using them.
+
 RBHive implements this client with the `RBHive::TCLIConnection` class.
 
 #### Warning!
@@ -129,6 +139,75 @@ In addition, you can explicitly set the Thrift protocol version according to thi
 | `:PROTOCOL_V6`  | V6                      | Updated during Hive 0.13 development, adds binary type for binary payload, uses columnar result set
 | `:PROTOCOL_V7`  | V7                      | Used by Hive 0.13 release, support for token-based delegation connections
 
+## Asynchronous execution with Hiveserver2
+
+In versions of Hive later than 0.12, the Thrift server supports asynchronous execution.
+
+The high-level view of using this feature is as follows:
+1. Submit your query using `async_execute(query)`. This function returns a hash
+   with the following keys: `:guid`, `:secret`, and `:session`. You don't need to
+   care about the internals of this hash - all methods that interact with an async
+   query require this hash, and you can just store it and hand it to the methods.
+2. To check the state of the query, call `async_state(handles)`, where `handles`
+   is the handles hash given to you when you called `async_execute(query)`.
+3. To retrieve results, call either `async_fetch(handles)` or `async_fetch_in_batch(handles)`,
+   which work like the non async methods.
+4. When you're done with the query, call `async_close_session(handles)`.
+
+### Memory leaks
+
+When you call `async_close_session(handles)`, *all async handles created during this
+session are closed*.
+
+If you do not close the sessions you create, *you will leak memory in the Hiveserver2 process*.
+Be very careful to close your sessions!
+
+### Method documentation
+
+#### `async_execute(query)`
+
+This method submits a query for async execution. The hash you get back is used in the other
+async methods, and will look like this:
+
+    {
+      :guid => (binary string),
+      :secret => (binary string),
+      :session => (binary string)
+    }
+
+The Thrift protocol specifies the strings as "binary" - which means they have no encoding.
+Be *extremely* careful when manipulating or storing these values, as they can quite easily
+get converted to UTF-8 strings, which will make them invalid when trying to retrieve async data.
+
+#### `async_state(handles)`
+
+`handles` is the hash returned by `async_execute(query)`. The state will be a symbol with
+one of the following values and meanings:
+
+| symbol                | meaning
+| --------------------- | -------
+| :initialized          | The query is initialized in Hive and ready to run
+| :running              | The query is running (either as a MapReduce job or within process)
+| :finished             | The query is completed and results can be retrieved
+| :cancelled            | The query was cancelled by a user
+| :closed               | Unknown at present
+| :error                | The query is invalid semantically or broken in another way
+| :unknown              | The query is in an unknown state
+| :pending              | The query is ready to run but is not running
+
+There are also the utility methods `async_is_complete?(handles)`, `async_is_running?(handles)`, 
+`async_is_failed?(handles)` and `async_is_cancelled?(handles)`.
+
+#### `async_cancel(handles)`
+
+Calling this method will cancel the query in execution.
+
+#### `async_fetch(handles)`, `async_fetch_in_batch(handles)`
+
+These methods let you fetch the results of the async query, if they are complete. If you call
+these methods on an incomplete query, they will raise an exception. They work in exactly the
+same way as the normal synchronous methods.
+
 ## Examples
 
 ### Fetching results

data/lib/rbhive/t_c_l_i_connection.rb

@@ -103,7 +103,6 @@ module RBHive
       @client = Hive2::Thrift::TCLIService::Client.new(@protocol)
       @session = nil
       @logger.info("Connecting to HiveServer2 #{server} on port #{port}")
-      @mutex = Mutex.new
     end
     
     def thrift_hive_protocol(version)
@@ -169,7 +168,11 @@ module RBHive
     end
 
     def execute(query)
-      execute_safe(query)
+      @logger.info("Executing Hive Query: #{query}")
+      req = prepare_execute_statement(query)
+      exec_result = client.ExecuteStatement(req)
+      raise_error_if_failed!(exec_result)
+      exec_result
     end
 
     def priority=(priority)
@@ -185,6 +188,118 @@ module RBHive
       self.execute("SET #{name}=#{value}")
     end
     
+    # Async execute
+    def async_execute(query)
+      @logger.info("Executing query asynchronously: #{query}")
+      op_handle = @client.ExecuteStatement(
+        Hive2::Thrift::TExecuteStatementReq.new(
+          sessionHandle: @session.sessionHandle,
+          statement: query,
+          runAsync: true
+        )
+      ).operationHandle
+      
+      # Return handles to get hold of this query / session again
+      {
+        session: @session.sessionHandle, 
+        guid: op_handle.operationId.guid, 
+        secret: op_handle.operationId.secret
+      }
+    end
+    
+    # Is the query complete?
+    def async_is_complete?(handles)
+      async_state(handles) == :finished
+    end
+    
+    # Is the query actually running?
+    def async_is_running?(handles)
+      async_state(handles) == :running
+    end
+    
+    # Has the query failed?
+    def async_is_failed?(handles)
+      async_state(handles) == :error
+    end
+    
+    def async_is_cancelled?(handles)
+      async_state(handles) == :cancelled
+    end
+    
+    def async_cancel(handles)
+      @client.CancelOperation(prepare_cancel_request(handles))
+    end
+    
+    # Map states to symbols
+    def async_state(handles)
+      response = @client.GetOperationStatus(
+        Hive2::Thrift::TGetOperationStatusReq.new(operationHandle: prepare_operation_handle(handles))
+      )
+      puts response.operationState
+      case response.operationState
+      when Hive2::Thrift::TOperationState::FINISHED_STATE
+        return :finished
+      when Hive2::Thrift::TOperationState::INITIALIZED_STATE
+        return :initialized
+      when Hive2::Thrift::TOperationState::RUNNING_STATE
+        return :running
+      when Hive2::Thrift::TOperationState::CANCELED_STATE
+        return :cancelled
+      when Hive2::Thrift::TOperationState::CLOSED_STATE
+        return :closed
+      when Hive2::Thrift::TOperationState::ERROR_STATE
+        return :error
+      when Hive2::Thrift::TOperationState::UKNOWN_STATE
+        return :unknown
+      when Hive2::Thrift::TOperationState::PENDING_STATE
+        return :pending
+      else
+        return :state_not_in_protocol
+      end
+    end
+        
+    # Async fetch results from an async execute
+    def async_fetch(handles, max_rows = 100)
+      # Can't get data from an unfinished query
+      unless async_is_complete?(handles)
+        raise "Can't perform fetch on a query in state: #{async_state(handles[:guid], handles[:secret])}"
+      end
+      
+      # Fetch and
+      fetch_rows(prepare_operation_handle(handles), :first, max_rows)
+    end
+    
+    # Performs a query on the server, fetches the results in batches of *batch_size* rows
+    # and yields the result batches to a given block as arrays of rows.
+    def async_fetch_in_batch(handles, batch_size = 1000, &block)
+      raise "No block given for the batch fetch request!" unless block_given?
+      # Can't get data from an unfinished query
+      unless async_is_complete?(handles)
+        raise "Can't perform fetch on a query in state: #{async_state(handles[:guid], handles[:secret])}"
+      end
+
+      # Now let's iterate over the results
+      loop do
+        rows = fetch_rows(prepare_operation_handle(handles), :next, batch_size)
+        break if rows.empty?
+        yield rows
+      end
+    end
+    
+    def async_close_session(handles)
+      validate_handles!(handles)
+      @client.CloseSession(Hive2::Thrift::TCloseSessionReq.new( sessionHandle: handles[:session] ))
+    end
+    
+    # Pull rows from the query result
+    def fetch_rows(op_handle, orientation = :first, max_rows = 1000)
+      fetch_req = prepare_fetch_results(op_handle, orientation, max_rows)
+      fetch_results = @client.FetchResults(fetch_req)
+      raise_error_if_failed!(fetch_results)
+      rows = fetch_results.results.rows
+      TCLIResultSet.new(rows, TCLISchemaDefinition.new(get_schema_for(op_handle), rows.first))
+    end
+        
     # Performs a explain on the supplied query on the server, returns it as a ExplainResult.
     # (Only works on 0.12 if you have this patch - https://issues.apache.org/jira/browse/HIVE-5492)
     def explain(query)
@@ -197,58 +312,37 @@ module RBHive
 
     # Performs a query on the server, fetches up to *max_rows* rows and returns them as an array.
     def fetch(query, max_rows = 100)
-      safe do
-        # Execute the query and check the result
-        exec_result = execute_unsafe(query)
-        raise_error_if_failed!(exec_result)
-
-        # Get search operation handle to fetch the results
-        op_handle = exec_result.operationHandle
-
-        # Prepare and execute fetch results request
-        fetch_req = prepare_fetch_results(op_handle, :first, max_rows)
-        fetch_results = client.FetchResults(fetch_req)
-        raise_error_if_failed!(fetch_results)
-
-        # Get data rows and format the result
-        rows = fetch_results.results.rows
-        the_schema = TCLISchemaDefinition.new(get_schema_for( op_handle ), rows.first)
-        TCLIResultSet.new(rows, the_schema)
-      end
+      # Execute the query and check the result
+      exec_result = execute(query)
+      raise_error_if_failed!(exec_result)
+
+      # Get search operation handle to fetch the results
+      op_handle = exec_result.operationHandle
+      
+      # Fetch the rows
+      fetch_rows(op_handle, :first, max_rows)
     end
 
     # Performs a query on the server, fetches the results in batches of *batch_size* rows
     # and yields the result batches to a given block as arrays of rows.
     def fetch_in_batch(query, batch_size = 1000, &block)
       raise "No block given for the batch fetch request!" unless block_given?
-      safe do
-        # Execute the query and check the result
-        exec_result = execute_unsafe(query)
-        raise_error_if_failed!(exec_result)
-
-        # Get search operation handle to fetch the results
-        op_handle = exec_result.operationHandle
-
-        # Prepare fetch results request
-        fetch_req = prepare_fetch_results(op_handle, :next, batch_size)
-
-        # Now let's iterate over the results
-        loop do
-          # Fetch next batch and raise an exception if it failed
-          fetch_results = client.FetchResults(fetch_req)
-          raise_error_if_failed!(fetch_results)
+      
+      # Execute the query and check the result
+      exec_result = execute(query)
+      raise_error_if_failed!(exec_result)
 
-          # Get data rows from the result
-          rows = fetch_results.results.rows
-          break if rows.empty?
+      # Get search operation handle to fetch the results
+      op_handle = exec_result.operationHandle
 
-          # Prepare schema definition for the row
-          schema_for_req ||= get_schema_for(op_handle)
-          the_schema ||= TCLISchemaDefinition.new(schema_for_req, rows.first)
+      # Prepare fetch results request
+      fetch_req = prepare_fetch_results(op_handle, :next, batch_size)
 
-          # Format the results and yield them to the given block
-          yield TCLIResultSet.new(rows, the_schema)
-        end
+      # Now let's iterate over the results
+      loop do
+        rows = fetch_rows(op_handle, :next, batch_size)
+        break if rows.empty?
+        yield rows
       end
     end
 
@@ -275,26 +369,6 @@ module RBHive
 
     private
 
-    def execute_safe(query)
-      safe do
-        exec_result = execute_unsafe(query)
-        raise_error_if_failed!(exec_result)
-        exec_result
-      end
-    end
-
-    def execute_unsafe(query)
-      @logger.info("Executing Hive Query: #{query}")
-      req = prepare_execute_statement(query)
-      client.ExecuteStatement(req)
-    end
-
-    def safe
-      ret = nil
-      @mutex.synchronize { ret = yield }
-      ret
-    end
-
     def prepare_open_session(client_protocol)
       req = ::Hive2::Thrift::TOpenSessionReq.new( @options[:sasl_params].nil? ? [] : @options[:sasl_params] )
       req.client_protocol = client_protocol
@@ -323,6 +397,27 @@ module RBHive
       )
     end
 
+    def prepare_operation_handle(handles)
+      validate_handles!(handles)
+      Hive2::Thrift::TOperationHandle.new(
+        operationId: Hive2::Thrift::THandleIdentifier.new(guid: handles[:guid], secret: handles[:secret]),
+        operationType: Hive2::Thrift::TOperationType::EXECUTE_STATEMENT,
+        hasResultSet: false
+      )
+    end
+    
+    def prepare_cancel_request(handles)
+      Hive2::Thrift::TCancelOperationReq.new(
+        operationHandle: prepare_operation_handle(handles)
+      )
+    end
+    
+    def validate_handles!(handles)
+      unless handles.has_key?(:guid) and handles.has_key?(:secret) and handles.has_key?(:session)
+        raise "Invalid handles hash: #{handles.inspect}"
+      end
+    end
+
     def get_schema_for(handle)
       req = ::Hive2::Thrift::TGetResultSetMetadataReq.new( operationHandle: handle )
       metadata = client.GetResultSetMetadata( req )

data/lib/rbhive/version.rb

@@ -1,3 +1,3 @@
 module RBHive
-  VERSION = '0.6.0'
+  VERSION = '1.0.0.pre'
 end

metadata

@@ -1,8 +1,8 @@
 --- !ruby/object:Gem::Specification
 name: rbhive
 version: !ruby/object:Gem::Version
-  version: 0.6.0
-  prerelease: 
+  version: 1.0.0.pre
+  prerelease: 6
 platform: ruby
 authors:
 - Forward3D
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-03-28 00:00:00.000000000 Z
+date: 2014-03-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thrift
@@ -134,16 +134,13 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2810079357689827941
+      hash: 2597338757284379755
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>='
+  - - ! '>'
     - !ruby/object:Gem::Version
-      version: '0'
-      segments:
-      - 0
-      hash: 2810079357689827941
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23