hbase-jruby 0.3.5-java → 0.4.0-java

data/CHANGELOG.md

@@ -1,6 +1,19 @@
 Changelog
 =========
 
+0.4.0
+-----
+- Added support for append operation: `HBase::Table#append`
+- Added support for atomic mutations on a single row: `HBase::Table#mutate`
+- Added support for batch operations: `HBase::Table#batch`
+  - This method does not take an argument and requires a block
+  - Don't be confused with the shortcut method to `HBase::Scoped#batch(batch_size)`
+- Changed `HBase::Table#increment` to return the updated values as a Hash
+- Fixed HBase.resolve_dependency!(:local) on CDH distribution
+- Empty-qualifier must be given as 'cf:', and not 'cf'
+- Added `HBase::Row#empty?` method
+- Added `HBase::ByteArray#to_s` method
+
 0.3.5
 -----
 - Improved `Scoped#count` method

data/README.md

@@ -11,6 +11,33 @@
 
     gem install hbase-jruby
 
+### Using hbase-jruby in HBase shell
+
+You can use this gem in HBase shell without external JRuby installation.
+
+First, clone this repository,
+
+```sh
+git clone https://github.com/junegunn/hbase-jruby.git
+```
+
+then start up the shell (`hbase shell`) and type in the following lines:
+
+```ruby
+$LOAD_PATH << 'hbase-jruby/lib'
+require 'hbase-jruby'
+```
+
+Now, you're all set.
+
+```ruby
+# Start using it!
+hbase = HBase.new
+
+hbase.list
+hbase[:my_table].create! :f
+```
+
 ## A quick example
 
 ```ruby
@@ -72,12 +99,18 @@ table.put 1,
   comment1:  'A must-have',
   comment2:  'Rewarding purchase'
 
-# GET
+# GET (using schema)
 book     = table.get(1)
 title    = book[:title]
 comment2 = book[:comment2]
 as_hash  = book.to_h
 
+# GET (not using schema)
+title    = book.string('cf1:title')       # cf:cq notation
+year     = book.short('cf1:year')
+reviews  = book.fixnum('cf2:reviews')
+stars    = book.fixnum(['cf2', 'stars'])  # Array notation of [cf, cq]
+
 # SCAN
 table.range(0..100)
      .project(:cf1, :reviews, :summary)
@@ -122,7 +155,7 @@ Call `HBase.resolve_dependency!` helper method passing one of the arguments list
 | cdh4.1[.*] | Cloudera CDH4.1          | cdh4.1.4        | mvn                 |
 | cdh3[u*]   | Cloudera CDH3            | cdh3u6          | mvn                 |
 | 0.95[.*]   | Apache HBase 0.95        | 0.95.0          | mvn                 |
-| 0.94[.*]   | Apache HBase 0.94        | 0.94.7          | mvn                 |
+| 0.94[.*]   | Apache HBase 0.94        | 0.94.9          | mvn                 |
 | 0.92[.*]   | Apache HBase 0.92        | 0.92.2          | mvn                 |
 | *POM PATH* | Custom Maven POM file    | -               | mvn                 |
 | `:local`   | Local HBase installation | -               | hbase               |
@@ -208,7 +241,19 @@ table.create! cf1: {},
               cf2: { compression: :snappy, bloomfilter: :row }
 ```
 
-## Basic operations
+## List of operations
+
+| Operation          | Description                                                                                   |
+| ------------------ | --------------------------------------------------------------------------------------------- |
+| PUT                | Puts data into the table                                                                      |
+| GET                | Retrieves data from the table by one or more rowkeys                                          |
+| SCAN               | Scans the table for a given range of rowkeys                                                  |
+| DELETE             | Deletes data in the table                                                                     |
+| INCREMENT          | Atomically increments one or more columns                                                     |
+| APPEND             | Appends values to one or more columns within a single row                                     |
+| Checked PUT/DELETE | Atomically checks if the pre-exising data matches the expected value and puts or deletes data |
+| MUTATE             | Performs multiple mutations (PUTS and DELETES) atomically on a single row                     |
+| Batch execution    | Performs multiple actions (PUT, GET, DELETE, INCREMENT, APPEND, and MUTATE) at once           |
 
 ### Defining table schema for easier data access
 
@@ -252,9 +297,9 @@ hbase.schema = {
 ```
 
 Columns that are not defined in the schema can be referenced
-using `FAMILY:QUALIFIER` notation or 2-element Array of column family name (as Symbol) and qualifier,
-however since there's no type information, they are returned as Java byte arrays,
-which have to be decoded manually.
+using `FAMILY:QUALIFIER` notation or 2-element Array of column family name (as
+Symbol) and qualifier, however since there's no type information, they are
+returned as Java byte arrays, which have to be decoded manually.
 
 ### PUT
 
@@ -304,7 +349,7 @@ extra = HBase::Util.from_bytes(:bigdecimal, book['cf2:extra'])
 extra = book.bigdecimal 'cf2:extra'
 ```
 
-### Batch-GET
+#### Batch-GET
 
 ```ruby
 # Pass an array of row keys as the parameter
@@ -316,8 +361,8 @@ books = table.get(['rowkey1', 'rowkey2', 'rowkey3'])
 `to_h` and `to_H` return the Hash representation of the row.
 (The latter returns all values with their timestamp)
 
-If a column is defined in the schema, it is referenced using its quailifier in Symbol type.
-If a column is not defined, it is represented as a 2-element Array
+If a column is defined in the schema, it is referenced using its quailifier in
+Symbol type. If a column is not defined, it is represented as a 2-element Array
 of column family in Symbol and column qualifier as ByteArray.
 Even so, to make it easier to reference those columns, an extended version of
 Hash is returned with which you can also reference them with `FAMILY:QUALIFIER`
@@ -431,6 +476,25 @@ table.delete_row 'rowkey1'
 table.delete_row 'rowkey1', 'rowkey2', 'rowkey3'
 ```
 
+### INCREMENT: Atomic increment of column values
+
+```ruby
+# Atomically increase cf2:reviews by one
+inc = table.increment('rowkey1', reviews: 1)
+puts inc[:reviews]
+
+# Atomically increase two columns by one and five respectively
+inc = table.increment('rowkey1', reviews: 1, stars: 5)
+puts inc[:stars]
+```
+
+### APPEND
+
+```ruby
+ret = table.append 'rowkey1', title: ' (limited edition)', summary: ' ...'
+puts ret[:title]   # Updated title
+```
+
 ### Checked PUT and DELETE
 
 ```ruby
@@ -444,19 +508,59 @@ table.check(:rowkey, in_print: false)
        #   https://github.com/junegunn/hbase-jruby#delete
 ```
 
-### Atomic increment of column values
+### MUTATE: Atomic mutations on a single row (PUTs and DELETEs)
 
 ```ruby
-# Atomically increase cf2:reviews by one
-table.increment('rowkey1', reviews: 1)
+# Currently Put and Delete are supported
+# - Refer to mutateRow method of org.apache.hadoop.hbase.client.HTable
+table.mutate(rowkey) do |m|
+  m.put comment3: 'Nice', comment4: 'Great'
+  m.delete :comment1, :comment2
+end
+```
 
-# Atomically increase two columns by one and five respectively
-table.increment('rowkey1', reviews: 1, stars: 5)
+### Batch execution
 
-# Increase column values of multiple rows.
-# - Atomicity is only guaranteed within each row.
-table.increment 'rowkey1' => { reviews: 1, stars: 5 },
-                'rowkey2' => { reviews: 1, stars: 3 }
+*Disclaimer*: The ordering of execution of the actions is not defined.
+Refer to the documentation of batch method of [HTable class](http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html).
+
+```ruby
+ret = table.batch do |b|
+  b.put rowkey1, 'cf1:a' => 100, 'cf1:b' => 'hello'
+  b.get rowkey2
+  b.append rowkey3, 'cf1:b' => 'world'
+  b.delete rowkey3, 'cf2', 'cf3:z'
+  b.increment rowkey3, 'cf1:a' => 200, 'cf1:c' => 300
+  b.mutate(rowkey4) do |m|
+    m.put 'cf3:z' => 3.14
+    m.delete 'cf3:y', 'cf4'
+  end
+end
+```
+
+`batch` method returns an Array of Hashes which contains the results of the
+actions in the order they are specified in the block. Each Hash has `:type` entry
+(:get, :put, :append, etc.) and `:result` entry. If the type of an action is
+:put, :delete, or :mutate, the `:result` will be given as a boolean. If it's an
+:increment or :append, a plain Hash will be returned as the `:result`, just like
+in [increment](https://github.com/junegunn/hbase-jruby#increment-atomic-increment-of-column-values)
+and [append](https://github.com/junegunn/hbase-jruby#append) methods.
+For :get action, `HBase::Row` instance will be returned or nil if not found.
+
+If one or more actions has failed, `HBase::BatchException` will be raised.
+Although you don't get to receive the return value from batch method,
+you can still access the partial results using `results` method of
+`HBase::BatchException`.
+
+```ruby
+results =
+  begin
+    table.batch do |b|
+      # ...
+    end
+  rescue HBase::BatchException => e
+    e.results
+  end
 ```
 
 ### SCAN
@@ -579,8 +683,8 @@ scope.range(1, 100).
 
 ### *filter*
 
-You can configure server-side filtering of rows and columns with `HBase::Scoped#filter` calls.
-Multiple calls have conjunctive effects.
+You can configure server-side filtering of rows and columns with
+`HBase::Scoped#filter` calls. Multiple calls have conjunctive effects.
 
 ```ruby
 # Range scanning the table with filters
@@ -617,10 +721,10 @@ end
 
 ### *while*
 
-`HBase::Scoped#while` method takes the same parameters as `filter` method, the difference is that
-each filtering condition passed to `while` method is wrapped by `WhileMatchFilter`,
-which aborts scan immediately when the condition is not met at a certain row.
-See the following example.
+`HBase::Scoped#while` method takes the same parameters as `filter` method, the
+difference is that each filtering condition passed to `while` method is wrapped
+by `WhileMatchFilter`, which aborts scan immediately when the condition is not
+met at a certain row. See the following example.
 
 ```ruby
 (0...30).each do |idx|
@@ -669,8 +773,8 @@ scoped.project(offset: 1000, limit: 10)
 
 When using column filters on *fat* rows with many columns,
 it's advised that you limit the batch size with `HBase::Scoped#batch` call
-to avoid fetching all columns at once.
-However setting batch size allows multiple rows with the same row key are returned during scan.
+to avoid fetching all columns at once. However setting batch size allows
+multiple rows with the same row key are returned during scan.
 
 ```ruby
 # Let's say that we have rows with more than 10 columns whose qualifiers start with `str`
@@ -818,8 +922,9 @@ table.raw_families
   #    "BLOCKCACHE"          => "true"}}
 ```
 
-These String key-value pairs are not really a part of the public API of HBase, and thus might change over time.
-However, they are most useful when you need to create a table with the same properties as the existing one.
+These String key-value pairs are not really a part of the public API of HBase,
+and thus might change over time. However, they are most useful when you need to
+create a table with the same properties as the existing one.
 
 ```ruby
 hbase[:dupe_table].create!(table.raw_families, table.raw_properties)
@@ -988,8 +1093,8 @@ end
 
 ### Lexicographic scan order
 
-HBase stores rows in the lexicographic order of the rowkeys in their byte array representations.
-Thus the type of row key affects the scan order.
+HBase stores rows in the lexicographic order of the rowkeys in their byte array
+representations. Therefore, the type of the row key affects the scan order.
 
 ```ruby
 (1..15).times do |i|

data/lib/hbase-jruby/batch_exception.rb

@@ -0,0 +1,12 @@
+class HBase
+class BatchException < RuntimeError
+  attr_reader :java_exception, :results
+
+  def initialize x, results
+    super x.to_s
+    @java_exception = x
+    @results = results
+  end
+end
+end
+

data/lib/hbase-jruby/byte_array.rb

@@ -37,6 +37,12 @@ class ByteArray
     initialize_(*values)
   end
 
+  # Returns the String representation of the underlying byte array
+  # @return [String]
+  def to_s
+    @java.to_s
+  end
+
   def each
     return enum_for(:each) unless block_given?
     @java.to_a.each { |byte| yield byte }

data/lib/hbase-jruby/dependency.rb

@@ -17,7 +17,7 @@ class HBase
     'cdh4.1' => 'cdh4.1.4',
     'cdh3'   => 'cdh3u6',
     '0.95'   => '0.95.0',
-    '0.94'   => '0.94.7',
+    '0.94'   => '0.94.9',
     '0.92'   => '0.92.2',
   }
 
@@ -51,7 +51,7 @@ class HBase
           # Check for hbase executable
           hbase = `which hbase`
           raise RuntimeError, "Cannot find `hbase` executable" if hbase.empty?
-          `hbase classpath`.split(':')
+          `hbase classpath`.strip.split(':').map { |e| Dir[e] }.flatten
         else
           # Check for Maven executable
           mvn = `which mvn`
@@ -99,8 +99,8 @@ class HBase
 
       # Load jars
       jars_loaded = jars.select { |jar|
-        File.exists?(jar) &&
-        File.extname(jar) == '.jar' &&
+        File.file?(jar) &&
+        File.extname(jar).downcase == '.jar' &&
         require(jar)
       }
 
@@ -158,9 +158,11 @@ class HBase
         HBase::Table => %w[
           org.apache.hadoop.hbase.HColumnDescriptor
           org.apache.hadoop.hbase.HTableDescriptor
+          org.apache.hadoop.hbase.client.Append
           org.apache.hadoop.hbase.client.Delete
           org.apache.hadoop.hbase.client.Increment
           org.apache.hadoop.hbase.client.Put
+          org.apache.hadoop.hbase.client.RowMutations
           org.apache.hadoop.hbase.io.hfile.Compression
           org.apache.hadoop.hbase.regionserver.StoreFile
         ],

data/lib/hbase-jruby/row.rb

@@ -1,11 +1,17 @@
 require 'bigdecimal'
 
 class HBase
-# Represents a row returned by HBase
+# Represents a set of key-values returned by HBase
 # @author Junegunn Choi <junegunn.c@gmail.com>
 class Row
   include Enumerable
 
+  # Returns if the returned row is empty
+  # @return [Boolean]
+  def empty?
+    @result.empty?
+  end
+
   # Returns the rowkey of the row
   # @param [Symbol] type The type of the rowkey
   #   Can be one of :string, :symbol, :fixnum, :float, :short, :int, :bigdecimal, :boolean and :raw.
@@ -269,7 +275,7 @@ class Row
 
 private
   def get_value col, with_versions = false
-    cf, cq, _ = @table.lookup_and_parse col
+    cf, cq, _ = @table.lookup_and_parse col, true
     if with_versions
       # Need to make it a Ruby hash:
       #   Prevents implicit conversion from ruby type to java type when updating the Hash

data/lib/hbase-jruby/schema.rb

@@ -86,9 +86,10 @@ class Schema
 
   # @private
   # @param [Symbol] table
-  def lookup_and_parse table, col
+  def lookup_and_parse table, col, expect_cq
     cf, cq, type = lookup table, col
     cf, cq = Util.parse_column_name(cf ? [cf, cq] : col)
+    raise ArgumentError, "Invalid column key: #{col}" if expect_cq && cq.nil?
     return [cf, cq, type]
   end
 

data/lib/hbase-jruby/scoped.rb

@@ -13,7 +13,7 @@ class Scoped
   # A clean HBase::Scoped object for the same table
   # @return [HBase::Scope] A clean HBase::Scoped object for the same table
   def unscope
-    Scoped.send(:new, @table)
+    Scoped.send(:new, @table, @dcaching)
   end
 
   # Count the number of rows in the scope
@@ -253,7 +253,7 @@ class Scoped
 
 private
   # @param [HBase::Table] table
-  def initialize table
+  def initialize table, default_caching
     @table    = table
     @filters  = []
     @project  = []
@@ -261,6 +261,7 @@ private
     @range    = nil
     @versions = nil
     @batch    = nil
+    @dcaching = default_caching
     @caching  = nil
     @limit    = nil
     @mlimit   = nil
@@ -533,7 +534,9 @@ private
           scan.setMaxResultSize(@limit)
         else
           @mlimit = @limit
-          scan.caching = [@mlimit, @caching].compact.min
+          if [@caching, @dcaching].compact.all? { |c| @mlimit < c }
+            scan.caching = @mlimit
+          end
         end
       end
 
@@ -615,7 +618,7 @@ private
       case f
       when Hash
         f.map { |col, val|
-          cf, cq, type = @table.lookup_and_parse col
+          cf, cq, type = @table.lookup_and_parse col, true
 
           case val
           when Array

data/lib/hbase-jruby/table/batch_action.rb

@@ -0,0 +1,69 @@
+class HBase
+class Table
+# Class used to register actions to perform in batch
+class BatchAction
+  attr_reader :actions
+  attr_reader :types
+
+  class BatchGetScoped
+    # @see HBase::Scoped#get
+    def get rowkey
+      @callback.call @scoped.send(:getify, rowkey)
+    end
+
+    [:range, :project, :filter, :versions, :time_range, :at].each do |method|
+      define_method(method) do |*args|
+        BatchGetScoped.send(:new, @scoped.send(method, *args), @callback)
+      end
+    end
+
+  private
+    def initialize scoped, callback
+      @scoped   = scoped
+      @callback = callback
+    end
+  end
+
+  # @see HBase::Table#put
+  def put *args
+    @actions << { :type => :put, :action => @mutation.put(*args) }
+  end
+
+  # @see HBase::Table#delete
+  def delete *args
+    @actions << { :type => :delete, :action => @mutation.delete(*args) }
+  end
+
+  # @see HBase::Table#append
+  def append *args
+    @actions << { :type => :append, :action => @mutation.append(*args) }
+  end
+
+  # @see HBase::Table#increment
+  def increment *args
+    @actions << { :type => :increment, :action => @mutation.increment(*args) }
+  end
+
+  # @see HBase::Table#mutate
+  def mutate *args, &blk
+    @actions << { :type => :mutate, :action => @mutation.mutate(*args, &blk) }
+  end
+
+  [:get, :range, :project, :filter, :versions, :time_range, :at].each do |method|
+    define_method(method) do |*args|
+      BatchGetScoped.send(:new, @table.scoped, proc { |get|
+        @actions << { :type => :get, :action => get }
+      }).send(method, *args)
+    end
+  end
+
+private
+  def initialize table, mutation
+    @table    = table
+    @mutation = mutation
+    @actions  = []
+  end
+end#BatchAction
+end#Table
+end#HBase
+

data/lib/hbase-jruby/table/checked_operation.rb

@@ -1,22 +1,25 @@
 class HBase
 class Table
   class CheckedOperation
-    def initialize table, rowkey, cf, cq, val
-      @table  = table
-      @rowkey = rowkey
-      @cf     = cf
-      @cq     = cq
-      @val    = val
+    def initialize table, mutation, rowkey, cf, cq, val
+      @table   = table
+      @mutation = mutation
+      @rowkey   = rowkey
+      @cf       = cf
+      @cq       = cq
+      @val      = val
     end
 
     # @param [Hash] props
     def put props
-      @table.htable.checkAndPut @rowkey, @cf, @cq, @val, @table.send(:make_put, @rowkey, props)
+      @table.htable.checkAndPut(
+        @rowkey, @cf, @cq, @val, @mutation.put(@rowkey, props))
     end
 
     # @param [Object] *extra Optional delete specification. Column family, qualifier, and timestamps
     def delete *extra
-      @table.htable.checkAndDelete @rowkey, @cf, @cq, @val, @table.send(:make_delete, @rowkey, *extra)
+      @table.htable.checkAndDelete(
+        @rowkey, @cf, @cq, @val, @mutation.delete(@rowkey, *extra))
     end
   end
 end

data/lib/hbase-jruby/table/mutation.rb

@@ -0,0 +1,130 @@
+class HBase
+class Table
+# Generate single-row mutation objects
+class Mutation
+  include HBase::Util
+
+  def initialize table
+    @table = table
+  end
+
+  def put rowkey, props
+    Put.new(Util.to_bytes rowkey).tap { |put|
+      props.each do |col, val|
+        next if val.nil?
+
+        cf, cq, type = @table.lookup_and_parse col, true
+
+        case val
+        when Hash
+          val.each do |t, v|
+            case t
+            # Timestamp / Ruby Time
+            when Time, Fixnum
+              put.add cf, cq, time_to_long(t), Util.to_typed_bytes(type, v)
+            # Types: :byte, :short, :int, ...
+            else
+              put.add cf, cq, Util.to_typed_bytes(t, v)
+            end unless v.nil?
+          end
+        else
+          put.add cf, cq, Util.to_typed_bytes(type, val)
+        end
+      end
+      raise ArgumentError, "no column to put" if put.empty?
+    }
+  end
+
+  def delete rowkey, *extra
+    Delete.new(Util.to_bytes rowkey).tap { |del|
+      cf = cq = nil
+      prcd = false
+
+      prc = lambda do
+        unless prcd
+          if cq
+            # Delete all versions
+            del.deleteColumns cf, cq
+          elsif cf
+            del.deleteFamily cf
+          end
+        end
+      end
+
+      extra.each do |x|
+        case x
+        when Fixnum, Time
+          if cq
+            del.deleteColumn cf, cq, time_to_long(x)
+            prcd = true
+          else
+            raise ArgumentError, 'qualifier not given'
+          end
+        else
+          prc.call
+          cf, cq, _ = @table.lookup_and_parse x, false
+          prcd = false
+        end
+      end
+      prc.call
+    }
+  end
+
+  def increment rowkey, *spec
+    if spec.first.is_a?(Hash)
+      spec = spec.first
+    else
+      c, b = spec
+      spec = { c => (b || 1) }
+    end
+
+    Increment.new(Util.to_bytes rowkey).tap { |inc|
+      spec.each do |col, by|
+        cf, cq, _ = @table.lookup_and_parse col, true
+        inc.addColumn cf, cq, by
+      end
+    }
+  end
+
+  def append rowkey, spec
+    Append.new(Util.to_bytes rowkey).tap { |apnd|
+      spec.each do |col, val|
+        cf, cq, _ = @table.lookup_and_parse col, true
+        apnd.add(cf, cq, Util.to_bytes(val))
+      end
+    }
+  end
+
+  def mutate rowkey
+    rm = Mutator.new(self, rowkey)
+    yield rm
+    org.apache.hadoop.hbase.client.RowMutations.new(Util.to_bytes rowkey).tap { |m|
+      rm.mutations.each do |action|
+        m.add action
+      end
+    }
+  end
+
+  class Mutator
+    attr_reader :mutations
+
+    def initialize mutation, rowkey
+      @mutation  = mutation
+      @rowkey    = rowkey
+      @mutations = []
+    end
+
+    # @param [Hash] props Column values
+    def put props
+      @mutations << @mutation.put(@rowkey, props)
+      self
+    end
+
+    def delete *args
+      @mutations << @mutation.delete(@rowkey, *args)
+      self
+    end
+  end#RowMutation
+end#Mutation
+end#Table
+end#HBase