hbase-jruby 0.1.2-java → 0.1.3-java

data/CHANGELOG.md

@@ -1,6 +1,12 @@
 Changelog
 =========
 
+0.1.3
+-----
+- Supports Ruby 1.8 compatibility mode
+- Fix: Correct return value from `HBase::resolve_dependency!`
+- Fix: Appropriately close result scanners
+
 0.1.2
 -----
 

data/README.md

@@ -2,9 +2,9 @@
 
 *hbase-jruby* is a Ruby-esque interface for accessing HBase from JRuby.
 
-You can of course just use the native Java APIs of HBase,
-but doing so requires a lot of keystrokes even for the most basic operations and
-can easily lead to overly verbose code that will be frowned upon by Rubyists.
+With JRuby, you can of course just use the native Java APIs of HBase,
+but doing so requires lots of keystrokes even for the most basic operations and
+can lead to having overly verbose code that will be frowned upon by Rubyists.
 Anyhow, JRuby is Ruby, not Java, right?
 
 *hbase-jruby* provides the followings:
@@ -34,9 +34,10 @@ string = row.string('cf1:b')
 table.range('rowkey1'..'rowkey9').
       filter('cf1:a' => 100..200,             # cf1:a between 100 and 200
              'cf1:b' => 'Hello',              # cf1:b = 'Hello'
-             'cf2:c' => /world/i).            # cf2:c matches /world/i
-             'cf2:d' => ['foo', /^BAR/i],     # cf2:d = 'foo' OR matches /^BAR/i
-      project('cf1:a', 'cf2').each do |row|
+             'cf2:c' => /world/i,             # cf2:c matches /world/i
+             'cf2:d' => ['foo', /^BAR/i]).    # cf2:d = 'foo' OR matches /^BAR/i
+      project('cf1:a', 'cf2').
+      each do |row|
   puts row.fixnum('cf1:a')
 end
 
@@ -46,16 +47,6 @@ table.delete(:rowkey9)
 
 ## Installation
 
-Add this line to your application's Gemfile:
-
-    gem 'hbase-jruby'
-
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
     $ gem install hbase-jruby
 
 ## Setting up
@@ -63,13 +54,10 @@ Or install it yourself as:
 ### Resolving Hadoop/HBase dependency
 
 To be able to access HBase from JRuby, Hadoop/HBase dependency must be satisfied.
-This can be done by setting up CLASSPATH variable beforehand
-or by `require`ing relevant JAR files after launch.
-However, downloading all the JAR files and manually putting them in CLASSPATH is a PITA,
-especially when HBase is not installed on local system.
-
-*hbase-jruby* includes `HBase.resolve_dependency!` helper method,
-which resolves Hadoop/HBase dependency.
+This can be done by either setting up CLASSPATH variable beforehand
+or by `require`ing relevant JAR files after launching JRuby.
+However, that's a lot of work, so *hbase-jruby* provides `HBase.resolve_dependency!` helper method,
+which automatically resolves Hadoop/HBase dependency.
 
 #### Preconfigured dependencies
 
@@ -88,10 +76,13 @@ require 'hbase-jruby'
 HBase.resolve_dependency! 'cdh4.1.2'
 ```
 
-#### Customized dependencies
+(If you're behind an http proxy, set up your ~/.m2/settings.xml file
+as described in [this page](http://maven.apache.org/guides/mini/guide-proxies.html))
+
+#### Custom dependency
 
-If you use another version of HBase and Hadoop,
-you can use your own Maven pom.xml file with its customized Hadoop/HBase dependency
+If you use other versions of HBase and Hadoop,
+you can use your own Maven pom.xml file with its Hadoop/HBase dependency.
 
 ```ruby
 HBase.resolve_dependency! '/project/my-hbase/pom.xml'
@@ -99,7 +90,7 @@ HBase.resolve_dependency! '/project/my-hbase/pom.xml'
 
 #### Using `hbase classpath` command
 
-If you have HBase installed on your system, it's possible to find the JAR files
+If you have HBase installed on your system, it's possible to locate the JAR files
 for that local installation with `hbase classpath` command.
 You can tell `resolve_dependency!` method to do so by passing it special `:hbase` parameter.
 
@@ -569,17 +560,16 @@ scoped.count
 
 ## Basic aggregation using coprocessor
 
-*hbase-jruby* provides a few basic aggregation methods using
-the built-in coprocessor called
+You can perform some basic aggregation using the built-in coprocessor called
 `org.apache.hadoop.hbase.coprocessor.AggregateImplementation`.
 
 To enable this feature, call `enable_aggregation!` method,
-which will first disable the table, add the coprocessor, then enable it.
+which adds the coprocessor to the table.
 
 ```ruby
 table.enable_aggregation!
-  # Just a shorthand notation for
-  #   table.add_coprocessor! 'org.apache.hadoop.hbase.coprocessor.AggregateImplementation'
+# Just a shorthand notation for
+#   table.add_coprocessor! 'org.apache.hadoop.hbase.coprocessor.AggregateImplementation'
 ```
 
 Then you can get the sum, average, minimum, maximum, row count, and standard deviation
@@ -598,8 +588,8 @@ table.project('cf1:a').aggregate(:row_count)
 table.project('cf1:a', 'cf1:b').aggregate(:sum)
 ```
 
-By default, aggregate method assumes the column values are 8-byte integers.
-For types other than that, you can pass your own ColumnInterpreter.
+By default, aggregate method assumes that the projected values are 8-byte integers.
+For other data types, you can pass your own ColumnInterpreter.
 
 ```ruby
 table.project('cf1:b').aggregate(:sum, MyColumnInterpreter.new)
@@ -685,26 +675,33 @@ table.remove_coprocessor! cp_class_name1
 
 You can perform other types of administrative tasks
 with Native Java [HBaseAdmin object](http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html),
-which can be obtained by `HBase#admin` method which will automatically close the object at the end of the given block.
+which can be obtained by `HBase#admin` method. Optionally, a block can be given
+so that the HBaseAdmin object is automatically closed at the end of the given block.
 
 ```ruby
 # Advanced table administration with HBaseAdmin object
 #   http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html
-hbase.admin do |admin|
-  # ...
-end
-
-# Without the block
 admin = hbase.admin
 # ...
 admin.close
+
+# With the block
+hbase.admin do |admin|
+  # ...
+end
 ```
 
 ## Test
 
-```
+```bash
+# Bash script
 export HBASE_JRUBY_TEST_ZK='your-hbaase.domain.net'
-rake test
+
+# Test both for 1.8 and 1.9
+for v in --1.8 --1.9; do
+  export JRUBY_OPTS=$v
+  rake test
+done
 ```
 
 ## Contributing

data/lib/hbase-jruby/dependency.rb

@@ -57,11 +57,13 @@ class HBase
         end
 
       # Load jars
-      jars.select { |jar| File.exists?(jar) && File.extname(jar) == '.jar' }.select do |jar|
+      jars_loaded = jars.select { |jar| File.exists?(jar) && File.extname(jar) == '.jar' }.select do |jar|
         require jar
       end
 
       Util.import_java_classes!
+
+      return jars_loaded
     ensure
       tempfiles.each { |tempfile| tempfile.unlink rescue nil }
     end

data/lib/hbase-jruby/scoped/aggregation.rb

@@ -7,7 +7,8 @@ module Aggregation
     # Enables aggregation support for the table
     # @return [nil]
     def enable_aggregation!
-      add_coprocessor! 'org.apache.hadoop.hbase.coprocessor.AggregateImplementation'
+      cpc = 'org.apache.hadoop.hbase.coprocessor.AggregateImplementation'
+      add_coprocessor! cpc unless has_coprocessor?(cpc)
     end
   end
 

data/lib/hbase-jruby/scoped.rb

@@ -19,14 +19,18 @@ class Scoped
   # @return [Fixnum, Bignum] The number of rows in the scope
   def count
     cnt = 0
-    if block_given?
-      htable.getScanner(filtered_scan).each do |result|
-        cnt += 1 if yield(Result.send(:new, result))
-      end
-    else
-      htable.getScanner(filtered_scan_minimum).each do
-        cnt += 1
+    begin
+      if block_given?
+        scanner = htable.getScanner(filtered_scan)
+        scanner.each do |result|
+          cnt += 1 if yield(Result.send(:new, result))
+        end
+      else
+        scanner = htable.getScanner(filtered_scan_minimum)
+        scanner.each { cnt += 1 }
       end
+    ensure
+      scanner.close if scanner
     end
     cnt
   end
@@ -111,7 +115,7 @@ class Scoped
   #     table.range(:prefix => ['2010', '2012'])
   def range *key_range
     if key_range.last.is_a?(Hash)
-      prefixes  = [*key_range.last[:prefix]]
+      prefixes  = [*key_range.last[:prefix]].compact
       raise ArgumentError,
         "Invalid range. Unknown option(s) specified." unless (key_range.last.keys - [:prefix]).empty?
       key_range = key_range[0...-1]
@@ -328,7 +332,7 @@ private
       end if range
 
       # Prefix filters
-      filters += [*build_prefix_filter]
+      filters += [*build_prefix_filter].compact
 
       # RowFilter must precede the others
       filters += @filters
@@ -365,7 +369,7 @@ private
               CompareFilter::CompareOp::LESS_OR_EQUAL
             when :eq, :==
               CompareFilter::CompareOp::EQUAL
-            when :ne, :!=
+            when :ne # , :!= # Ruby 1.8 compatibility
               CompareFilter::CompareOp::NOT_EQUAL
             else
               raise ArgumentError, "Unknown operator: #{op}"
@@ -375,7 +379,7 @@ private
             # XXX TODO Undocumented feature
             FilterList.new(
               case op
-              when :ne, :!=
+              when :ne # , :!=
                 FilterList::Operator::MUST_PASS_ALL
               else
                 FilterList::Operator::MUST_PASS_ONE
@@ -428,7 +432,7 @@ private
       scan.caching = @caching if @caching
 
       # Filters
-      prefix_filter = [*build_prefix_filter]
+      prefix_filter = [*build_prefix_filter].compact
       filters = prefix_filter + @filters
       filters += process_projection!(scan)
 

data/lib/hbase-jruby/table.rb

@@ -462,11 +462,11 @@ private
   end
 
   def const_shortcut base, v, message
-    vs = v.to_s.upcase.to_sym
-    #if base.constants.map { |c| base.const_get c }.include?(v)
+    vs = v.to_s.upcase
+    # const_get doesn't work with symbols in 1.8 compatibility mode
     if base.constants.map { |c| base.const_get c }.any? { |cv| v == cv }
       v
-    elsif base.constants.include? vs
+    elsif base.constants.map(&:to_s).include?(vs)
       base.const_get vs
     else
       raise ArgumentError, [message, v.to_s].join(': ')

data/lib/hbase-jruby/util.rb

@@ -89,7 +89,7 @@ module Util
         raise ArgumentError, "Column family not specified"
       else
         cf, cq = KeyValue.parseColumn(col.to_s.to_java_bytes)
-        cq = JAVA_BYTE_ARRAY_EMPTY if cq.nil? && col.to_s[-1] == ':'
+        cq = JAVA_BYTE_ARRAY_EMPTY if cq.nil? && col.to_s[-1, 1] == ':'
         return cf, cq
       end
     end
@@ -171,6 +171,8 @@ module Util
           org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter
         ]
       end
+
+      nil
     end
   end
 end#Util

data/lib/hbase-jruby/version.rb

@@ -1,5 +1,5 @@
 class HBase
 module JRuby
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end
 end

data/test/helper.rb

@@ -11,7 +11,10 @@ $LOAD_PATH.unshift File.expand_path('../../lib', __FILE__)
 require "hbase-jruby"
 
 # Required
-HBase.resolve_dependency! 'cdh4.1.2'
+unless HBase.resolve_dependency!('cdh4.1.2').all? { |f| File.exists? f }
+  puts "Invalid return value from HBase.resolve_dependency!"
+  exit 1
+end
 
 class TestHBaseJRubyBase < Test::Unit::TestCase
   TABLE = 'test_hbase_jruby'

data/test/test_aggregation.rb

@@ -3,22 +3,23 @@
 $LOAD_PATH.unshift File.expand_path('..', __FILE__)
 require 'helper'
 
-class TestAggregation < TestHBaseJRubyBase 
+class TestAggregation < TestHBaseJRubyBase
   def test_aggregation
     (1..100).each do |idx|
       @table.put idx, 'cf1:a' => idx, 'cf1:b' => idx * 2
     end
 
-    @table.enable_aggregation!
+    assert_nil @table.enable_aggregation!
+    assert_nil @table.enable_aggregation! # no prob!
 
     lci = org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter.new
     [nil, :fixnum, :int, :integer, lci].each do |ci|
-      assert_equal 100,  @table.project('cf1:a').aggregate(:row_count, *[*ci])
-      assert_equal 5050, @table.project('cf1:a').aggregate(:sum, *[*ci])
-      assert_equal 1,    @table.project('cf1:a').aggregate(:min, *[*ci])
-      assert_equal 100,  @table.project('cf1:a').aggregate(:max, *[*ci])
-      assert_equal 50.5, @table.project('cf1:a').aggregate(:avg, *[*ci])
-      assert_equal 28,   @table.project('cf1:a').aggregate(:std, *[*ci]).to_i # FIXME: 28 or 29?
+      assert_equal 100,  @table.project('cf1:a').aggregate(:row_count, *[*ci].compact)
+      assert_equal 5050, @table.project('cf1:a').aggregate(:sum, *[*ci].compact)
+      assert_equal 1,    @table.project('cf1:a').aggregate(:min, *[*ci].compact)
+      assert_equal 100,  @table.project('cf1:a').aggregate(:max, *[*ci].compact)
+      assert_equal 50.5, @table.project('cf1:a').aggregate(:avg, *[*ci].compact)
+      assert_equal 28,   @table.project('cf1:a').aggregate(:std, *[*ci].compact).to_i # FIXME: 28 or 29?
     end
 
     [%w[cf1:a cf1:b], %w[cf1]].each do |prj|

data/test/test_scoped.rb

@@ -45,8 +45,8 @@ class TestScoped < TestHBaseJRubyBase
 
   def test_invalid_filter
     assert_raise(ArgumentError) { @table.filter(3.14) }
-    assert_raise(ArgumentError) { @table.filter('cf1:a' => { xxx: 50 }) }
-    assert_raise(ArgumentError) { @table.filter('cf1:a' => { eq: { 1 => 2 } }) }
+    assert_raise(ArgumentError) { @table.filter('cf1:a' => { :xxx => 50 }) }
+    assert_raise(ArgumentError) { @table.filter('cf1:a' => { :eq => { 1 => 2 } }) }
   end
 
   def test_each_and_count
@@ -132,13 +132,13 @@ class TestScoped < TestHBaseJRubyBase
       assert_equal 0,  table.filter('cf1:a' => 50, 'cf2:b' => 90...100).to_a.length
       assert_equal 6,  table.filter('cf1:a' => 50..60, 'cf2:b' => 100..110).to_a.length
       assert_equal 10, table.filter('cf1:a' => { :> => 50,  :<= => 60 }).to_a.length
-      assert_equal 9,  table.filter('cf1:a' => { :> => 50,  :<= => 60, :!= => 55 }).to_a.length
-      assert_equal 10, table.filter('cf1:a' => { :>= => 50, :<= => 60, :!= => 55 }).to_a.length
-      assert_equal 9,  table.filter('cf1:a' => { :>= => 50, :< => 60,  :!= => 55 }).to_a.length
+      assert_equal 9,  table.filter('cf1:a' => { :> => 50,  :<= => 60, :ne => 55 }).to_a.length
+      assert_equal 10, table.filter('cf1:a' => { :>= => 50, :<= => 60, :ne => 55 }).to_a.length
+      assert_equal 9,  table.filter('cf1:a' => { :>= => 50, :< => 60,  :ne => 55 }).to_a.length
       assert_equal 1,  table.filter('cf1:a' => { :> => 50,  :<= => 60, :== => 55 }).to_a.length
       assert_equal 2,  table.filter('cf1:a' => { :> => 50,  :<= => 60, :== => [55, 57] }).to_a.length
-      assert_equal 9,  table.filter('cf1:a' => { gte: 50, lt: 60, ne: 55 }).to_a.length
-      assert_equal 7,  table.filter('cf1:a' => { gte: 50, lt: 60, ne: [55, 57, 59] }).to_a.length
+      assert_equal 9,  table.filter('cf1:a' => { :gte => 50, :lt => 60, :ne => 55 }).to_a.length
+      assert_equal 7,  table.filter('cf1:a' => { :gte => 50, :lt => 60, :ne => [55, 57, 59] }).to_a.length
 
       # filter: Hash + additive
       assert_equal 6, table.filter('cf1:a' => 50..60).filter('cf2:b' => 100..110).to_a.length
@@ -149,7 +149,7 @@ class TestScoped < TestHBaseJRubyBase
       assert_equal 3, table.filter(ColumnPaginationFilter.new(3, 1)).first.to_hash.keys.length
 
       # filter: Java filter list
-      import org.apache.hadoop.hbase.filter.FilterList 
+      import org.apache.hadoop.hbase.filter.FilterList
       import org.apache.hadoop.hbase.filter.ColumnRangeFilter
       assert_equal %w[cf2:b cf3:c],
           table.filter(FilterList.new [
@@ -216,7 +216,7 @@ class TestScoped < TestHBaseJRubyBase
 
     hash = @table.get('rowkey').to_hash(
       HBase::ColumnKey('cf1', 1) => :fixnum,
-      HBase::ColumnKey('cf1', 2) => :fixnum,
+      HBase::ColumnKey('cf1', 2) => :fixnum
     )
     assert_equal 1, hash[HBase::ColumnKey(:cf1, 1)]
     assert_equal 2, hash[HBase::ColumnKey(:cf1, 2)]
@@ -325,12 +325,12 @@ class TestScoped < TestHBaseJRubyBase
       @table.put idx, 'cf1:a' => idx % 10, 'cf2:b' => 'Hello'
     end
 
-    assert_equal 20, @table.filter('cf1:a' => { lte: 1 }, 'cf2:b' => 'Hello').count
-    assert_equal 2,  @table.while( 'cf1:a' => { lte: 1 }, 'cf2:b' => 'Hello').count
+    assert_equal 20, @table.filter('cf1:a' => { :lte => 1 }, 'cf2:b' => 'Hello').count
+    assert_equal 2,  @table.while( 'cf1:a' => { :lte => 1 }, 'cf2:b' => 'Hello').count
 
     # while == filter for gets
-    assert_equal 20, @table.filter('cf1:a' => { lte: 1 }, 'cf2:b' => 'Hello').get((0..100).to_a).compact.length
-    assert_equal 20, @table.while( 'cf1:a' => { lte: 1 }, 'cf2:b' => 'Hello').get((0..100).to_a).compact.length
+    assert_equal 20, @table.filter('cf1:a' => { :lte => 1 }, 'cf2:b' => 'Hello').get((0..100).to_a).compact.length
+    assert_equal 20, @table.while( 'cf1:a' => { :lte => 1 }, 'cf2:b' => 'Hello').get((0..100).to_a).compact.length
   end
 
   def test_min_max
@@ -356,7 +356,7 @@ class TestScoped < TestHBaseJRubyBase
     assert_equal 26, @table.filter('cf1:a' => /g$/).count
     assert_equal  2, @table.filter('cf1:a' => /gg|ff/).count
     assert_equal 28, @table.filter('cf1:a' => ['aa', 'cc', /^g/]).count
-    assert_equal 54, @table.filter('cf1:a' => ['aa', 'cc', /^g/, { gte: 'xa', lt: 'y'}]).count
+    assert_equal 54, @table.filter('cf1:a' => ['aa', 'cc', /^g/, { :gte => 'xa', :lt => 'y'}]).count
   end
 end
 

data/test/test_table.rb

@@ -137,7 +137,7 @@ class TestScoped < TestHBaseJRubyBase
       'cf1:c' => 3.14,
       'cf2:d' => :world,
       'cf2:e' => false,
-      'cf3:f' => BigDecimal.new(1234567890123456789012345678901234567890),
+      'cf3:f' => BigDecimal.new('1234567890123456789012345678901234567890'),
       'cf3'   => true
     }
     schema = {

data/test/test_table_admin.rb

@@ -3,7 +3,7 @@
 $LOAD_PATH.unshift File.expand_path('..', __FILE__)
 require 'helper'
 
-class TestTableAdmin < TestHBaseJRubyBase 
+class TestTableAdmin < TestHBaseJRubyBase
   def teardown
     @table.drop! if @table.exists?
   end
@@ -11,7 +11,7 @@ class TestTableAdmin < TestHBaseJRubyBase
   def test_create_table_symbol_string
     t = @hbase.table(:test_hbase_jruby_create_table)
     t.drop! if t.exists?
-    [ :cf, 'cf', :cf => {} ].each do |cf|
+    [ :cf, 'cf', {:cf => {}} ].each do |cf|
       assert_false t.exists?
       t.create! cf
       assert t.exists?

metadata

@@ -2,14 +2,14 @@
 name: hbase-jruby
 version: !ruby/object:Gem::Version
   prerelease:
-  version: 0.1.2
+  version: 0.1.3
 platform: java
 authors:
 - Junegunn Choi
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2012-11-28 00:00:00.000000000 Z
+date: 2012-11-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: test-unit