massive_record 0.1.1 → 0.2.0.beta

data/CHANGELOG.md

@@ -1,10 +1,34 @@
 # v0.2.0 (git develop)
 
 
-# v0.1.2 (git master)
-
-
-
+# v0.2.0.beta (git master)
+
+- ORM will now take care of serialize and de-serialize of attributes like arrays, hashes etc. It is doing so
+  based on the type of your fields. You can select either JSON or YAML serialization for your data. As a default it
+  will use JSON. You can also, by chaining multiple coders together add support for multiple serialization types
+  when reading data from the database.
+- Thrift-adapter will no longer auto-serialize objects likes hashes and arrays. Its vlaues must now be strings, and it
+  will only take care of encoding/decoding it to and from what Thrift expects (binary encoding).
+- Compare Person === proxy_targeting_a_person will now be true. Makes case-when-constructions doable.
+- Single table inheritance is supported. By default you can have an attribute called type to give you support for it in a table.
+- A default_scope is possible to set on classes. For instance: Calling default_scope select(:only_this_column_family)
+  inside of a class will execute finder operations with this as default scope. If you need to fetch records of class
+  without your preset default scope you can use Model.unscoped.
+- We now have some ActiveRecord like chaining of method calls when we do find-operations. Like Person.select(:column_family).limit(2)
+  is the same as Person.all(:select => ['column_family', :limit => 2])
+- references_many has first() and limit() which uses the target array if loaded, or load only what it needs from the database.
+- Wrapper::Thrift has been moved into Adapter::Thrift. Adding more adapters should be not that hard now.
+- References many is now possible. We have to strategies: Store an array of foreign keys in the proxy_owner,
+  or supply a ids-starts-with and open up a scanner and read from that point.
+- Setting a non-parsable value on date/time field will no longer raise an error.
+- Scanner no longer fetches with a limit of 10 by default. It is set to 100000000.
+- References one relations support polymorphic relations.
+- Simple implementation of references_one relation. This is where you have a foreign key you will look up in a different table.
+
+
+# v0.1.2
+- Fixed, or at least made better, the is_yaml? method in Wrapper::Cell.This functionality of serialize/de-serialize
+  should be moved up into the ORM asap, but for now a hot fix has been applied.
 
 
 # v0.1.1
@@ -18,7 +42,6 @@
 - Bugfix: Database cleaner no longer tries to remove tables with same name twice.
 
 
-
 # v0.1.0
 
 - Communication with Hbase via Thrift.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    massive_record (0.1.1)
+    massive_record (0.2.0.beta)
       activemodel
       activesupport
       thrift (>= 0.5.0)
@@ -9,23 +9,23 @@ PATH
 GEM
   remote: http://rubygems.org/
   specs:
-    activemodel (3.0.3)
-      activesupport (= 3.0.3)
+    activemodel (3.0.5)
+      activesupport (= 3.0.5)
       builder (~> 2.1.2)
       i18n (~> 0.4)
-    activesupport (3.0.3)
+    activesupport (3.0.5)
     builder (2.1.2)
     diff-lcs (1.1.2)
     i18n (0.5.0)
-    rspec (2.4.0)
-      rspec-core (~> 2.4.0)
-      rspec-expectations (~> 2.4.0)
-      rspec-mocks (~> 2.4.0)
-    rspec-core (2.4.0)
-    rspec-expectations (2.4.0)
+    rspec (2.5.0)
+      rspec-core (~> 2.5.0)
+      rspec-expectations (~> 2.5.0)
+      rspec-mocks (~> 2.5.0)
+    rspec-core (2.5.1)
+    rspec-expectations (2.5.0)
       diff-lcs (~> 1.1.2)
-    rspec-mocks (2.4.0)
-    thrift (0.5.0)
+    rspec-mocks (2.5.0)
+    thrift (0.6.0)
 
 PLATFORMS
   ruby

data/README.md

@@ -54,6 +54,8 @@ Both MassiveRecord::ORM::Table and MassiveRecord::ORM::Column do now have some f
     - Casting of attributes
     - Serialization of array / hashes
     - Timestamps like created_at and updated_at. Updated at will always be available, created_at must be defined. See example down:
+    - Finder scopes. Like: Person.select(:only_columns_from_this_family).limit(10).collect(&:name)
+    - Ability to set a default scope.
 
 Tables also have:
     - Persistencey method calls like create, save and destroy (but they do not actually save things to hbase)
@@ -63,11 +65,19 @@ Tables also have:
     - Save / update methods
     - Auto-creation of table and column families on save if table does not exists.
     - Destroy records
+    - Relations: See MassiveRecord::ORM::Relations::Interface ClassMethods for documentation
 
 
 Here is an example of usage, both for Table and Column:
 
     class Person < MassiveRecord::ORM::Table
+      references_one :boss, :class_name => "Person", :store_in => :info
+      references_one :attachment, :polymorphic => true
+      references_many :friends, :store_in => :info
+      references_many :cars, :records_starts_from => :cars_start_id
+
+      default_scope select(:info)
+
       column_family :info do
         field :name
         field :email
@@ -75,12 +85,29 @@ Here is an example of usage, both for Table and Column:
         field :points, :integer, :default => 0
         field :date_of_birth, :date
         field :newsletter, :boolean, :default => false
+        field :type # Used for single table inheritance
 
         timestamps # ..or field :created_at, :time
       end
 
+      column_family :misc do
+        field :with_a_lot_of_uninteresting_data
+      end
+
+
       validates_presence_of :name, :email
       validates_format_of :email, :with => /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\Z/i
+
+      # Returns the id the scanner should start from in the Car table
+      # to fetch cars related to this person
+      def cars_start_id
+        id+'-'
+      end
+    end
+
+    class Friend < Person
+      # This one will be stored in Person's table with it's type set to Friend.
+      # Calling Person.all will return object back as a Friend.
     end
 
     
@@ -148,7 +175,8 @@ You can, if you'd like, work directly against the adapter.
 ## Planned work
 
 - Rename Wrapper to Adapter, and make it easy to switch from Thrift to another way of communicating with Hbase.
-- Associations and embedded objects.
+- Embedded objects.
+- Cache the decoded values of attributes, not use the value_is_already_decoded?. This will fix possible problem with YAML as coder backend.
 - Implement other Adapters, for instance using jruby and the Java API.
 
 

data/lib/massive_record/adapters/initialize.rb

@@ -0,0 +1,18 @@
+module MassiveRecord
+  def self.adapter=(name)
+    @adapter = name
+  end
+  
+  def self.adapter
+    @adapter
+  end
+end
+
+# Default adapter is set to thrift
+MassiveRecord.adapter = :thrift
+
+# Check the adapter is valid
+raise "The adapter can only be 'thrift'." unless [:thrift].include?(MassiveRecord.adapter)
+
+# Load specific adapter
+require "massive_record/adapters/#{MassiveRecord.adapter}/adapter"

data/lib/massive_record/adapters/thrift/adapter.rb

@@ -0,0 +1,25 @@
+module MassiveRecord
+  module Adapters
+    module Thrift
+    end
+  end
+end
+
+ADAPTER = MassiveRecord::Adapters::Thrift
+
+# Thrift Gems
+require 'thrift'
+require 'thrift/transport/socket'
+require 'thrift/protocol/binary_protocol'
+
+# Generated Ruby classes from Thrift for HBase
+require 'massive_record/adapters/thrift/hbase/hbase_constants'
+require 'massive_record/adapters/thrift/hbase/hbase_types'
+require 'massive_record/adapters/thrift/hbase/hbase'
+
+# Adapter
+require 'massive_record/adapters/thrift/column_family'
+require 'massive_record/adapters/thrift/connection'
+require 'massive_record/adapters/thrift/row'
+require 'massive_record/adapters/thrift/scanner'
+require 'massive_record/adapters/thrift/table'

data/lib/massive_record/adapters/thrift/column_family.rb

@@ -0,0 +1,24 @@
+module MassiveRecord
+  module Adapters
+    module Thrift
+      class ColumnFamily
+      
+        attr_accessor :name, :max_versions, :columns
+    
+        def initialize(column_name, opts = {})
+          @name = column_name
+          @max_versions = opts[:max_versions] || 10
+          @columns = opts[:columns] || []
+        end
+    
+        def descriptor
+          Apache::Hadoop::Hbase::Thrift::ColumnDescriptor.new do |col|
+            col.name = "#{name}:"
+            col.maxVersions = max_versions
+          end
+        end
+    
+      end
+    end
+  end
+end

data/lib/massive_record/adapters/thrift/connection.rb

@@ -0,0 +1,73 @@
+module MassiveRecord
+  module Adapters
+    module Thrift
+      class Connection
+  
+        attr_accessor :host, :port, :timeout
+    
+        def initialize(opts = {})
+          @timeout = 4000
+          @host    = opts[:host]
+          @port    = opts[:port] || 9090
+        end
+      
+        def transport
+          @transport ||= ::Thrift::BufferedTransport.new(::Thrift::Socket.new(@host, @port, @timeout))
+        end
+      
+        def open
+          protocol = ::Thrift::BinaryProtocol.new(transport)
+          @client = Apache::Hadoop::Hbase::Thrift::Hbase::Client.new(protocol)
+        
+          begin
+            transport.open()
+            true
+          rescue
+            raise MassiveRecord::Wrapper::Errors::ConnectionException.new, "Unable to connect to HBase on #{@host}, port #{@port}"
+          end
+        end
+      
+        def close
+          @transport.close.nil?
+        end
+          
+        def client
+          @client
+        end
+      
+        def open?
+          @transport.try("open?")
+        end
+      
+        def tables
+          collection = MassiveRecord::Wrapper::TablesCollection.new
+          collection.connection = self
+          getTableNames().each{|table_name| collection.push(table_name)}
+          collection
+        end
+    
+        def load_table(table_name)
+          MassiveRecord::Wrapper::Table.new(self, table_name)
+        end
+    
+        # Wrapp HBase API to be able to catch errors and try reconnect
+        def method_missing(method, *args)
+          begin
+            open if not @client
+            client.send(method, *args) if @client
+          rescue IOError
+            @client = nil
+            open
+            client.send(method, *args) if @client
+          rescue ::Thrift::TransportException
+            @transport = nil
+            @client = nil
+            open
+            client.send(method, *args) if @client
+          end
+        end
+    
+      end
+    end
+  end
+end

data/lib/massive_record/adapters/thrift/row.rb

@@ -0,0 +1,150 @@
+module MassiveRecord
+  module Adapters
+    module Thrift
+      class Row
+    
+        attr_accessor :id, :column_families, :columns, :new_record, :table
+    
+        def initialize(opts = {})
+          @id              = opts[:id]
+          self.values      = opts[:values] || {}
+          @table           = opts[:table]
+          @column_families = opts[:column_families] || []
+          @columns         = opts[:columns] || {}
+          @new_record      = true
+        end
+      
+        def column_names
+          columns.keys
+        end
+    
+        def fetch_all_column_families
+          @table.fetch_column_family
+          fetch_column_families(@table.column_family_names)
+        end
+    
+        def fetch_column_families(list)
+          @column_families = table.column_families.collect do |column_name, description|
+             MassiveRecord::Wrapper::ColumnFamily.new(column_name.split(":").first, {
+              :row          => self,
+              :name         => description.name,
+              :max_versions => description.maxVersions,
+              :compression  => description.compression,
+              :in_memory    => description.inMemory
+              # bloomFilterType, bloomFilterVectorSize, bloomFilterNbHashes, blockCacheEnabled, timeToLive
+            })
+          end
+        end
+    
+        # = Parse columns / cells and create a Hash from them
+        def values
+          @columns.inject({"id" => id}) {|h, (column_name, cell)| h[column_name] = cell.value; h}
+        end
+
+        def values=(data)
+          @values = {}
+          update_columns(data)
+        end
+    
+        def update_columns(data = {})
+          data.each do |column_family_name, columns|
+            columns.each do |column_name, values|
+              update_column(column_family_name, column_name, values)
+            end
+          end
+        end
+    
+        def update_column(column_family_name, column_name, value)
+          column = "#{column_family_name}:#{column_name}"
+      
+          if @columns[column].nil?
+            @columns[column] =  MassiveRecord::Wrapper::Cell.new({:value =>  value, :created_at => Time.now})
+          else
+            @columns[column].value = value
+          end
+        end
+    
+        # = Parse columns cells and save them
+        def save
+          mutations = []
+      
+          @columns.each do |column_name, cell|
+            m        = Apache::Hadoop::Hbase::Thrift::Mutation.new
+            m.column = column_name
+            m.value  = cell.value_to_thrift
+        
+            mutations.push(m)
+          end
+
+          @table.client.mutateRow(@table.name, id.to_s, mutations).nil?
+        end
+
+
+        #
+        # FIXME
+        # 
+        # The thrift wrapper is only working with strings as far as I can see,
+        # and the atomicIncrement call on strings kinda doesn't make sense on strings
+        #
+        # For now I'll implement this without atomicIncrement, to get the behaviour we want.
+        # Guess this in time will either be fixed or raised an not-supported-error. If the
+        # latter is the case I guess we'll need to shift over to a jruby adapter and use the
+        # java api instead of thrift.
+        #
+        def atomic_increment(column_name, by = 1)
+          # @table.client.atomicIncrement(@table.name, id.to_s, column_name, by) 
+          value_to_increment = @columns[column_name.to_s].value
+
+          raise "Value to increment (#{value_to_increment}) doesnt seem to be a number!" unless value_to_increment =~ /^\d+$/
+          raise "Argument by must be an integer" unless by.is_a? Fixnum
+
+          value_to_increment = value_to_increment.to_i
+          value_to_increment += by
+          value_to_increment = value_to_increment.to_s
+
+          mutation = Apache::Hadoop::Hbase::Thrift::Mutation.new
+          mutation.column = column_name
+          mutation.value = value_to_increment
+
+          if @table.client.mutateRow(@table.name, id.to_s, [mutation]).nil?
+            value_to_increment
+          end
+        end
+    
+        def self.populate_from_trow_result(result, connection, table_name, column_families = [])
+          row                 = self.new
+          row.id              = result.row
+          row.new_record      = false
+          row.table           = Table.new(connection, table_name)
+          row.column_families = column_families
+
+          result.columns.each do |name, value|
+            row.columns[name] =  MassiveRecord::Wrapper::Cell.new({
+              :value      => value.value,
+              :created_at => Time.at(value.timestamp / 1000, (value.timestamp % 1000) * 1000)
+            })
+          end
+      
+          row
+        end
+    
+        def destroy
+          @table.client.deleteAllRow(@table.name, @id).nil?
+        end
+    
+        def new_record?
+          @new_record
+        end
+    
+        def prev
+          self
+        end
+    
+        def updated_at
+          columns.values.collect(&:created_at).max
+        end
+        
+      end
+    end  
+  end
+end