massive_record 0.2.0 → 0.2.1.rc1

data/CHANGELOG.md

@@ -1,7 +1,46 @@
-# v0.2.1 (git develop)
-
-
-# v0.2.0 (git master)
+# v0.2.2 (git develop)
+
+# v0.2.1.rc1 (git master)
+
+- Models without any default_id will now by default get an id via next_id(). You can turn it off
+  via the setting auto_increment_id on ORM::Base or on the Model class itself.
+- record.reload now resets relations.
+- If you have a persisted record and you set one attribute to nil that attribute will be
+  deleted from HBase to represent the nil-value. The fact that the schema of that record
+  class knows of that attribute will be the reason for it to still respond and return nil.
+- Thrift adapter will no longer return UTF-8 encoded strings. The reason for this is that
+  binary representation of integers cannot be set with a UTF-8 encoding; it must be BINARY.
+  It is the client`s responsibility of the adapter to set correct encoding, and the ORM now
+  does this too.
+- If you do have a database where integers are stored as string, you should enable
+  ORM::Base.backward_compatibility_integers_might_be_persisted_as_strings. It will, before for
+  instance atomic_increment! is called, ensure that the integers has been persisted as hex.
+- Based on the previous change we are now able to do real atomic incrementation of integer values.
+  HBase will do the incrementation and guarantee the integrity of that incrementation.
+- Fixnum and Bignum are stored as a 64 bit signed integer representation, not as strings. Fixnum and bignum
+  values are no longer encoded by the ORM; that responsibility has been moved down to the adapter.
+- We can now, by setting Base.check_record_uniqueness_on_create to true, do a quick and simple (and
+  kinda insecure) check if the record id exists on a new_record when doing a create call. Just a simple sanity check.
+- We now have mass assignment of attributes, just like ActiveRecord. It uses the same module,
+  so attr_accessible and attr_protected methods are available. By default the id and inheritable attribute
+  are protected. If you where doing Person.new(:id => 'ID') you might want to change this now as it will no
+  longer work.
+- Ids can now be assigned on new/create/create! with: Person.new("id", attributes). Person.new(id: "id") will
+  is soon to be disallowed.
+- Polymorphic type attribute is no longer called underscore on. Should be backwards compatible when finding records.
+- Time can now be time zone aware. Times are being presented in the Time.zone, and persisted 
+  in UTC. You enable it with MassiveRecord::ORM::Base.time_zone_aware_attributes = true.
+  In a Rails application this will be set to true by default.
+- No longer checking if table exist on a find. Instead we rescue error the few times a table does not exist,
+  and create it based on current schema. This gained a lot of speed on our production server.
+- Changed the way Thrift::Table executes find(). It is now doing the work ~3 times faster.
+- In Rails, we are now included in the 200-ok-completed-log like: Completed 200 OK in 798ms (Views: 277.5ms | MassiveRecord: 9.2ms)
+- Subscribed to events from query instruments and printing out time spent in database per call.
+- Added ActiveSupport Notifications instruments around database query calls in the ORM level.
+- Removed inclusion of ActiveModel::Translation in ORM::Base class, as including ActiveModel::Validations
+  extends ORM::Base with Translation as well (and it should never have been included; it should have been extended with..)
+
+# v0.2.0
 
 - Intersection and union operations on arrays containing MassiveRecord objects is now working as expected.
 - You can now disallow nil values, and in that case we will ensure that given field has its default value.

data/Gemfile.lock

@@ -1,10 +1,11 @@
 PATH
   remote: .
   specs:
-    massive_record (0.2.0)
+    massive_record (0.2.1.rc1)
       activemodel
       activesupport
       thrift (>= 0.5.0)
+      tzinfo
 
 GEM
   remote: http://rubygems.org/
@@ -26,6 +27,7 @@ GEM
       diff-lcs (~> 1.1.2)
     rspec-mocks (2.5.0)
     thrift (0.6.0)
+    tzinfo (0.3.27)
 
 PLATFORMS
   ruby

data/README.md

@@ -59,6 +59,9 @@ Both MassiveRecord::ORM::Table and MassiveRecord::ORM::Column do now have some f
     - Timestamps like created_at and updated_at. Updated at will always be available, created_at must be defined. See example down:
     - Finder scopes. Like: Person.select(:only_columns_from_this_family).limit(10).collect(&:name)
     - Ability to set a default scope.
+    - Time zone aware time attributes.
+    - Basic instrumentation and logging of query times.
+    - Attribute mass assignment security.
 
 Tables also have:
     - Persistencey method calls like create, save and destroy (but they do not actually save things to hbase)
@@ -98,6 +101,7 @@ Here is an example of usage, both for Table and Column:
         field :with_a_lot_of_uninteresting_data
       end
 
+      attr_accessible :name, :email, :phone_number, :date_of_birth
 
       validates_presence_of :name, :email
       validates_format_of :email, :with => /\A([^@\s]+)@((?:[-a-z0-9]+\.)+[a-z]{2,})\Z/i
@@ -122,6 +126,7 @@ Here is an example of usage, both for Table and Column:
       field :nice_place, :boolean, :default => true
     end
 
+You can find a small example application here: https://github.com/thhermansen/massive_record_test_app
 
 ### Related gems
 

data/lib/massive_record/adapters/thrift/connection.rb

@@ -1,3 +1,5 @@
+require 'active_support/notifications'
+
 module MassiveRecord
   module Adapters
     module Thrift
@@ -9,21 +11,30 @@ module MassiveRecord
           @timeout = 4000
           @host    = opts[:host]
           @port    = opts[:port] || 9090
+          @instrumenter = ActiveSupport::Notifications.instrumenter
         end
       
         def transport
           @transport ||= ::Thrift::BufferedTransport.new(::Thrift::Socket.new(@host, @port, @timeout))
         end
       
-        def open
-          protocol = ::Thrift::BinaryProtocol.new(transport)
-          @client = Apache::Hadoop::Hbase::Thrift::Hbase::Client.new(protocol)
-        
-          begin
-            transport.open()
-            true
-          rescue
-            raise MassiveRecord::Wrapper::Errors::ConnectionException.new, "Unable to connect to HBase on #{@host}, port #{@port}"
+        def open(options = {})
+          options = options.merge({
+            :adapter => 'Thrift',
+            :host => @host,
+            :port => @port
+          })
+
+          @instrumenter.instrument "adapter_connecting.massive_record", options do
+            protocol = ::Thrift::BinaryProtocol.new(transport)
+            @client = Apache::Hadoop::Hbase::Thrift::Hbase::Client.new(protocol)
+
+            begin
+              transport.open()
+              true
+            rescue
+              raise MassiveRecord::Wrapper::Errors::ConnectionException.new, "Unable to connect to HBase on #{@host}, port #{@port}"
+            end
           end
         end
       
@@ -55,14 +66,10 @@ module MassiveRecord
           begin
             open if not @client
             client.send(method, *args) if @client
-          rescue IOError
-            @client = nil
-            open
-            client.send(method, *args) if @client
-          rescue ::Thrift::TransportException
+          rescue ::Thrift::TransportException => error
             @transport = nil
             @client = nil
-            open
+            open(:reconnecting => true, :reason => error.class)
             client.send(method, *args) if @client
           end
         end
@@ -70,4 +77,4 @@ module MassiveRecord
       end
     end
   end
-end
+end

data/lib/massive_record/adapters/thrift/row.rb

@@ -69,46 +69,26 @@ module MassiveRecord
           mutations = []
       
           @columns.each do |column_name, cell|
-            m        = Apache::Hadoop::Hbase::Thrift::Mutation.new
-            m.column = column_name
-            m.value  = cell.value_to_thrift
-        
-            mutations.push(m)
+            mutations << Apache::Hadoop::Hbase::Thrift::Mutation.new(:column => column_name).tap do |mutation|
+              if new_value = cell.value_to_thrift
+                mutation.value = new_value
+              else
+                mutation.isDelete = true
+              end
+            end
           end
 
           @table.client.mutateRow(@table.name, id.to_s.dup.force_encoding(Encoding::BINARY), mutations).nil?
         end
 
 
-        #
-        # FIXME
-        # 
-        # The thrift wrapper is only working with strings as far as I can see,
-        # and the atomicIncrement call on strings kinda doesn't make sense on strings
-        #
-        # For now I'll implement this without atomicIncrement, to get the behaviour we want.
-        # Guess this in time will either be fixed or raised an not-supported-error. If the
-        # latter is the case I guess we'll need to shift over to a jruby adapter and use the
-        # java api instead of thrift.
-        #
+        
         def atomic_increment(column_name, by = 1)
-          # @table.client.atomicIncrement(@table.name, id.to_s, column_name, by) 
-          value_to_increment = @columns[column_name.to_s].value
-
-          raise "Value to increment (#{value_to_increment}) doesnt seem to be a number!" unless value_to_increment =~ /^\d+$/
-          raise "Argument by must be an integer" unless by.is_a? Fixnum
-
-          value_to_increment = value_to_increment.to_i
-          value_to_increment += by
-          value_to_increment = value_to_increment.to_s
-
-          mutation = Apache::Hadoop::Hbase::Thrift::Mutation.new
-          mutation.column = column_name
-          mutation.value = value_to_increment
-
-          if @table.client.mutateRow(@table.name, id.to_s, [mutation]).nil?
-            value_to_increment
-          end
+          @table.client.atomicIncrement(@table.name, id.to_s, column_name, by) 
+        end
+        
+        def read_atomic_integer_value(column_name)
+          atomic_increment(column_name, 0)
         end
     
         def self.populate_from_trow_result(result, connection, table_name, column_families = [])

data/lib/massive_record/adapters/thrift/table.rb

@@ -42,6 +42,7 @@ module MassiveRecord
     
         def destroy
           disable
+          @table_exists = false
           client.deleteTable(name).nil?
         end
     
@@ -114,22 +115,35 @@ module MassiveRecord
         # table.get("my_id", :info, :name) # => "Bob"
         #
         def get(id, column_family_name, column_name)
-          MassiveRecord::Wrapper::Cell.new(:value => connection.get(name, id, "#{column_family_name.to_s}:#{column_name.to_s}").first.value).value
+          if value = connection.get(name, id, "#{column_family_name.to_s}:#{column_name.to_s}").first.try(:value)
+            MassiveRecord::Wrapper::Cell.new(:value => value).value # might seems a bit strange.. Just to "enforice" that the value is a supported type
+          end
         end
         
+        #
+        # Finds one or multiple ids
+        #
+        # Returns nil if id is not found
+        #
         def find(*args)
-          arg  = args[0]
-          opts = args[1] || {}
-          if arg.is_a?(Array)
-            arg.collect{|id| first(opts.merge(:start => id))}
+          what_to_find = args.first
+          options = args.extract_options!.symbolize_keys
+
+          if what_to_find.is_a?(Array)
+            what_to_find.collect { |id| find(id, options) }
           else
-            # need to replace by connection.getRowWithColumns("companies_development", "NO0000000812676342", ["info:name", "info:org_num"]).first
-            first(opts.merge(:start => arg))
+            if column_families_to_find = options[:select]
+              column_families_to_find = column_families_to_find.collect { |c| c.to_s }
+            end
+
+            if t_row_result = connection.getRowWithColumns(name, what_to_find, column_families_to_find).first
+              Row.populate_from_trow_result(t_row_result, connection, name, column_families_to_find)
+            end
           end
         end
-    
+
         def find_in_batches(opts = {})        
-          results_limit = opts.delete(:limit)
+          results_limit = opts[:limit]
           results_found = 0
           
           scanner(opts) do |s|
@@ -148,7 +162,7 @@ module MassiveRecord
         end
     
         def exists?
-          connection.tables.include?(name)
+          @table_exists ||= connection.tables.include?(name)
         end
     
         def regions

data/lib/massive_record/orm/attribute_methods.rb

@@ -1,12 +1,31 @@
+unless ActiveModel::AttributeMethods.const_defined? 'COMPILABLE_REGEXP'
+  ActiveModel::AttributeMethods::COMPILABLE_REGEXP = /\A[a-zA-Z_]\w*[!?=]?\Z/
+end
+
 module MassiveRecord
   module ORM
     module AttributeMethods
       extend ActiveSupport::Concern
       include ActiveModel::AttributeMethods      
+      include ActiveModel::MassAssignmentSecurity
 
       module ClassMethods
         def define_attribute_methods
           super(known_attribute_names)
+          @attribute_methods_generated = true
+        end
+
+        def attribute_methods_generated?
+          @attribute_methods_generated ||= false
+        end
+
+        def undefine_attribute_methods(*args)
+          super
+          @attribute_methods_generated = false
+        end
+
+        def attributes_protected_by_default
+          ['id', inheritance_attribute]
         end
       end
 
@@ -18,7 +37,7 @@ module MassiveRecord
       def attributes=(new_attributes)
         return unless new_attributes.is_a?(Hash)
 
-        new_attributes.each do |attr, value|
+        sanitize_for_mass_assignment(new_attributes).each do |attr, value|
           writer_method = "#{attr}="
           if respond_to? writer_method
             send(writer_method, value)
@@ -43,6 +62,13 @@ module MassiveRecord
         super
       end
 
+
+      protected
+
+      def attribute_method?(attr_name)
+        attr_name == 'id' || (defined?(@attributes) && @attributes.include?(attr_name))
+      end
+
       private
 
       def attributes_raw=(new_attributes)

data/lib/massive_record/orm/attribute_methods/dirty.rb

@@ -62,11 +62,11 @@ module MassiveRecord
         end
 
         def will_change_attribute?(attr_name, value)
-          read_attribute(attr_name) != value
+          read_attribute(attr_name) != decode_attribute(attr_name, value)
         end
 
         def will_change_back_to_original_value?(attr_name, value)
-          original_attribute_value(attr_name) == value
+          original_attribute_value(attr_name) == decode_attribute(attr_name, value)
         end
 
         def clear_dirty_states!

data/lib/massive_record/orm/attribute_methods/read.rb

@@ -8,9 +8,40 @@ module MassiveRecord
           attribute_method_suffix ""
         end
         
+
+        module ClassMethods
+          protected
+
+          def define_method_attribute(attr_name)
+            internal_read_method = "_#{attr_name}"
+
+            if attr_name =~ ActiveModel::AttributeMethods::COMPILABLE_REGEXP
+              generated_attribute_methods.module_eval <<-RUBY, __FILE__, __LINE__
+                def #{internal_read_method}
+                  decode_attribute('#{attr_name}', @attributes['#{attr_name}'])
+                end
+
+                alias #{attr_name} #{internal_read_method}
+              RUBY
+            else
+              generated_attribute_methods.send(:define_method, internal_read_method) do
+                decode_attribute(attr_name, @attributes[attr_name])
+              end
+              alias_method "#{attr_name}", "#{internal_read_method}"
+            end
+          end
+        end
+
+
         def read_attribute(attr_name)
           attr_name = attr_name.to_s
-          attributes_schema[attr_name].nil? ? @attributes[attr_name] : attributes_schema[attr_name].decode(@attributes[attr_name])
+          internal_read_method = "_#{attr_name}"
+
+          if respond_to? internal_read_method
+            send(internal_read_method)
+          else
+            decode_attribute(attr_name, @attributes[attr_name])
+          end
         end
 
         private
@@ -18,6 +49,10 @@ module MassiveRecord
         def attribute(attr_name)
           read_attribute(attr_name)
         end
+
+        def decode_attribute(attr_name, value)
+          attributes_schema[attr_name].nil? ? value : attributes_schema[attr_name].decode(value)
+        end
       end
     end
   end

data/lib/massive_record/orm/attribute_methods/time_zone_conversion.rb

@@ -0,0 +1,81 @@
+module MassiveRecord
+  module ORM
+    module AttributeMethods
+
+      #
+      # Module for handling time zones
+      #
+      # Related attributes and configurations:
+      #   Base.default_timezone                         
+      #     is the zone persisted in the database
+      #   Base.time_zone_aware_attributes
+      #     is flag for disable / enable it altogether
+      #   Base.skip_time_zone_conversion_for_attributes
+      #     makes it possible to skip specific conversions on given attributes
+      #
+      #
+      module TimeZoneConversion
+        extend ActiveSupport::Concern
+
+        included do
+          # Determines whether to use Time.local (using :local) or Time.utc (using :utc)
+          # when pulling dates and times from the database. This is set to :local by default.
+          cattr_accessor :default_timezone, :instance_writer => false
+          self.default_timezone = :local
+
+          cattr_accessor :time_zone_aware_attributes, :instance_writer => false
+          self.time_zone_aware_attributes = false
+
+          class_attribute :skip_time_zone_conversion_for_attributes, :instance_writer => false
+          self.skip_time_zone_conversion_for_attributes = []
+        end
+
+
+
+        module ClassMethods
+          protected
+
+          # Redefine reader method if we are to do time zone configuration on field
+          def define_method_attribute(attr_name)
+            if time_zone_conversion_on_field?(attributes_schema[attr_name])
+              internal_read_method = "_#{attr_name}"
+
+              if attr_name =~ ActiveModel::AttributeMethods::COMPILABLE_REGEXP
+                generated_attribute_methods.module_eval <<-RUBY, __FILE__, __LINE__
+                  def #{internal_read_method}
+                    if time = decode_attribute('#{attr_name}', @attributes['#{attr_name}'])
+                      time.in_time_zone
+                    end
+                  end
+
+                  alias #{attr_name} #{internal_read_method}
+                RUBY
+              else
+                generated_attribute_methods.send(:define_method, internal_read_method) do
+                  if time = decode_attribute(attr_name, @attributes[attr_name])
+                    time.in_time_zone
+                  end
+                end
+              end
+            else
+              super
+            end
+          end
+
+          # Redefine writer method if we are to do time zone configuration on field
+          def define_method_attribute=(attr_name)
+            # Nothing special goes on here, at the moment
+            super
+          end
+
+
+          private
+
+          def time_zone_conversion_on_field?(field)
+            field && time_zone_aware_attributes && !skip_time_zone_conversion_for_attributes.include?(field.name) && field.type == :time
+          end
+        end
+      end
+    end
+  end
+end