ocean-dynamo 0.3.10 → 0.3.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f6249788253589b08c22ba6f39f134b14ac986b5
-  data.tar.gz: 1d170f9c05448c3136fe7df37de6f6f43f56666f
+  metadata.gz: d5fb4f095350a4a1cc43debd8405c550277ec7c1
+  data.tar.gz: e1684c9a9481b00b6ba44657076eeda1930e3628
 SHA512:
-  metadata.gz: 84d2847bba2ae68f2c29aa52844081bab0a0e61976b8b9b5fc25cb185f8eb039481418b501ce36db628a8912bfbc95fce488121abe766de45cca39525f5fc433
-  data.tar.gz: 238e095ec0105f540326430af4e0445c02cdf9ee557780e87c6e10284f53aa6e97aa43be57c29747e343b4051a51c1f3bdfef782abea06e60f8164a5a7a3511b
+  metadata.gz: c8673fa49c6af6b931103b669fc1a597e74b68a864354ab26cd5cddf3d1acc509f0d50c171f8deff2c63227b21aaa1eab2875b8684682e6558886d9cf3abc376
+  data.tar.gz: 6be8bab8ca04c7a5e3e3cf4364627fef8a0cac2894d480dda2a12cb31ac72c1b2773cc88fb1ac600e51d2ba6ff3db4a5f92aecc820d6b4f0ff7a44b0b678ff1d

data/README.rdoc

@@ -8,7 +8,7 @@ OceanDynamo requires Ruby 2.0 and Ruby on Rails 4.0.0 or later.
 {<img src="https://badge.fury.io/rb/ocean-dynamo.png" alt="Gem Version" />}[http://badge.fury.io/rb/ocean-dynamo]
 
 
-==== Features
+=== Features
 
 As one important use case for OceanDynamo is to facilitate the conversion of SQL based
 ActiveRecord models to DynamoDB based models, it is important that the syntax and semantics
@@ -19,7 +19,7 @@ is of course based on ActiveModel.
 The attribute and persistence layer of OceanDynamo is modeled on that of ActiveRecord:
 there's +save+, +save!+, +create+, +update+, +update!+, +update_attributes+, +find_each+,
 and all the other methods you're used to. The design goal is always to implement as much 
-as possible of the ActiveRecord interface, without sacrificing scalability. This makes the 
+of the ActiveRecord interface as possible, without compromising scalability. This makes the 
 task of switching from SQL to no-SQL much easier.
 
 Thanks to its structural similarity to ActiveRecord, OceanDynamo works with FactoryGirl.
@@ -28,7 +28,7 @@ To facilitate testing, future versions will keep track of and delete instances a
 OceanDynamo uses primary and secondary indices to retrieve related table items, 
 which means it will scale without limits.
 
-==== Example
+=== Example
 
 The following example shows the syntax. 
 
@@ -93,18 +93,54 @@ controllers. Furthermore, OceanDynamo implements much of the infrastructure of A
 for instance, +read_attribute+, +write_attribute+, and much of the control logic and
 parameters.
 
-At the moment, Model.find only takes a single id. Model.all and Model.count work as
-expected. Model.find_each and Model.find_in_batches are also available.
++Model.all+ and +.count+ work as expected. +Model.find_each+ is also available.
 
-Work has begun on the +has_many+ / +belongs_to+ association which requires only a primary 
-index. This association is even more efficient than its ActiveRecord counterpart.
++belongs_to+ is now operational. The other side, +has_many+, is much simpler and should 
+follow very soon (probably today). 
+
+Restrictions for +belongs_to+ tables: 
+* * The hash key must be specified and must not be +:id+. 
+* * The range key must not be specified at all.
+* * belongs_to can be specified only once in each class.
+
+These restrictions allow OceanDynamo to implement the +has_many+ / +belongs_to+ 
+relation in a very efficient and massively scalable way. 
+
++belongs_to+ claims the range key and uses it to store its own UUID, which normally
+would be stored in the hash key attribute. Instead, the hash key attribute holds the
+UUID of the parent. We have thus reversed the roles of these two fields. As a result, 
+all children have their parent UUID as their hash key, and their own UUID in their 
+range key.
+
+This type of association is even more efficient than its ActiveRecord counterpart as
+it uses only the primary index of the child model for finds and scans in both directions
+of the +has_many+ / +belongs_to+ association.
+
+Furthermore, since DynamoDB has powerful primary index searches involving substrings
+and matching, the fact that the range key is a string can be used to implement
+wildcard matching of additional attributes. This gives, amongst other things, the 
+equivalent of an SQL GROUP BY request, again without requiring any secondary indices.
+
+It's our goal to use a similar technique to implement has_and_belongs_to_many relations, 
+which means that secondary indices won't be necessary for the vast majority of 
+OceanDynamo use cases. This ultimately means reduced operational costs, as well as
+reduced complexity.
+
++.has_belongs_to?+ returns true if the class has a belongs_to association.
+
++.belongs_to_class returns the class of the belongs_to association, or false if none.
+
++.find+ can now take an array arg.
+
++#hash_key+ and +#range_key+ readers added.
 
 
 === Future milestones
 
 After the +has_many+ / +belongs_to+ association, the +has_and_belongs_to_many+ assocation
-will be implemented. This will require secondary indices. This, and other associations will 
-use secondary indices. OceanDynamo will use association proxies to achieve the same kind of
+will be implemented. This will require secondary indices, unlike +has_many+ / +belongs_to+.
+
+OceanDynamo will use association proxies to achieve the same kind of
 interface as ActiveRecord, e.g.: <code>blog_entry.comments.build(body: "Cool!").save!</code>
 
 
@@ -115,7 +151,7 @@ e.g. to implement critical job queues. It will be used increasingly as features
 added to OceanDynamo and will eventually replace all ActiveRecord tables in Ocean.
 
 
-==== Installation
+== Installation
 
  gem install ocean-dynamo
 

data/lib/ocean-dynamo.rb

@@ -9,11 +9,12 @@ require "ocean-dynamo/exceptions"
 require "ocean-dynamo/class_variables"
 require "ocean-dynamo/tables"
 require "ocean-dynamo/schema"
-require "ocean-dynamo/callbacks"
 require "ocean-dynamo/attributes"
-require "ocean-dynamo/queries"
+require "ocean-dynamo/callbacks"
 require "ocean-dynamo/persistence"
-require "ocean-dynamo/associations"
+require "ocean-dynamo/queries"
+
+require "ocean-dynamo/associations/belongs_to"
 
 
 module OceanDynamo

data/lib/ocean-dynamo/associations/belongs_to.rb

@@ -0,0 +1,124 @@
+module OceanDynamo
+  class Base
+
+    def self.belongs_to(target)                        # :master, "master", Master
+      target_attr = target.to_s.underscore             # "master"
+      target_attr_id = "#{target_attr}_id"             # "master_id"
+      target_class = target_attr.camelize.constantize  # Master
+
+      assert_only_one_belongs_to!
+
+      self.table_range_key = table_hash_key            # The RANGE KEY is variable
+      self.table_hash_key = target_attr_id.to_sym      # The HASH KEY is the parent UUID
+
+      attribute table_hash_key, :string                # Define :master_id
+      define_attribute_accessors(table_hash_key)       # Define master_id, master_id=, master_id?
+
+      # Make sure there always is a parent
+      validates(table_hash_key, presence: true)        # Can't save without a parent_id
+
+      # Define the range attribute (our unique UUID)
+      attribute(table_range_key, :string, default: "") # Define :uuid
+      define_attribute_accessors(table_range_key)      # define uuid, uuid=, uuid?
+
+
+
+      # Define the parent id attribute 
+      attribute target_attr_id, :reference, default: nil, target_class: target_class,
+                                association: :belongs_to
+
+      self.class_eval "def #{target_attr}
+                         read_and_maybe_load_pointer('#{target_attr_id}')
+                       end"
+
+      self.class_eval "def #{target_attr}=(value) 
+                         write_attribute('#{target_attr_id}', value) 
+                         @#{target_attr} = value
+                       end"
+
+      self.class_eval "def #{target_attr_id}
+                         get_pointer_id(@#{target_attr})
+                       end"
+
+      self.class_eval "def #{target_attr_id}=(value)
+                         write_attribute('#{target_attr_id}', value) 
+                         @#{target_attr} = value
+                       end"
+      # TODO: "?" methods
+    end
+
+
+    #
+    # Returns true if the class has a belongs_to association.
+    #
+    def self.has_belongs_to?
+      fields[table_hash_key]['association'] == :belongs_to
+    end
+
+
+
+    #
+    # Returns the class of the belongs_to association, or false if none.
+    #
+    def self.belongs_to_class
+      has_belongs_to? && fields[table_hash_key]['target_class']
+    end
+
+
+
+    protected
+
+    #
+    # belongs_to can be specified only once in each model, since we use the range key to
+    # store its UUID and the hash key to store the UUID of the parent, as in
+    # ["parent_uuid", "child_uuid"]. This allows the parent to find all its children 
+    # extremely efficiently by using only the primary index. It also allows the child
+    # to find its parent using only its own hash key. Presto: scalability without any
+    # secondary indices in the has_many/belongs_to association.
+    #
+    # Caveat: the parent must have a simple primary key, not a composite one. It *is*
+    # possible to use a composite key, but then the children must use scans to find
+    # their parents. We could conditionalise this, of course, so that the lookup
+    # strategy is transparent to the user.
+    #
+    def self.assert_only_one_belongs_to!  # :nodoc:
+      if has_belongs_to?
+        raise OceanDynamo::AssociationMustBeUnique, 
+              "#{self} already belongs_to #{belongs_to_class}" 
+      end
+      false
+    end
+
+
+    #
+    # This is run by #initialize and by #assign_attributes to set the
+    # association variables (@master, for instance) and its associated attribute
+    # (such as master_id) to the value given.
+    #
+    def assign_associations(attrs)  # :nodoc: 
+      if attrs && attrs.include?(:master)
+        @master = attrs[:master]
+        write_attribute('master_id', @master)
+      end
+    end
+
+
+    def read_and_maybe_load_pointer(name)  # :nodoc:
+      ptr = read_attribute(name)
+      return nil if ptr.blank?
+      if persisted? && ptr.is_a?(String)
+        parent = fields[name][:target_class].find(ptr, consistent: true)  # TODO: true?
+        write_attribute(name, parent)  # Keep the instance we've just read
+      else
+        ptr
+      end
+    end
+
+
+    def get_pointer_id(ptr)  # :nodoc:
+      return nil if ptr.blank?
+      ptr.is_a?(String) ? ptr : ptr.id
+    end
+
+  end
+end

data/lib/ocean-dynamo/attributes.rb

@@ -11,6 +11,23 @@ module OceanDynamo
     attr_reader :dynamo_item    # :nodoc:
 
 
+    #
+    # Returns the value of the hash key attribute
+    #
+    def hash_key
+      read_attribute(table_hash_key)
+    end
+
+
+    #
+    # Returns the value of the range key attribute or false if the
+    # table doesn't have a range_key.
+    #
+    def range_key
+      table_range_key && read_attribute(table_range_key)
+    end
+
+
     def initialize(attrs={})
       run_callbacks :initialize do
         @attributes = Hash.new
@@ -22,6 +39,7 @@ module OceanDynamo
         @new_record = true
         raise UnknownPrimaryKey unless table_hash_key
       end
+      assign_associations(attrs)
       attrs && attrs.delete_if { |k, v| !fields.has_key?(k) }
       super(attrs)
     end
@@ -89,6 +107,7 @@ module OceanDynamo
     def assign_attributes(values, without_protection: false)
       return if values.blank?
       values = values.stringify_keys
+      assign_associations(values)
       # if values.respond_to?(:permitted?)
       #   unless values.permitted?
       #     raise ActiveModel::ForbiddenAttributesError
@@ -140,7 +159,7 @@ module OceanDynamo
       if respond_to?("#{k}=")
         raise
       else
-        raise UnknownAttributeError, "unknown attribute: #{k}"
+        raise UnknownAttributeError, "unknown attribute: `#{k}'"
       end
     end
 

data/lib/ocean-dynamo/exceptions.rb

@@ -23,7 +23,7 @@ module OceanDynamo
     def initialize(record) # :nodoc:
       @record = record
       errors = @record.errors.full_messages.join(", ")
-      super(I18n.t(:"#{@record.class.i18n_scope}.errors.messages.record_invalid", :errors => errors, :default => :"errors.messages.record_invalid"))
+      super(:"#{@record.class}.errors.messages.record_invalid")
     end
   end
 
@@ -38,7 +38,7 @@ module OceanDynamo
     def initialize(record) # :nodoc:
       @record = record
       errors = @record.errors.full_messages.join(", ")
-      super(I18n.t(:"#{@record.class.i18n_scope}.errors.messages.record_invalid", :errors => errors, :default => :"errors.messages.record_invalid"))
+      super(:"#{@record.class}.errors.messages.record_invalid")
     end
   end
  
@@ -54,4 +54,10 @@ module OceanDynamo
 
   class AssociationTypeMismatch < DynamoError; end
 
+
+  class BelongsToError < DynamoError; end
+    class AssociationMustBeUnique < BelongsToError; end
+    class RangeKeyMustNotBeSpecified < BelongsToError; end
+    class HashKeyMayNotBeNamedId < BelongsToError; end
+
 end

data/lib/ocean-dynamo/persistence.rb

@@ -122,8 +122,13 @@ module OceanDynamo
       run_callbacks :commit do
         run_callbacks :save do
           run_callbacks :create do
-            k = read_attribute(table_hash_key)
-            write_attribute(table_hash_key, SecureRandom.uuid) if k == "" || k == nil
+            # Default the correct hash key to a UUID
+            if self.class.has_belongs_to?
+              write_attribute(table_range_key, SecureRandom.uuid) if range_key.blank?
+            else
+              write_attribute(table_hash_key, SecureRandom.uuid) if hash_key.blank?
+            end
+
             set_timestamps
             dynamo_persist
             true
@@ -171,8 +176,8 @@ module OceanDynamo
 
 
     def reload(**keywords)
-      range_key = table_range_key && attributes[table_range_key]
-      new_instance = self.class.find(id, range_key, **keywords)
+      raise "HELLISHNESS" if id == range_key
+      new_instance = self.class.find(hash_key, range_key, **keywords)
       assign_attributes(new_instance.attributes)
       self
     end
@@ -295,16 +300,15 @@ module OceanDynamo
 
     def serialize_attribute(attribute, value, metadata=fields[attribute],
                             target_class: metadata[:target_class],
-                            type: metadata[:type],
-                            no_save: metadata[:no_save]
+                            type: metadata[:type]
                             )
       return nil if value == nil
+      #value = value.id if value.kind_of?(target_class)
       case type
       when :reference
-        return nil if no_save
         raise DynamoError, ":reference must always have a :target_class" unless target_class
         return value if value.is_a?(String)
-        return value.id if value.is_a?(target_class)
+        return value.id if value.kind_of?(target_class)
         raise AssociationTypeMismatch, "can't save a #{value.class} in a #{target_class} :reference"
       when :string
         return nil if ["", []].include?(value)

data/lib/ocean-dynamo/queries.rb

@@ -2,9 +2,13 @@ module OceanDynamo
   class Base
 
     def self.find(hash, range=nil, consistent: false)
+      return hash.collect {|elem| find elem, range, consistent: consistent } if hash.is_a?(Array)
       _late_connect?
+      hash = hash.id if hash.kind_of?(Base)    # TODO: We have (innocuous) leakage, fix!
       item = dynamo_items[hash, range]
-      raise RecordNotFound, "can't find a #{self} with primary key ['#{hash}', #{range.inspect}]" unless item.exists?
+      unless item.exists?
+        raise RecordNotFound, "can't find a #{self} with primary key ['#{hash}', #{range.inspect}]" 
+      end
       new._setup_from_dynamo(item, consistent: consistent)
     end
 

data/lib/ocean-dynamo/schema.rb

@@ -41,13 +41,7 @@ module OceanDynamo
       attribute(lock_attribute, :integer, default: 0) if locking
       block.call
       # Define attribute accessors
-      fields.each do |name, md| 
-        name = name.to_s
-        # We define accessors even if the name is 'id' (for which we already have methods)
-        self.class_eval "def #{name}; read_attribute('#{name}'); end"
-        self.class_eval "def #{name}=(value); write_attribute('#{name}', value); end"
-        self.class_eval "def #{name}?; read_attribute('#{name}').present?; end"
-      end
+      fields.each { |name, md| define_attribute_accessors(name) }
       # Connect to AWS
       establish_db_connection if connect == true
       # Finally return the full table name
@@ -55,6 +49,14 @@ module OceanDynamo
     end
 
 
+    def self.define_attribute_accessors(name)
+      name = name.to_s
+      self.class_eval "def #{name}; read_attribute('#{name}'); end"
+      self.class_eval "def #{name}=(value); write_attribute('#{name}', value); end"
+      self.class_eval "def #{name}?; read_attribute('#{name}').present?; end"
+    end
+
+
     def self.compute_table_name
       name.pluralize.underscore
     end

data/lib/ocean-dynamo/tables.rb

@@ -43,19 +43,30 @@ module OceanDynamo
 
 
     def self.set_dynamo_table_keys
-      dynamo_table.hash_key = [table_hash_key, fields[table_hash_key][:type]]
+      hash_key_type = fields[table_hash_key][:type]
+      hash_key_type = :string if hash_key_type == :reference
+      dynamo_table.hash_key = [table_hash_key, hash_key_type]
+
       if table_range_key
-        dynamo_table.range_key = [table_range_key, fields[table_range_key][:type]]
+        range_key_type = fields[table_range_key][:type]
+        #range_key_type = :string if range_key_type == :reference
+        dynamo_table.range_key = [table_range_key, range_key_type]
       end
     end
 
 
     def self.create_table
+      hash_key_type = fields[table_hash_key][:type]
+      hash_key_type = :string if hash_key_type == :reference
+
+      range_key_type = table_range_key && fields[table_range_key][:type]
+      #range_key_type = :string if range_key_type == :reference
+
       self.dynamo_table = dynamo_client.tables.create(table_full_name, 
         table_read_capacity_units, table_write_capacity_units,
-        hash_key: { table_hash_key => fields[table_hash_key][:type]},
-        range_key: table_range_key && { table_range_key => fields[table_range_key][:type]}
-        )
+        hash_key: { table_hash_key => hash_key_type},
+        range_key: table_range_key && { table_range_key => range_key_type }
+      )
       sleep 1 until dynamo_table.status == :active
       setup_dynamo
       true

data/lib/ocean-dynamo/version.rb

@@ -1,3 +1,3 @@
 module OceanDynamo
-  VERSION = "0.3.10"
+  VERSION = "0.3.11"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ocean-dynamo
 version: !ruby/object:Gem::Version
-  version: 0.3.10
+  version: 0.3.11
 platform: ruby
 authors:
 - Peter Bengtson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-09-15 00:00:00.000000000 Z
+date: 2013-09-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk
@@ -160,12 +160,12 @@ extensions: []
 extra_rdoc_files: []
 files:
 - config/routes.rb
-- lib/ocean-dynamo/associations.rb
+- lib/ocean-dynamo/associations/belongs_to.rb
+- lib/ocean-dynamo/associations/collection_proxy.rb
 - lib/ocean-dynamo/attributes.rb
 - lib/ocean-dynamo/base.rb
 - lib/ocean-dynamo/callbacks.rb
 - lib/ocean-dynamo/class_variables.rb
-- lib/ocean-dynamo/collection_proxy.rb
 - lib/ocean-dynamo/engine.rb
 - lib/ocean-dynamo/exceptions.rb
 - lib/ocean-dynamo/persistence.rb

data/lib/ocean-dynamo/associations.rb

@@ -1,51 +0,0 @@
-module OceanDynamo
-  class Base
-
-    def self.belongs_to(target)                        # :api_user, "api_user", ApiUser
-      target_attr = target.to_s.underscore             # "api_user"
-      target_attr_id = "#{target_attr}_id"             # "api_user_id"
-      target_class = target_attr.camelize.constantize  # ApiUser
-      attribute target_attr_id, :reference, default: nil, target_class: target_class
-      attribute target_attr,    :reference, default: nil, target_class: target_class, no_save: true
-
-      self.class_eval "def #{target_attr}
-                         read_and_maybe_load_pointer('#{target_attr_id}')
-                       end"
-
-      self.class_eval "def #{target_attr}=(value) 
-                         write_attribute('#{target_attr_id}', value) 
-                         write_attribute('#{target_attr}', value)
-                       end"
-
-      self.class_eval "def #{target_attr_id}
-                         read_pointer_id('#{target_attr}')
-                       end"
-
-      self.class_eval "def #{target_attr_id}=(value)
-                         write_attribute('#{target_attr_id}', value) 
-                         write_attribute('#{target_attr}', value)
-                       end"
-      # TODO: "?" methods
-    end
-
-
-    protected
-
-    def read_and_maybe_load_pointer(name)  # :nodoc:
-      ptr = read_attribute(name)
-      return nil if ptr.blank?
-      if persisted? && ptr.is_a?(String)
-        write_attribute(name, fields[name][:target_class].find(ptr))  # Keep the instance we've just read
-      else
-        ptr
-      end
-    end
-
-    def read_pointer_id(name)  # :nodoc:
-      ptr = read_attribute(name)
-      return nil if ptr.blank?
-      ptr.is_a?(String) ? ptr : ptr.id
-    end
-
-  end
-end