ocean-dynamo 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5a5d6a93318aea2c36029d1f40d3496ad5cc16a1
-  data.tar.gz: 17f2f4d1edf4885662aea89e61948924d7e2bfdb
+  metadata.gz: 0a277c429cc29e02d55452ba4bca425a2b330cd4
+  data.tar.gz: d21e7ae3f0dcd90e5cf993bc43f5ccb4e6148e26
 SHA512:
-  metadata.gz: d3f9aff5b1b2f3402946acb6c5806b32582e3baeefa758ea4d35183386cde1bb937e88204a8feb8fddabc107b16c13540c334bd0ddf04ce91b39822e9a3c6284
-  data.tar.gz: 1cfb33f43e6888a7f7f7417900baf99b9d777ef2797a52d96f5af36e090bedd46f2543210f62237f7e4db547570a67094cb1bd3e094ec13e36671b11154fe1eb
+  metadata.gz: eac3188b5e301a633ee829651adae2c9f228f824e35d3ffb798abb3ff6e382ca76b9ed0e985957aebdd83207875608fc2e31843cbe25cf8547988a3a44dc4885
+  data.tar.gz: 42b49827d66a23bb4a65ec9ed55c24ad25a34fdebca03b61648b6615aed753c5339e737eca41e6f58ab3057d381defc3887946df00cfb632350dedfbcf853c98

data/README.rdoc

@@ -33,7 +33,7 @@ with FactoryGirl.
 
 === Current State
 
-* Local secondary indices are now supported! See below for more information.
+* Secondary indices are now supported! See below for more information.
 * Version 2 of the AWS Ruby SDK is now used. This required an internal reorganisation, 
   but it also gives us access to local and global secondary indices.
 * Work begun on association proxies, etc.
@@ -76,9 +76,9 @@ will not be considered unless all tests pass and coverage is equally high or hig
 All contributed code must therefore also be exhaustively tested.
 
 
-=== Examples
+== Examples
 
-==== Basic syntax
+=== Basic syntax
 
 The following example shows the basic syntax for declaring a DynamoDB-based schema. 
 
@@ -104,7 +104,7 @@ The following example shows the basic syntax for declaring a DynamoDB-based sche
     
   end
 
-==== Attributes
+=== Attributes
 
 Each attribute has a name, a type (+:string+, +:integer+, +:float+, +:datetime+, +:boolean+, 
 or +:serialized+) where +:string+ is the default. Each attribute also optionally has a default 
@@ -117,7 +117,7 @@ Sets are represented as arrays, may not contain duplicates and may not be empty.
 All attributes except the +:string+ type can take the value +nil+. Storing +nil+ for a string
 value will return the empty string, <tt>""</tt>.
 
-==== Schema args and options
+=== Schema args and options
 
 +dynamo_schema+ takes args and many options. Here's the full syntax:
 
@@ -139,9 +139,9 @@ value will return the empty string, <tt>""</tt>.
    ...
  end
 
-=== +has_many+ and +belongs_to+
+== +has_many+ and +belongs_to+
 
-==== Example
+=== Example
 
 The following example shows how to set up +has_many+ / +belongs_to+ relations:
  
@@ -175,7 +175,7 @@ is required as the Topic class itself has a +belongs_to+ relation and thus has
 a composite key. This must be declared in the child class as it needs to know
 how to retrieve its parent.
 
-==== Restrictions
+=== Restrictions
 
 Restrictions for +belongs_to+ tables: 
 * The hash key must be specified and must not be +:id+. 
@@ -189,7 +189,7 @@ Restrictions for +has_many+ tables:
 These restrictions allow OceanDynamo to implement the +has_many+ / +belongs_to+ 
 relation in a very efficient and massively scalable way. 
 
-==== Implementation
+=== Implementation
 
 +belongs_to+ claims the range key and uses it to store its own id, which normally
 would be stored in the hash key attribute. Instead, the hash key attribute holds the
@@ -214,16 +214,24 @@ reduced complexity.
 Nevertheless, as we now have switched to v2 of the DynamoDB API, we will be adding
 the possibility to define both local and secondary indices for Tables. 
 
+== Secondary Indices
+
+We now have basic support for secondary indices. 
+
+We will add high-level support for specifying that a secondary index is to be used 
+in +in_batches+ and +find_each+ etc, but for now you will have to specify that a
+secondary index is to be used explicitly in the options passed to +in_batches+, 
++query+, and +scan+. 
+* +query+: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Table.html#query-instance_method
+* +scan+: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Table.html#scan-instance_method
+* +in_batches+, +find_each+: lib/ocean-dynamo/queries.rb
 
 === Local Secondary Indices
 
-We now have basic support for local secondary indices. Up to five attributes can
-be declared as local secondary indices, in the following manner:
+Up to five attributes can be declared as local secondary indices, in the following manner:
 
   class Authentication < OceanDynamo::Table
-    dynamo_schema(:username, :expires_at,
-                  timestamps: nil, locking: false
-                  table_name_suffix: Api.basename_suffix) do
+    dynamo_schema(:username, :expires_at) do
       attribute :token,       :string,   local_secondary_index: true
       attribute :max_age,     :integer
       attribute :created_at,  :datetime
@@ -232,23 +240,42 @@ be declared as local secondary indices, in the following manner:
     end
   end
 
-The items of the above table can be accessed by using the hash key :username and
-the range key :expires_at. The +local_secondary_index+ declaration makes it possible
-to access items using the same hash key :username but with :token as an alternate
+The items of the above table can be accessed by using the hash key +:username+ and
+the range key +:expires_at+. The +local_secondary_index+ declaration makes it possible
+to access items using the same hash key +:username+ but with +:token+ as an alternate
 range key. Local secondary indices all use the same hash key as the primary index,
 substituting another attribute instead as the range key.
 
-NB: The primary index [:username, :expires_at] requires all items to have a unique
+NB: The primary index <tt>[:username, :expires_at]</tt> requires all items to have a unique
 combination of keys. Secondary indices don't require the range key to be unique for
 the same hash key. This means that secondary index searches always will return a
 collection.
 
-We will add support for specifying that a secondary index is to be used in +find_each+ 
-and its derivatives, but for now you will have to specify this explicitly in the
-options passed to +in_batches+, +query+, and +scan+. 
-* +query+: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Table.html#query-instance_method.
-* +scan+: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Table.html#scan-instance_method
-* +in_batches+, +find_each+: lib/ocean-dynamo/queries.rb
+=== Global Secondary Indices
+
+Global secondary indices are declared after all attributes, but still within the +do+
+block:
+
+  class Authentication < OceanDynamo::Table
+    dynamo_schema(:username, :expires_at) do
+      attribute :token,       :string,   local_secondary_index: true
+      attribute :max_age,     :integer
+      attribute :created_at,  :datetime
+      attribute :expires_at,  :datetime
+      attribute :api_user_id, :string
+
+      global_secondary_index :token, projection: :all
+      global_secondary_index :api_user_id, :expires_at, read_capacity_units: 100
+      global_secondary_index :expires_at, write_capacity_units: 50
+    end
+  end
+
+Each +global_secondary_index+ clause takes the following arguments:
+* +hash_value+ (required), 
+* +range_value+ (optional), 
+* +:projection (default :keys_only, :all for all attributes)
+* read_capacity_units (default 10)
+* write_capacity_units (default 5)
 
 
 == Installation

data/lib/ocean-dynamo/associations/has_many.rb

@@ -119,6 +119,7 @@ module OceanDynamo
       }
     end
 
+
     #
     # Reads all children of a has_many relation.
     #
@@ -184,9 +185,9 @@ module OceanDynamo
       child_hash_key = child_class.table_hash_key.to_s
       child_range_key = child_class.table_range_key && child_class.table_range_key.to_s
       child_class.in_batches :query, opts do |attrs|
-        # There is no way in the DynamoDB API to update a key attribute. Delete the child item.
+        # There is no way in the DynamoDB API to update a primary key attribute. 
+        # Delete the child item and recreate it with the updated key.
         child_class.delete attrs[child_hash_key], child_range_key && attrs[child_range_key]
-        # Create a new one with NULL for key
         attrs[child_hash_key] = "NULL"
         child_class.dynamo_table.put_item(item: attrs)
       end

data/lib/ocean-dynamo/attributes.rb

@@ -25,6 +25,7 @@ module OceanDynamo
         # Init
         self.fields = HashWithIndifferentAccess.new
         attribute(table_hash_key, :string, default: "")
+        self.global_secondary_indexes = Hash.new
         if table_range_key
           attribute(table_range_key, :string, default: "")
           self.validates(table_range_key, presence: true)

data/lib/ocean-dynamo/class_variables.rb

@@ -52,5 +52,8 @@ module OceanDynamo
     class_attribute :relations, instance_writer: false
     self.relations = nil
 
+    class_attribute :global_secondary_indexes, instance_writer: false
+    self.global_secondary_indexes = nil
+
   end
 end

data/lib/ocean-dynamo/queries.rb

@@ -59,6 +59,10 @@ module OceanDynamo
     # +message+ must be either :scan or :query.
     # +options+ is the hash of options to pass to the scan or query operation.
     #
+    # TODO: Add support for
+    #   index_name: "IndexName",
+    #   select: "ALL_ATTRIBUTES", # ALL_ATTRIBUTES, ALL_PROJECTED_ATTRIBUTES, SPECIFIC_ATTRIBUTES, COUNT
+    #
     def in_batches(message, options, &block)
       _late_connect?
       loop do
@@ -74,14 +78,17 @@ module OceanDynamo
 
     #
     # Looping through a collection of records from the database (using the +all+ method, 
-    # for example) is very inefficient since it will try to instantiate all the objects at once.
-    #
-    # In that case, batch processing methods allow you to work with the records in batches, 
+    # for example) is very inefficient since it will try to instantiate all the objects at 
+    # once. Batch processing methods allow you to work with the records in batches, 
     # thereby greatly reducing memory consumption.
     #
-    def find_each(limit: nil, batch_size: 1000, consistent: false)
+    # TODO: Add support for
+    #   index_name: "IndexName",
+    #   select: "ALL_ATTRIBUTES", # ALL_ATTRIBUTES, ALL_PROJECTED_ATTRIBUTES, SPECIFIC_ATTRIBUTES, COUNT
+    #
+    def find_each(consistent: false, limit: nil, batch_size: nil)
       options = { consistent_read: consistent }
-      batch_size = limit if limit && limit < batch_size
+      batch_size = limit if limit && batch_size && limit < batch_size
       options[:limit] = batch_size if batch_size   
       in_batches :scan, options do |attrs|
         if limit

data/lib/ocean-dynamo/schema.rb

@@ -39,6 +39,26 @@ module OceanDynamo
     end
 
 
+    def global_secondary_index(hash_key, range_key=nil, 
+                               projection: :keys_only,
+                               read_capacity_units: 10,
+                               write_capacity_units: 5)
+      if range_key
+        name = "#{hash_key}_#{range_key}"
+        keys = [hash_key.to_s, range_key.to_s]
+      else
+        name = "#{hash_key}"
+        keys = [hash_key.to_s]
+      end
+      self.global_secondary_indexes[name] = { 
+        "keys" => keys, 
+        "projection_type" => projection.to_s.upcase,
+        "read_capacity_units" => read_capacity_units,
+        "write_capacity_units" => write_capacity_units
+      }   
+    end
+
+
     protected
 
     def dangerous_attributes # :nodoc:

data/lib/ocean-dynamo/tables.rb

@@ -94,7 +94,7 @@ module OceanDynamo
       end
 
 
-       def create_table
+      def create_table
         attrs = table_attribute_definitions  # This already includes secondary indices
         keys = table_key_schema
         options = {
@@ -108,6 +108,8 @@ module OceanDynamo
         }
         lsi = local_secondary_indexes.collect { |n| local_secondary_index_declaration n }
         options[:local_secondary_indexes] = lsi unless lsi.blank?
+        gsi = global_secondary_indexes.collect { |k, v| global_secondary_index_declaration k, v }
+        options[:global_secondary_indexes] = gsi unless gsi.blank?
         dynamo_resource.create_table(options)
         sleep 1 until dynamo_table.table_status == "ACTIVE"
         setup_dynamo
@@ -128,6 +130,31 @@ module OceanDynamo
       end
 
 
+      def table_attribute_definitions
+        attrs = []
+        attrs << { attribute_name: table_hash_key.to_s, attribute_type: attribute_type(table_hash_key) }
+        attrs << { attribute_name: table_range_key.to_s, attribute_type: attribute_type(table_range_key) }  if table_range_key
+        local_secondary_indexes.each do |name|
+          attrs << { attribute_name: name, attribute_type: attribute_type(name) }
+        end
+        global_secondary_indexes.each do |index_name, data|
+          data["keys"].each do |name| 
+            next if attrs.any? { |a| a[:attribute_name] == name }
+            attrs << { attribute_name: name, attribute_type: attribute_type(name) }
+          end
+        end
+        attrs
+      end
+
+
+      def table_key_schema(hash_key: table_hash_key, range_key: table_range_key)
+        keys = []
+        keys << { attribute_name: hash_key.to_s, key_type: "HASH" }
+        keys << { attribute_name: range_key.to_s, key_type: "RANGE" } if range_key
+        keys
+      end
+
+
       def local_secondary_indexes
         @local_secondary_indexes ||= begin
           result = []
@@ -141,28 +168,23 @@ module OceanDynamo
 
       def local_secondary_index_declaration(name)
         { index_name: name,
-          key_schema: [{ attribute_name: table_hash_key.to_s, key_type: "HASH" },
-                       { attribute_name: name.to_s, key_type: "RANGE" }],
+          key_schema: table_key_schema(range_key: name),
           projection: { projection_type: "KEYS_ONLY" }
         }
       end
 
 
-      def table_attribute_definitions
-        attrs = []
-        attrs << { attribute_name: table_hash_key.to_s, attribute_type: attribute_type(table_hash_key) }
-        attrs << { attribute_name: table_range_key.to_s, attribute_type: attribute_type(table_range_key) }  if table_range_key
-        local_secondary_indexes.each do |name|
-          attrs << { attribute_name: name, attribute_type: attribute_type(name) }
-        end
-        attrs
-      end
-
-      def table_key_schema
-        keys = []
-        keys << { attribute_name: table_hash_key.to_s, key_type: "HASH" }
-        keys << { attribute_name: table_range_key.to_s, key_type: "RANGE" } if table_range_key
-        keys
+      def global_secondary_index_declaration(index_name, data)
+        hash_key, range_key = data["keys"]
+        key_schema = table_key_schema(hash_key: hash_key, range_key: range_key)
+        { index_name: index_name,
+          key_schema: key_schema,
+          projection: { projection_type: data["projection_type"] },
+          provisioned_throughput: {
+            read_capacity_units: data["read_capacity_units"],
+            write_capacity_units: data["write_capacity_units"]
+          }
+        }
       end
 
 

data/lib/ocean-dynamo/version.rb

@@ -1,3 +1,3 @@
 module OceanDynamo
-  VERSION = "1.1.0"
+  VERSION = "1.2.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ocean-dynamo
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Peter Bengtson
@@ -136,20 +136,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: "== OceanDynamo\n\nOceanDynamo is a massively scalable Amazon DynamoDB
-  near drop-in replacement for \nActiveRecord.\n\nAs one important use case for OceanDynamo
-  is to facilitate the conversion of SQL\ndatabases to no-SQL DynamoDB databases,
-  it is important that the syntax and semantics\nof OceanDynamo are as close as possible
-  to those of ActiveRecord. This includes\ncallbacks, exceptions and method chaining
-  semantics. OceanDynamo follows this pattern \nclosely and is of course based on
-  ActiveModel.\n\nThe attribute and persistence layer of OceanDynamo is modeled on
-  that of ActiveRecord:\n+save+, +save!+, +create+, +update+, +update!+, +update_attributes+,
-  +find_each+,\n+destroy_all+, +delete_all+ and all the other methods you're used
-  to are available. \nThe design goal is always to implement as much of the ActiveRecord
-  interface as possible, \nwithout compromising scalability. This makes the task of
-  switching from SQL to no-SQL \nmuch easier.\n\nThanks to its structural similarity
-  to ActiveRecord, OceanDynamo works with FactoryGirl.\n\nOceanDynamo uses primary
-  indices to retrieve related table items, meaning it scales without \nlimits.\n\nSee
+description: "== OceanDynamo\n\nAs one important use case for OceanDynamo is to facilitate
+  the conversion of SQL\ndatabases to no-SQL DynamoDB databases, it is important that
+  the syntax and semantics\nof OceanDynamo are as close as possible to those of ActiveRecord.
+  This includes\ncallbacks, exceptions and method chaining semantics. OceanDynamo
+  follows this pattern \nclosely and is of course based on ActiveModel.\n\nThe attribute
+  and persistence layer of OceanDynamo is modeled on that of ActiveRecord:\nthere's
+  +save+, +save!+, +create+, +update+, +update!+, +update_attributes+, +find_each+,\n+destroy_all+,
+  +delete_all+, +read_attribute+, +write_attribute+ and all the other \nmethods you're
+  used to. The design goal is always to implement as much of the ActiveRecord\ninterface
+  as possible, without compromising scalability. This makes the task of switching
+  \nfrom SQL to no-SQL much easier.\n\nOceanDynamo uses only primary indices to retrieve
+  related table items and collections, \nwhich means it will scale without limits.\n\nOceanDynamo
+  is fully usable as an ActiveModel and can be used by Rails\ncontrollers. Thanks
+  to its structural similarity to ActiveRecord, OceanDynamo works \nwith FactoryGirl.\n\nSee
   also Ocean, a Rails framework for creating highly scalable SOAs in the cloud, in
   which\nocean-dynamo is used as a central component: http://wiki.oceanframework.net"
 email: