dynamoid 0.3.2 → 0.4.0

data/Dynamoid.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "dynamoid"
-  s.version = "0.3.2"
+  s.version = "0.4.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Josh Symonds"]
-  s.date = "2012-04-13"
+  s.date = "2012-04-26"
   s.description = "Dynamoid is an ORM for Amazon's DynamoDB that supports offline development, associations, querying, and everything else you'd expect from an ActiveRecord-style replacement."
   s.email = "josh@joshsymonds.com"
   s.extra_rdoc_files = [
@@ -106,6 +106,7 @@ Gem::Specification.new do |s|
     "spec/app/models/magazine.rb",
     "spec/app/models/sponsor.rb",
     "spec/app/models/subscription.rb",
+    "spec/app/models/tweet.rb",
     "spec/app/models/user.rb",
     "spec/dynamoid/adapter/aws_sdk_spec.rb",
     "spec/dynamoid/adapter/local_spec.rb",

data/README.markdown

@@ -28,6 +28,7 @@ Then you need to initialize it to get it going. Put code similar to this somewhe
     config.partition_size = 200  # Determine the key space size that writes are randomly spread across.
     config.read_capacity = 100 # Read capacity for your tables
     config.write_capacity = 20 # Write capacity for your tables
+    config.endpoint = 'dynamodb.us-east-1.amazonaws.com' # Set the regional endpoint
   end
 
 ```
@@ -45,6 +46,20 @@ class User
 end
 ```
 
+### Table
+
+Dynamoid has some sensible defaults for you when you create a new table, including the table name and the primary key column. But you can change those if you like on table creation.
+
+```ruby
+class User
+  include Dynamoid::Document
+  
+  table :name => :awesome_users, :key => :user_id, :read_capacity => 400, :write_capacity => 400
+end
+```
+
+These fields will not change an existing table: so specifying a new read_capacity and write_capacity here only works correctly for entirely new tables. Similarly, while Dynamoid will look for a table named `awesome_users` in your namespace, it won't change any existing tables to use that name; and if it does find a table with the correct name, it won't change its hash key, which it expects will be user_id. If this table doesn't exist yet, however, Dynamoid will create it with these options.
+
 ### Fields
 
 You'll have to define all the fields on the model and the data type of each field. Every field on the object must be included here; if you miss any they'll be completely bypassed during DynamoDB's initialization and will not appear on the model objects.
@@ -79,7 +94,7 @@ class User
   index :email          
   index [:name, :email] 
   index :created_at, :range => true
-  index :name, :range => :joined_at
+  index :name, :range_key => :joined_at
   
 end
 ```
@@ -152,7 +167,9 @@ end
 
 ## Usage
 
-Dynamoid's syntax is very similar to ActiveRecord's.
+### Object Creation
+
+Dynamoid's syntax is generally very similar to ActiveRecord's. Making new objects is simple:
 
 ```ruby
 u = User.new(:name => 'Josh')
@@ -174,6 +191,8 @@ address.city = 'Chicago'
 address.save
 ```
 
+### Querying
+
 Querying can be done in one of three ways:
 
 ```ruby
@@ -190,6 +209,23 @@ u.addresses.where(:city => 'Chicago').all
 
 But keep in mind Dynamoid -- and document-based storage systems in general -- are not drop-in replacements for existing relational databases. The above query does not efficiently perform a conditional join, but instead finds all the user's addresses and naively filters them in Ruby. For large associations this is a performance hit compared to relational database engines.
 
+You can also limit returned results, or select a record from which to start, to support pagination:
+
+```ruby
+Address.limit(5).start(address) # Only 5 addresses.
+```
+
+### Consistent Reads
+
+Querying supports consistent reading. By default, DynamoDB reads are eventually consistent: if you do a write and then a read immediately afterwards, the results of the previous write may not be reflected. If you need to do a consistent read (that is, you need to read the results of a write immediately) you can do so, but keep in mind that consistent reads are twice as expensive as regular reads for DynamoDB.
+
+```ruby
+Address.find(address.id, :consistent_read => true)  # Find an address, ensure the read is consistent.
+Address.where(:city => 'Chicago').consistent.all    # Find all addresses where the city is Chicago, with a consistent read.
+```
+
+### Range Finding
+
 If you have a range index, Dynamoid provides a number of additional other convenience methods to make your life a little easier:
 
 ```ruby
@@ -205,7 +241,7 @@ DynamoDB achieves much of its speed by relying on a random pattern of writes and
 
 Dynamoid attempts to obviate this problem transparently by employing a partitioning strategy to divide up keys randomly across DynamoDB's servers. Each ID is assigned an additional number (by default 0 to 199, but you can increase the partition size in Dynamoid's configuration) upon save; when read, all 200 hashes are retrieved simultaneously and the most recently updated one is returned to the application. This results in a significant net performance increase, and is usually invisible to the application itself. It does, however, bring up the important issue of provisioning your DynamoDB tables correctly.
 
-When your read or write provisioning exceed your table's allowed throughput, DynamoDB will wait on connections until throughput is available again. This will appear as very, very slow requests and can be somewhat frustrating. Partitioning significantly increases the amount of throughput tables will experience; though DynamoDB will ignore keys that don't exist, if you have 20 partitioned keys representing one object, all will be retrieved every time the object is requested. Ensure that your tables are set up for this kind of throughput, or turn provisioning off, to make sure that DynamoDB doesn't throttle your requests.
+When your read or write throughput exceed your table's allowed provisioning, DynamoDB will wait on connections until throughput is available again. This will appear as very, very slow requests and can be somewhat frustrating. Partitioning significantly increases the amount of throughput tables will experience; though DynamoDB will ignore keys that don't exist, if you have 20 partitioned keys representing one object, all will be retrieved every time the object is requested. Ensure that your tables are set up for this kind of throughput, or turn provisioning off, to make sure that DynamoDB doesn't throttle your requests.
 
 ## Credits
 

data/VERSION

@@ -1 +1 @@
-0.3.2
+0.4.0

data/lib/dynamoid.rb

@@ -23,6 +23,8 @@ require 'dynamoid/adapter'
 
 module Dynamoid
   extend self
+
+  MAX_ITEM_SIZE = 65_536
   
   def configure
     block_given? ? yield(Dynamoid::Config) : Dynamoid::Config

data/lib/dynamoid/adapter.rb

@@ -67,7 +67,8 @@ module Dynamoid
     # @param [Number] range_key the range key of the record
     #
     # @since 0.2.0
-    def read(table, ids, range_key = nil)
+    def read(table, ids, options = {})
+      range_key = options[:range_key]
       if ids.respond_to?(:each)
         ids = ids.collect{|id| range_key ? [id, range_key] : id}
         if Dynamoid::Config.partitioning?
@@ -82,7 +83,7 @@ module Dynamoid
           results = benchmark('Partitioned Get Item', ids) {batch_get_item(table => id_with_partitions(ids))}
           result_for_partition(results[table]).first
         else
-          benchmark('Get Item', ids) {get_item(table, ids, range_key)}
+          benchmark('Get Item', ids) {get_item(table, ids, options)}
         end
       end
     end
@@ -94,13 +95,13 @@ module Dynamoid
     # @param [Number] range_key the range key of the record
     #
     # @since 0.2.0
-    def delete(table, id, range_key = nil)
+    def delete(table, id, options = {})
       if Dynamoid::Config.partitioning?
         benchmark('Delete Item', id) do
-          id_with_partitions(id).each {|i| delete_item(table, i, range_key)}
+          id_with_partitions(id).each {|i| delete_item(table, i, options)}
         end
       else
-        benchmark('Delete Item', id) {delete_item(table, id, range_key)}
+        benchmark('Delete Item', id) {delete_item(table, id, options)}
       end
     end
     
@@ -110,12 +111,12 @@ module Dynamoid
     # @param [Hash] scan_hash a hash of attributes: matching records will be returned by the scan
     #
     # @since 0.2.0    
-    def scan(table, query)
+    def scan(table, query, opts = {})
       if Dynamoid::Config.partitioning?
-        results = benchmark('Scan', table, query) {adapter.scan(table, query)}
+        results = benchmark('Scan', table, query) {adapter.scan(table, query, opts)}
         result_for_partition(results)
       else
-        adapter.scan(table, query)
+        adapter.scan(table, query, opts)
       end
     end
     

data/lib/dynamoid/adapter/aws_sdk.rb

@@ -18,7 +18,7 @@ module Dynamoid
       #
       # @since 0.2.0
       def connect!
-        @@connection = AWS::DynamoDB.new(:access_key_id => Dynamoid::Config.access_key, :secret_access_key => Dynamoid::Config.secret_key)
+        @@connection = AWS::DynamoDB.new(:access_key_id => Dynamoid::Config.access_key, :secret_access_key => Dynamoid::Config.secret_key, :dynamo_db_endpoint => Dynamoid::Config.endpoint)
       end
     
       # Return the established connection.
@@ -64,7 +64,6 @@ module Dynamoid
       # @since 0.2.0
       def create_table(table_name, key = :id, options = {})
         options[:hash_key] ||= {key.to_sym => :string}
-        options[:range_key] = {options[:range_key].to_sym => :number} if options[:range_key]
         read_capacity = options[:read_capacity] || Dynamoid::Config.read_capacity
         write_capacity = options[:write_capacity] || Dynamoid::Config.write_capacity
         table = @@connection.tables.create(table_name, read_capacity, write_capacity, options)
@@ -79,7 +78,8 @@ module Dynamoid
       # @param [Number] range_key the range key of the item to delete, required if the table has a composite key
       #
       # @since 0.2.0
-      def delete_item(table_name, key, range_key = nil)
+      def delete_item(table_name, key, options = {})
+        range_key = options.delete(:range_key)
         table = get_table(table_name)
         result = if table.composite_key?
           table.items.at(key, range_key)
@@ -111,13 +111,18 @@ module Dynamoid
       # @return [Hash] a hash representing the raw item in DynamoDB
       #
       # @since 0.2.0
-      def get_item(table_name, key, range_key = nil)
+
+
+
+      def get_item(table_name, key, options = {})
+        range_key = options.delete(:range_key)
         table = get_table(table_name)
+
         result = if table.composite_key?
           table.items.at(key, range_key)
         else
           table.items[key]
-        end.attributes.to_h
+        end.attributes.to_h(options)
         if result.empty?
           nil
         else
@@ -161,10 +166,11 @@ module Dynamoid
       # @since 0.2.0
       def query(table_name, opts = {})
         table = get_table(table_name)
-        
+
+        consistent_opts = { :consistent_read => opts[:consistent_read] || false }
         if table.composite_key?
           results = []
-          table.items.query(opts).each {|data| results << data.attributes.to_h.symbolize_keys!}
+          table.items.query(opts).each {|data| results << data.attributes.to_h(consistent_opts).symbolize_keys!}
           results
         else
           get_item(table_name, opts[:hash_value])
@@ -180,10 +186,10 @@ module Dynamoid
       # @return [Array] an array of all matching items
       #
       # @since 0.2.0
-      def scan(table_name, scan_hash)
+      def scan(table_name, scan_hash, select_opts)
         table = get_table(table_name)
         results = []
-        table.items.where(scan_hash).select do |data|
+        table.items.where(scan_hash).select(select_opts) do |data|
           results << data.attributes.symbolize_keys!
         end
         results

data/lib/dynamoid/adapter/local.rb

@@ -38,7 +38,7 @@ module Dynamoid
             table = data[table_name]
             if table[:range_key]
               Array(keys).each do |hash_key, range_key|
-                hash[table_name] << get_item(table_name, hash_key, range_key)
+                hash[table_name] << get_item(table_name, hash_key, :range_key => range_key)
               end
             else
               Array(keys).each do |key|
@@ -57,7 +57,8 @@ module Dynamoid
       #
       # @since 0.2.0
       def create_table(table_name, key, options = {})
-        data[table_name] = {:hash_key => key, :range_key => options[:range_key], :data => {}}
+        range_key = options[:range_key] && options[:range_key].keys.first
+        data[table_name] = {:hash_key => key, :range_key => range_key, :data => {}}
       end
     
       # Removes an item from the hash.
@@ -67,7 +68,8 @@ module Dynamoid
       # @param [Number] range_key the range key of the item to delete, required if the table has a composite key
       #
       # @since 0.2.0
-      def delete_item(table_name, key, range_key = nil)
+      def delete_item(table_name, key, options = {})
+        range_key = options.delete(:range_key)
         data[table_name][:data].delete("#{key}.#{range_key}")
       end
     
@@ -91,7 +93,8 @@ module Dynamoid
       # @return [Hash] a hash representing the raw item
       #
       # @since 0.2.0
-      def get_item(table_name, key, range_key = nil)
+      def get_item(table_name, key, options = {})
+        range_key = options[:range_key]
         if data[table_name][:data]
           data[table_name][:data]["#{key}.#{range_key}"]
         else
@@ -116,6 +119,8 @@ module Dynamoid
         table = data[table_name]
         table[:data][object[table[:hash_key]]]
         table[:data]["#{object[table[:hash_key]]}.#{object[table[:range_key]]}"] = object.delete_if{|k, v| v.nil? || (v.respond_to?(:empty?) && v.empty?)}
+      rescue
+        raise data.inspect
       end
     
       # Query the hash.
@@ -134,20 +139,26 @@ module Dynamoid
       # @since 0.2.0
       def query(table_name, opts = {})
         id = opts[:hash_value]
+        hash_key = data[table_name][:hash_key]
         range_key = data[table_name][:range_key]
-        if opts[:range_value]
-          data[table_name][:data].values.find_all{|v| v[:id] == id && !v[range_key].nil? && opts[:range_value].include?(v[range_key])}
+
+        results = if opts[:range_value]
+          data[table_name][:data].values.find_all{|v| v[hash_key] == id && !v[range_key].nil? && opts[:range_value].include?(v[range_key])}
         elsif opts[:range_greater_than]
-          data[table_name][:data].values.find_all{|v| v[:id] == id && !v[range_key].nil? && v[range_key] > opts[:range_greater_than]}
+          data[table_name][:data].values.find_all{|v| v[hash_key] == id && !v[range_key].nil? && v[range_key] > opts[:range_greater_than]}
         elsif opts[:range_less_than]
-          data[table_name][:data].values.find_all{|v| v[:id] == id && !v[range_key].nil? && v[range_key] < opts[:range_less_than]}
+          data[table_name][:data].values.find_all{|v| v[hash_key] == id && !v[range_key].nil? && v[range_key] < opts[:range_less_than]}
         elsif opts[:range_gte]
-          data[table_name][:data].values.find_all{|v| v[:id] == id && !v[range_key].nil? && v[range_key] >= opts[:range_gte]}
+          data[table_name][:data].values.find_all{|v| v[hash_key] == id && !v[range_key].nil? && v[range_key] >= opts[:range_gte]}
         elsif opts[:range_lte]  
-          data[table_name][:data].values.find_all{|v| v[:id] == id && !v[range_key].nil? && v[range_key] <= opts[:range_lte]}
+          data[table_name][:data].values.find_all{|v| v[hash_key] == id && !v[range_key].nil? && v[range_key] <= opts[:range_lte]}
         else
-          get_item(table_name, id)
+          data[table_name][:data].values.find_all{|v| v[hash_key] == id}
         end
+
+        results = drop_till_start(results, opts[:next_token], range_key, hash_key)
+        results = results.take(opts[:limit]) if opts[:limit]
+        results
       end
     
       # Scan the hash.
@@ -158,9 +169,23 @@ module Dynamoid
       # @return [Array] an array of all matching items
       #
       # @since 0.2.0
-      def scan(table_name, scan_hash)
+      def scan(table_name, scan_hash, opts = {})
         return [] if data[table_name].nil?
-        data[table_name][:data].values.flatten.select{|d| scan_hash.all?{|k, v| !d[k].nil? && d[k] == v}}
+        results = data[table_name][:data].values.flatten.select{|d| scan_hash.all?{|k, v| !d[k].nil? && d[k] == v}}
+        results = drop_till_start(results, opts[:next_token], data[table_name][:range_key], data[table_name][:hash_key])
+        results = results.take(opts[:limit]) if opts[:limit]
+        results
+      end
+
+      def drop_till_start(results, next_token, range_key, hash_key)
+        return results unless next_token
+
+        hash_value = next_token[:hash_key_element].values.first
+        range_value = next_token[:range_key_element].values.first if next_token[:range_key_element]
+
+        results = results.drop_while do |r|
+          (r[hash_key] != hash_value or r[range_key] != range_value)
+        end.drop(1)
       end
     
       # @todo Add an UpdateItem method.
@@ -168,4 +193,4 @@ module Dynamoid
       # @todo Add an UpdateTable method.
     end
   end
-end
+end

data/lib/dynamoid/associations.rb

@@ -91,17 +91,16 @@ module Dynamoid
         field "#{name}_ids".to_sym, :set
         self.associations[name] = options.merge(:type => type)
         define_method(name) do
-          @associations ||= {}
-          @associations[name] ||= Dynamoid::Associations.const_get(type.to_s.camelcase).new(self, name, options)
+          @associations[:"#{name}_ids"] ||= Dynamoid::Associations.const_get(type.to_s.camelcase).new(self, name, options)
         end
         define_method("#{name}=".to_sym) do |objects|
-          @associations ||= {}
-          @associations[name] ||= Dynamoid::Associations.const_get(type.to_s.camelcase).new(self, name, options)
-          @associations[name].setter(objects)
+          @associations[:"#{name}_ids"] ||= Dynamoid::Associations.const_get(type.to_s.camelcase).new(self, name, options)
+          @associations[:"#{name}_ids"].setter(objects)
         end
       end
     end
-    
+
+
   end
   
 end

data/lib/dynamoid/associations/association.rb

@@ -6,7 +6,7 @@ module Dynamoid #:nodoc:
   # The target is the object which is referencing by this association.
   module Associations
     module Association
-      attr_accessor :name, :options, :source
+      attr_accessor :name, :options, :source, :loaded
 
       # Create a new association.
       #
@@ -24,6 +24,28 @@ module Dynamoid #:nodoc:
         @name = name
         @options = options
         @source = source
+        @loaded = false
+      end
+
+      def loaded?
+        @loaded
+      end
+
+      def find_target
+      end
+
+      def target
+        unless loaded?
+          @target = find_target
+          @loaded = true
+        end
+
+        @target
+      end
+
+      def reset
+        @target = nil
+        @loaded = false
       end
 
       private

data/lib/dynamoid/associations/belongs_to.rb

@@ -5,7 +5,6 @@ module Dynamoid #:nodoc:
   # object to which the association object is associated.
   module Associations
     class BelongsTo
-      include Association
       include SingleAssociation
 
       private

data/lib/dynamoid/associations/has_and_belongs_to_many.rb

@@ -4,7 +4,6 @@ module Dynamoid #:nodoc:
   # The has and belongs to many association.
   module Associations
     class HasAndBelongsToMany
-      include Association
       include ManyAssociation
 
       private

data/lib/dynamoid/associations/has_many.rb

@@ -4,7 +4,6 @@ module Dynamoid #:nodoc:
   # The has_many association.
   module Associations
     class HasMany
-      include Association
       include ManyAssociation
 
       private

data/lib/dynamoid/associations/many_association.rb

@@ -3,6 +3,7 @@ module Dynamoid #:nodoc:
 
   module Associations
     module ManyAssociation
+      include Association
 
       attr_accessor :query
 
@@ -20,13 +21,15 @@ module Dynamoid #:nodoc:
       # @return the association records; depending on which association this is, either a single instance or an array
       #
       # @since 0.2.0
-      def records
-        results = Array(target_class.find(source_ids.to_a))
+      def find_target
+        Array(target_class.find(source_ids.to_a))
+      end
 
+      def records
         if query.empty?
-          results
+          target
         else
-          results_with_query(results)
+          results_with_query(target)
         end
       end
 
@@ -78,7 +81,7 @@ module Dynamoid #:nodoc:
       #
       # @since 0.2.0
       def setter(object)
-        records.each {|o| delete(o)}
+        target.each {|o| delete(o)}
         self << (object)
         object
       end
@@ -120,7 +123,7 @@ module Dynamoid #:nodoc:
       #
       # @since 0.2.0
       def destroy_all
-        objs = records
+        objs = target
         source.update_attribute(source_attribute, nil)
         objs.each(&:destroy)
       end
@@ -129,7 +132,7 @@ module Dynamoid #:nodoc:
       #
       # @since 0.2.0
       def delete_all
-        objs = records
+        objs = target
         source.update_attribute(source_attribute, nil)
         objs.each(&:delete)
       end