dm-adapter-simpledb 1.3.0 → 1.4.0

data/History.txt

@@ -1,3 +1,60 @@
+== 1.4.0 2010-01-24
+
+* Major enhancements:
+  * Completely rewritten query generation engine queries supports
+    arbitrarily deep nesting of complex AND/OR/NOT operators.  This means that
+    code such as:
+
+      Post.all(:title => "foo") | Post.all(:title => "bar", :body => "baz")
+
+    will generate a SELECT statement similar to the following
+
+      SELECT title from mydomain where title = "foo" OR ( title = "bar" AND body = "baz" )
+
+  * New query engine supports range and IN predicates to the extent possible on
+    SimpleDB, including converting empty inclusion predicates to a form that
+    SimpleDB can understand. Exclusive ranges e.g. (0...5) now raise a
+    NotImplementedError instead of being silently flaky.
+
+  * Support for native condition expressions, e.g.:
+
+      Post.all(:conditions => 'title in "%banannas%"')
+
+    This includes support for variable interpolation:
+
+      Post.all(:conditions => ['title = ?', "foo"])
+
+    or:
+
+      post.all(:conditions => ['title = :title', {:title => "foo"}]
+
+    Interpolated values will be quoted according to SimpleDB value quoting
+    rules.
+
+  * Support for arbitrarily large limits. The concept of a query limit and a
+    batch limit have been completely separated in this release. If the batch
+    limit is set to 100 and a query is limited to 201 items, it will generate
+    three selects: two with "LIMIT 100" and one with "LIMIT 1".
+
+  * Vastly improved logging and benchmarking. For a given high-level operation,
+    such as a DataMapper "read", the adapter can output:
+    
+    * Number of individual AWS calls made (e.g. individual SELECTs)
+    * Aggregate AWS box usage
+    * User CPU time
+    * System CPU time
+    * Wallclock time
+
+* Minor enhancements:
+  * Even better quoting. With the new SELECT translator in place, all domain
+    names, attribute names, and values should be quoted properly according to
+    SimpleDB rules.
+  * No direct dependency on RightAws.  All operations are performed via SDBTools
+    now.
+  * New "batch_limit" option to configure the maximum results requested per
+    SELECT call.  Amazon sets a cap of 250 on this value.
+
+
 == 1.3.0 2010-01-19
 
 * 1 major enhancement:
@@ -6,6 +63,8 @@
 
 * 1 minor enhancement:
   * Better quoting of select statements using the 'sdbtools' library.
+  * "Null mode" can be set with :null => true. Null mode logs DB operations but
+    does not actually connect to AWS.
 
 == 1.2.0 2010-01-13
 

data/README

@@ -7,8 +7,8 @@ A DataMapper adapter for Amazon's SimpleDB service.
 Features:
  * Uses the RightAWS gem for efficient SimpleDB operations.
  * Full set of CRUD operations
- * Supports all DataMapper query predicates.
- * Can translate many queries into efficient native SELECT operations.
+ * Supports nearly all DataMapper query predicates.
+ * Full support for complex nested union, intersaction, and negation in queries
  * Migrations
  * DataMapper identity map support for record caching
  * Lazy-loaded attributes
@@ -17,6 +17,7 @@ Features:
  * Basic aggregation support (Model.count("..."))
  * String "chunking" permits attributes to exceed the 1024-byte limit
  * Support for efficient :limit and :offset, for result set paging
+ * Robust quoting of names in values in selects
 
 Note: as of version 1.0.0, this gem supports supports the DataMapper 0.10.*
 series and breaks backwards compatibility with DataMapper 0.9.*.
@@ -39,9 +40,6 @@ http://github.com/devver/dm-adapter-simpledb/
 
 == TODO
 
- * More complete handling of NOT conditions in queries
- * Support for ORs, parens, and nested queries in general
- * Robust quoting in SELECT calls
  * Handle exclusive ranges natively
    Implement as inclusive range + filter step
  * Tests for associations
@@ -58,6 +56,7 @@ http://github.com/devver/dm-adapter-simpledb/
  * Silence SSL warnings
    See http://pivotallabs.com/users/carl/blog/articles/1079-standup-blog-11-24-2009-model-validations-without-backing-store-associations-to-array-and-ssl-with-aws
  * Token cache for reduced requests when given an offset
+ * Optimize key queries
 
 == Usage
   

data/Rakefile

@@ -54,7 +54,6 @@ begin
 A DataMapper adapter for Amazon's SimpleDB service. 
 
 Features:
- * Uses the RightAWS gem for efficient SimpleDB operations.
  * Full set of CRUD operations
  * Supports all DataMapper query predicates.
  * Can translate many queries into efficient native SELECT operations.
@@ -81,7 +80,7 @@ END
     gem.add_dependency('dm-migrations',         '~> 0.10.0')
     gem.add_dependency('dm-types',              '~> 0.10.0')
     gem.add_dependency('uuidtools',             '~> 2.0')
-    gem.add_dependency('sdbtools',              '~> 0.2')
+    gem.add_dependency('sdbtools',              '~> 0.4')
   end
   Jeweler::GemcutterTasks.new
 rescue LoadError

data/VERSION

@@ -1 +1 @@
-1.3.0
+1.4.0

data/lib/dm-adapter-simpledb.rb

@@ -6,13 +6,14 @@ gem 'dm-core',       '~> 0.10.0'
 require 'dm-core'
 require 'dm-aggregates'
 require 'digest/sha1'
-require 'right_aws'
 require 'uuidtools'
 require 'sdbtools'
 
+$:.unshift(File.dirname(__FILE__))
 require 'dm-adapter-simpledb/sdb_array'
 require 'dm-adapter-simpledb/utils'
 require 'dm-adapter-simpledb/record'
 require 'dm-adapter-simpledb/table'
+require 'dm-adapter-simpledb/where_expression'
 require 'dm-adapter-simpledb/migrations/simpledb_adapter'
 require 'dm-adapter-simpledb/adapters/simpledb_adapter'

data/lib/dm-adapter-simpledb/adapters/simpledb_adapter.rb

@@ -3,7 +3,9 @@ module DataMapper
     class SimpleDBAdapter < AbstractAdapter
       include DmAdapterSimpledb::Utils
 
-      attr_reader :sdb_options
+      attr_reader   :sdb_options
+      attr_reader   :batch_limit
+      attr_accessor :logger
 
       # For testing purposes ONLY. Seriously, don't enable this for production
       # code.
@@ -18,7 +20,8 @@ module DataMapper
         @sdb_options[:secret_key] = options.fetch(:secret_key) { 
           options[:password] 
         }
-        @sdb_options[:logger] = options.fetch(:logger) { DataMapper.logger }
+        @logger = options.fetch(:logger) { DataMapper.logger }
+        @sdb_options[:logger] = @logger
         @sdb_options[:server] = options.fetch(:host) { 'sdb.amazonaws.com' }
         @sdb_options[:port]   = options[:port] || 443 # port may be set but nil
         @sdb_options[:domain] = options.fetch(:domain) { 
@@ -33,18 +36,27 @@ module DataMapper
         # RightAWS's nil-token replacement altogether, but that does not appear
         # to be an option.
         @sdb_options[:nil_representation] = "<[<[<NIL>]>]>"
+        @null_mode   = options.fetch(:null) { false }
+        @batch_limit = options.fetch(:batch_limit) {
+          SDBTools::Selection::DEFAULT_RESULT_LIMIT
+        }.to_i
+
+        if @null_mode
+          logger.info "SimpleDB adapter for domain #{domain_name} is in null mode"
+        end
+
         @consistency_policy = 
           normalised_options.fetch(:wait_for_consistency) { false }
         @sdb = options.fetch(:sdb_interface) { nil }
-        if @sdb_options[:create_domain] && !domains.include?(@sdb_options[:domain])
-          @sdb_options[:logger].info "Creating domain #{domain}"
-          @sdb.create_domain(@sdb_options[:domain])
+        if @sdb_options[:create_domain] && !domains.include?(domain_name)
+          @sdb_options[:logger].info "Creating domain #{domain_name}"
+          database.create_domain(domain_name)
         end
       end
 
       def create(resources)
         created = 0
-        time = Benchmark.realtime do
+        transaction("CREATE #{resources.size} objects") do
           resources.each do |resource|
             uuid = UUIDTools::UUID.timestamp_create
             initialize_serial(resource, uuid.to_i)
@@ -52,63 +64,63 @@ module DataMapper
             record     = DmAdapterSimpledb::Record.from_resource(resource)
             attributes = record.writable_attributes
             item_name  = record.item_name
-            sdb.put_attributes(domain, item_name, attributes)
+            domain.put(item_name, attributes, :replace => true)
             created += 1
           end
         end
-        DataMapper.logger.debug(format_log_entry("(#{created}) INSERT #{resources.inspect}", time))
         modified!
         created
       end
       
       def delete(collection)
         deleted = 0
-        time = Benchmark.realtime do
+        transaction("DELETE #{collection.query.conditions}") do
           collection.each do |resource|
             record = DmAdapterSimpledb::Record.from_resource(resource)
             item_name = record.item_name
-            sdb.delete_attributes(domain, item_name)
+            domain.delete(item_name)
             deleted += 1
           end
+
+          # TODO no reason we can't select a bunch of item names with an
+          # arbitrary query and then delete them.
           raise NotImplementedError.new('Only :eql on delete at the moment') if not_eql_query?(collection.query)
-        end; DataMapper.logger.debug(format_log_entry("(#{deleted}) DELETE #{collection.query.conditions.inspect}", time))
+        end
         modified!
         deleted
       end
 
       def read(query)
         maybe_wait_for_consistency
-        table = DmAdapterSimpledb::Table.new(query.model)
-        conditions, order, unsupported_conditions = 
-          set_conditions_and_sort_order(query, table.simpledb_type)
-        results = get_results(query, conditions, order)
-        records = results.map{|result| 
-          DmAdapterSimpledb::Record.from_simpledb_hash(result)
-        }
+        transaction("READ #{query.model.name} #{query.conditions}") do |t|
+          query = query.dup
 
-        proto_resources = records.map{|record|
-          record.to_resource_hash(query.fields)
-        }
-        query.conditions.operands.reject!{ |op|
-          !unsupported_conditions.include?(op)
-        }
+          selection = selection_from_query(query)
 
-        # This used to be a simple call to Query#filter_records(), but that
-        # caused the result limit to be re-imposed on an already limited result
-        # set, with the upshot that too few records were returned. So here we do
-        # everything filter_records() does EXCEPT limiting.
-        records = proto_resources
-        records = records.uniq if query.unique?
-        records = query.match_records(records)
-        records = query.sort_records(records)
+          records = selection.map{|name, attributes| 
+            DmAdapterSimpledb::Record.from_simpledb_hash(name => attributes)
+          }
 
+          proto_resources = records.map{|record|
+            record.to_resource_hash(query.fields)
+          }
+          
+          # This used to be a simple call to Query#filter_records(), but that
+          # caused the result limit to be re-imposed on an already limited result
+          # set, with the upshot that too few records were returned. So here we do
+          # everything filter_records() does EXCEPT limiting.
+          records = proto_resources
+          records = records.uniq if query.unique?
+          records = query.match_records(records)
+          records = query.sort_records(records)
 
-        records
+          records
+        end
       end
       
       def update(attributes, collection)
         updated = 0
-        time = Benchmark.realtime do
+        transaction("UPDATE #{collection.query} with #{attributes.inspect}") do
           collection.each do |resource|
             updated_resource = resource.dup
             updated_resource.attributes = attributes
@@ -117,40 +129,24 @@ module DataMapper
             attrs_to_delete = record.deletable_attributes
             item_name       = record.item_name
             unless attrs_to_update.empty?
-              sdb.put_attributes(domain, item_name, attrs_to_update, :replace)
+              domain.put(item_name, attrs_to_update, :replace => true)
             end
             unless attrs_to_delete.empty?
-              sdb.delete_attributes(domain, item_name, attrs_to_delete)
+              domain.delete(item_name, attrs_to_delete)
             end
             updated += 1
           end
           raise NotImplementedError.new('Only :eql on delete at the moment') if not_eql_query?(collection.query)
         end
-        DataMapper.logger.debug(format_log_entry("UPDATE #{collection.query.conditions.inspect} (#{updated} times)", time))
         modified!
         updated
       end
       
-      def query(query_call, query_limit = 999999999)
-        SDBTools::Operation.new(sdb, :select, query_call).inject([]){
-          |a, results| 
-          a.concat(results[:items].map{|i| i.values.first})
-        }[0...query_limit]
-      end
-      
       def aggregate(query)
-        raise ArgumentError.new("Only count is supported") unless (query.fields.first.operator == :count)
-        table    = DmAdapterSimpledb::Table.new(query.model)
-        sdb_type = table.simpledb_type
-        conditions, order, unsupported_conditions = set_conditions_and_sort_order(query, sdb_type)
-
-        query_call = "SELECT count(*) FROM #{domain} "
-        query_call << "WHERE #{conditions.compact.join(' AND ')}" if conditions.length > 0
-        results = nil
-        time = Benchmark.realtime do
-          results = sdb.select(query_call)
-        end; DataMapper.logger.debug(format_log_entry(query_call, time))
-        [results[:items][0].values.first["Count"].first.to_i]
+        raise NotImplementedError, "Only count is supported" unless (query.fields.first.operator == :count)
+        transaction("AGGREGATE") do |t|
+          [selection_from_query(query).count]
+        end
       end
 
       # For testing purposes only.
@@ -158,142 +154,25 @@ module DataMapper
         return unless @current_consistency_token
         token = :none
         begin
-          results = sdb.get_attributes(domain, '__dm_consistency_token', '__dm_consistency_token')
+          results = domain.get('__dm_consistency_token', '__dm_consistency_token')
           tokens  = Array(results[:attributes]['__dm_consistency_token'])
         end until tokens.include?(@current_consistency_token)
       end
 
       def domains
-        result = []
-        token  = nil
-        begin
-          response = sdb.list_domains(nil, token)
-          result.concat(response[:domains])
-          token = response[:next_token]
-        end while(token)
-        result
+        database.domains
       end
 
     private
-      # Returns the domain for the model
       def domain
-        @sdb_options[:domain]
+        @domain ||= database.domain(@sdb_options[:domain])
       end
 
-      #sets the conditions and order for the SDB query
-      def set_conditions_and_sort_order(query, sdb_type)
-        unsupported_conditions = []
-        conditions = ["simpledb_type = '#{sdb_type}'"]
-        # look for query.order.first and insure in conditions
-        # raise if order if greater than 1
-
-        if query.order && query.order.length > 0
-          query_object = query.order[0]
-          #anything sorted on must be a condition for SDB
-          conditions << "#{query_object.target.name} IS NOT NULL" 
-          order = "ORDER BY #{query_object.target.name} #{query_object.operator}"
-        else
-          order = ""
-        end
-        query.conditions.each do |op|
-          case op.slug
-          when :regexp
-            unsupported_conditions << op
-          when :eql
-            conditions << if op.value.nil?
-              "#{op.subject.name} IS NULL"
-            else
-              "#{op.subject.name} = '#{op.value}'"
-            end
-          when :not then
-            comp = op.operands.first
-            if comp.slug == :like
-              conditions << "#{comp.subject.name} not like '#{comp.value}'"
-              next
-            end
-            case comp.value
-            when Range, Set, Array, Regexp
-              unsupported_conditions << op
-            when nil
-              conditions << "#{comp.subject.name} IS NOT NULL"
-            else
-              conditions << "#{comp.subject.name} != '#{comp.value}'"
-            end
-          when :gt then conditions << "#{op.subject.name} > '#{op.value}'"
-          when :gte then conditions << "#{op.subject.name} >= '#{op.value}'"
-          when :lt then conditions << "#{op.subject.name} < '#{op.value}'"
-          when :lte then conditions << "#{op.subject.name} <= '#{op.value}'"
-          when :like then conditions << "#{op.subject.name} like '#{op.value}'"
-          when :in
-            case op.value
-            when Array, Set
-              values = op.value.collect{|v| "'#{v}'"}.join(',')
-              values = "'__NULL__'" if values.empty?                       
-              conditions << "#{op.subject.name} IN (#{values})"
-            when Range
-              if op.value.exclude_end?
-                unsupported_conditions << op
-              else
-                conditions << "#{op.subject.name} between '#{op.value.first}' and '#{op.value.last}'"
-              end
-            else
-              raise ArgumentError, "Unsupported inclusion op: #{op.value.inspect}"
-            end
-          when :or
-              # TODO There's no reason not to support OR
-              unsupported_conditions << op
-          else raise "Invalid query op: #{op.inspect}"
-          end
-        end
-        [conditions,order,unsupported_conditions]
+      # Returns the domain for the model
+      def domain_name
+        @sdb_options[:domain]
       end
-      
-      #gets all results or proper number of results depending on the :limit
-      def get_results(query, conditions, order)
-        fields_to_request = query.fields.map{|f| f.field}
-        fields_to_request << DmAdapterSimpledb::Record::METADATA_KEY
-        
-        selection = SDBTools::Selection.new(
-          sdb,
-          domain,
-          :attributes => fields_to_request)
-
-        if query.order && query.order.length > 0
-          query_object = query.order[0]
-          #anything sorted on must be a condition for SDB
-          conditions << "#{query_object.target.name} IS NOT NULL"
-          selection.order_by = query_object.target.name
-          selection.order    = case query_object.operator
-                               when :asc then :ascending
-                               when :desc then :descending
-                               else raise "Unrecognized sort direction"
-                               end
-        end
-        selection.conditions = conditions.compact.inject([]){|conds, cond|
-          conds << "AND" unless conds.empty?
-          conds << cond
-        }
-        if query.limit.nil?
-          selection.limit = :none
-        else
-          selection.limit = query.limit
-        end
-        unless query.offset.nil?
-          selection.offset = query.offset
-        end
-
-        items = []
-        time = Benchmark.realtime do
-          # TODO update Record to be created from name/attributes pair
-          selection.each do |name, value| 
-            items << {name => value}
-          end
-        end
-        DataMapper.logger.debug(format_log_entry(selection.to_s, time))
 
-        items
-      end
-      
       # Creates an item name for a query
       def item_name_for_query(query)
         sdb_type = simpledb_type(query.model)
@@ -313,23 +192,23 @@ module DataMapper
         selectors = [ :gt, :gte, :lt, :lte, :not, :like, :in ]
         return (selectors - conditions).size != selectors.size
       end
+
+      def database
+        options = sdb ? {:sdb_interface => sdb} : {}
+        @database ||= SDBTools::Database.new(
+          @sdb_options[:access_key], 
+          @sdb_options[:secret_key],
+          options)
+      end
       
       # Returns an SimpleDB instance to work with
       def sdb
-        access_key = @sdb_options[:access_key]
-        secret_key = @sdb_options[:secret_key]
-        @sdb ||= RightAws::SdbInterface.new(access_key,secret_key,@sdb_options)
-        @sdb
+        @sdb ||= (@null_mode ? NullSdbInterface.new(logger) : nil)
       end
       
-      def format_log_entry(query, ms = 0)
-        'SDB (%.1fs)  %s' % [ms, query.squeeze(' ')]
-      end
-
       def update_consistency_token
         @current_consistency_token = UUIDTools::UUID.timestamp_create.to_s
-        sdb.put_attributes(
-          domain, 
+        domain.put(
           '__dm_consistency_token', 
           {'__dm_consistency_token' => [@current_consistency_token]})
       end
@@ -368,6 +247,92 @@ module DataMapper
         end
       end
 
+      # WARNING This method updates +query+ as a side-effect
+      def selection_from_query(query)
+        query.update(extra_conditions(query))
+        where_expression  = 
+          DmAdapterSimpledb::WhereExpression.new(query.conditions, :logger => logger)
+        selection_options = {
+          :attributes => fields_to_request(query),
+          :conditions => where_expression,
+          :batch_limit => batch_limit,
+          :limit      => query_limit(query),
+          :logger     => logger
+        }
+        selection_options.merge!(sort_instructions(query))
+        selection = domain.selection(selection_options)
+        selection.offset = query.offset unless query.offset.nil?
+        query.clear
+        query.update(:conditions => where_expression.unsupported_conditions)
+        selection
+      end
+
+      def transaction(description, &block)
+        on_close = SDBTools::Transaction.log_transaction_close(logger)
+        SDBTools::Transaction.open(description, on_close, &block)
+      end
+
+      def fields_to_request(query)
+        fields = []
+        fields.concat(query.fields.map{|f| 
+            f.field if f.respond_to?(:field)
+          }.compact)
+        fields.concat(DmAdapterSimpledb::Record::META_KEYS)
+        fields.uniq!
+        fields
+      end
+
+      def sort_instructions(query)
+        direction = first_order_direction(query)
+        if direction
+          order_by = direction.target.field
+          order    = case direction.operator
+                     when :asc then :ascending
+                     when :desc then :descending
+                     else raise "Unrecognized sort direction"
+                     end
+          {:order_by => order_by, :order => order}
+        else
+          {}
+        end
+      end
+
+      def extra_conditions(query)
+        # SimpleDB requires all sort-by attributes to also be included in a
+        # predicate.
+        conditions = if (direction = first_order_direction(query))
+                       { direction.target.field.to_sym.not => nil }
+                     else
+                       {}
+                     end
+        table      = DmAdapterSimpledb::Table.new(query.model)
+        meta_key   = DmAdapterSimpledb::Record::METADATA_KEY
+
+        # The simpledb_type key is deprecated
+        old_table_key = DmAdapterSimpledb::Record::STORAGE_NAME_KEY
+
+        quoted_table_key = SDBTools::Selection.quote_name(old_table_key)
+        quoted_key = SDBTools::Selection.quote_name(meta_key)
+        conditions.merge!(
+          :conditions => [
+            "( #{quoted_key} = ? OR #{quoted_table_key} = ? )", 
+            table.token,
+            table.simpledb_type
+          ])
+        conditions
+      end
+
+      def query_limit(query)
+        query.limit.nil? ? :none : query.limit
+      end
+
+      # SimpleDB only supports a single sort-by field. Further sorting has to be
+      # handled locally.
+      def first_order_direction(query)
+        Array(query.order).first
+      end
+
+
     end # class SimpleDBAdapter