simple_record 2.0.0 → 2.0.1

data/README.markdown

@@ -19,7 +19,7 @@ Brought to you by: [![Appoxy](http://www.simpledeployr.com/images/global/appoxy-
         require 'simple_record'
     
         class MyModel < SimpleRecord::Base
-           has_attributes :name
+           has_strings :name
            has_ints :age
         end
 
@@ -45,25 +45,26 @@ More about ModelAttributes below.
         puts 'got=' + mm2.name + ' and he/she is ' + mm.age.to_s + ' years old'
         # Or more advanced queries? mms = MyModel?.find(:all, ["age=?", 32], :order=>"name", :limit=>10)
 
+That's literally all you need to do to get started. No database install, no other setup required.
 
 ## Attributes and modifiers for models
 
 NOTE: All objects will automatically have :id, :created, :updated attributes.
 
-### has_attributes
+### has_strings
 
 Add string attributes.
 
     class MyModel < SimpleRecord::Base
-      has_attributes :name
+      has_strings :name
     end
 
 ### has_ints, has_dates and has_booleans
 
-Lets simple_record know that certain attributes defined in has_attributes should be treated as integers, dates or booleans. This is required because SimpleDB only has strings so SimpleRecord needs to know how to convert, pad, offset, etc.
+This is required because SimpleDB only has strings so SimpleRecord needs to know how to convert, pad, offset, etc.
 
     class MyModel < SimpleRecord::Base
-      has_attributes :name
+      has_strings :name
       has_ints :age, :height
       has_dates :birthday
       has_booleans :is_nerd
@@ -75,7 +76,7 @@ Creates a many-to-one relationship. Can only have one per belongs_to call.
 
     class MyModel < SimpleRecord::Base
         belongs_to :school
-        has_attributes :name
+        has_strings :name
         has_ints :age, :height
         has_dates :birthday
         has_booleans :is_nerd
@@ -194,11 +195,21 @@ This is most helpful on windows so Rails doesn't need sqlite or mysql gems/drive
 
 Typical databases support BLOB's and/or CLOB's, but SimpleDB has a 1024 character per attribute maximum so larger
 values should be stored in S3. Fortunately SimpleRecord takes care of this for you by defining has_clobs for a large
-string value.
+string value. There is no support for blobs yet.
 
     has_clobs :my_clob
 
-These clob values will be stored in s3 under a bucket named: "#{aws_access_key}_lobs"
+These clob values will be stored in s3 under a bucket named "#{aws_access_key}_lobs"
+OR "simple_record_#{aws_access_key}/lobs" if you set :new_bucket=>true in establish_connection (RECOMMENDED).
+
+If you have more than one clob on an object and if it makes sense for performance reasons, you can set a configuration option on the class to store all clobs
+as one item on s3 which means it will do a single put to s3 and a single get for all the clobs on the object.
+This would generally be good for somewhat small clob values or when you know you will always be accessing
+all the clobs on the object. 
+
+    sr_config :single_clob=>true
+
+Setting this will automatically use :new_bucket=>true as well.
 
 ## Tips and Tricks and Things to Know
 
@@ -236,20 +247,36 @@ or
 
     o.something_id = x
 
-### Batch Save
+Accessing the id can prevent a database call so if you only need the ID, then you
+should use this.
+
+## Batch Save
 
 To do a batch save using SimpleDB's batch saving feature to improve performance, simply create your objects, add them to an array, then call:
 
     MyClass.batch_save(object_list)
 
+## Batch Delete
+
+To do a batch delete using SimpleDB's batch delete feature to improve performance, simply create your objects, add them to an array, then call:
+
+    MyClass.batch_delete(object_list or list_of_ids)
+
+## Operations across a Query
+
+    MyClass.delete_all(find_options)
+    MyClass.destroy_all(find_options)
+
+find_options can include anything you'd add after a find(:all, find_options) including :conditions, :limit, etc.
+
 ## Caching
 
 You can use any cache that supports the ActiveSupport::Cache::Store interface.
 
     SimpleRecord::Base.cache_store = my_cache_store
 
-If you want a simple in memory cache store, try: <http://gemcutter.org/gems/local_cache>. It supports max cache size and
-timeouts. You can also use memcached or http://www.quetzall.com/cloudcache.
+If you want a simple in memory cache store that supports max cache size and expiration, try: <http://gemcutter.org/gems/local_cache>.
+You can also use memcached or http://www.quetzall.com/cloudcache.
 
 ## Encryption
 
@@ -286,6 +313,11 @@ The :shards function should return a list of shard names, for example: ['CA', 'F
 
 The :map function should return which shard name the object should be stored to.
 
+When executing a find() operation, you can explicitly specify the shard(s) you'd like to find on. This is
+particularly useful if you know in advance which shard the data will be in.
+
+    MyClass.find(:all, :conditions=>....., :shard=>["CA", "FL"])
+
 You can see some [example classes here](https://github.com/appoxy/simple_record/blob/master/test/my_sharded_model.rb).
 
 ## Kudos

data/lib/simple_record.rb

@@ -434,6 +434,7 @@ module SimpleRecord
                         options[:domain] = sharded_domain
                     end
 
+                    # todo: Instead of doing the domain_ok, below, pass in the new option to aws lib :create_domain=>true, does the same thing now
                     if super(options)
                         self.class.cache_results(self)
                         delete_niled(to_delete)
@@ -470,30 +471,51 @@ module SimpleRecord
         def save_lobs(dirty=nil)
 #            puts 'dirty.inspect=' + dirty.inspect
             dirty = @dirty if dirty.nil?
+            all_clobs   = {}
+            dirty_clobs = {}
             defined_attributes_local.each_pair do |k, v|
+                # collect up the clobs in case it's a single put
                 if v.type == :clob
-#                    puts 'storing clob '
+                    val          = @lobs[k]
+                    all_clobs[k] = val
                     if dirty.include?(k.to_s)
-                        begin
-                            val = @lobs[k]
-#                            puts 'val=' + val.inspect
-                            s3_bucket.put(s3_lob_id(k), val)
-                        rescue Aws::AwsError => ex
-                            if ex.include? /NoSuchBucket/
-                                s3_bucket(true).put(s3_lob_id(k), val)
-                            else
-                                raise ex
-                            end
-                        end
-                        SimpleRecord.stats.s3_puts += 1
+                        dirty_clobs[k] = val
                     else
 #                        puts 'NOT DIRTY'
                     end
 
                 end
             end
+            if dirty_clobs.size > 0
+                if self.class.get_sr_config[:single_clob]
+                    # all clobs in one chunk
+                    # using json for now, could change later
+                    val = all_clobs.to_json
+                    puts 'val=' + val.inspect
+                    put_lob(single_clob_id, val, :new_bucket=>true)
+                else
+                    dirty_clobs.each_pair do |k, val|
+                        put_lob(s3_lob_id(k), val)
+                    end
+                end
+            end
+
+        end
+
+        def put_lob(k, val, options={})
+            begin
+                s3_bucket(false, options).put(k, val)
+            rescue Aws::AwsError => ex
+                if ex.include? /NoSuchBucket/
+                    s3_bucket(true, options).put(k, val)
+                else
+                    raise ex
+                end
+            end
+            SimpleRecord.stats.s3_puts += 1
         end
 
+
         def is_dirty?(name)
             # todo: should change all the dirty stuff to symbols?
 #            puts '@dirty=' + @dirty.inspect
@@ -509,8 +531,15 @@ module SimpleRecord
             Aws::S3.new(SimpleRecord.aws_access_key, SimpleRecord.aws_secret_key)
         end
 
-        def s3_bucket(create=false)
-            s3.bucket(s3_bucket_name, create)
+        # options:
+        #   :new_bucket => true/false. True if want to use new bucket. Defaults to false for backwards compatability.
+        def s3_bucket(create=false, options={})
+            s3.bucket(options[:new_bucket] || SimpleRecord.options[:new_bucket] ? s3_bucket_name2 : s3_bucket_name, create)
+        end
+
+        # this is the bucket that will be used going forward for anything related to s3
+        def s3_bucket_name2
+            "simple_record_#{SimpleRecord.aws_access_key}"
         end
 
         def s3_bucket_name
@@ -521,6 +550,10 @@ module SimpleRecord
             self.id + "_" + name.to_s
         end
 
+        def single_clob_id 
+            "lobs/#{self.id}_single_clob"
+        end
+
         def save!(options={})
             save(options) || raise(RecordNotSaved)
         end
@@ -639,6 +672,14 @@ module SimpleRecord
             results
         end
 
+        # Pass in an array of objects
+        def self.batch_delete(objects, options={})
+            if objects
+                # 25 item limit, we should maybe handle this limit in here.
+                connection.batch_delete_attributes @domain, objects.collect { |x| x.id }
+            end
+        end
+
         #
         # Usage: ClassName.delete id
         #
@@ -646,9 +687,10 @@ module SimpleRecord
             connection.delete_attributes(domain, id)
         end
 
-        def self.delete_all(*params)
+        # Pass in the same OPTIONS you'd pass into a find(:all, OPTIONS)
+        def self.delete_all(options)
             # could make this quicker by just getting item_names and deleting attributes rather than creating objects
-            obs = self.find(params)
+            obs = self.find(:all, options)
             i   = 0
             obs.each do |a|
                 a.delete
@@ -657,8 +699,9 @@ module SimpleRecord
             return i
         end
 
-        def self.destroy_all(*params)
-            obs = self.find(params)
+        # Pass in the same OPTIONS you'd pass into a find(:all, OPTIONS)
+        def self.destroy_all(options)
+            obs = self.find(:all, options)
             i   = 0
             obs.each do |a|
                 a.destroy
@@ -681,7 +724,6 @@ module SimpleRecord
         end
 
 
-
         def delete_niled(to_delete)
 #            puts 'to_delete=' + to_delete.inspect
             if to_delete.size > 0

data/lib/simple_record/active_sdb.rb

@@ -631,10 +631,10 @@ module SimpleRecord
                 # This will currently return and's, or's and betweens. Doesn't hurt anything, but could remove.
                 def parse_condition_fields(conditions)
                     return nil unless conditions && conditions.present?
-                    rx     = /\b(\w*)[\s|>=|<=|!=|=|>|<|like]/
+                    rx     = /\b(\w*)[\s|>=|<=|!=|=|>|<|like|between]/
                     fields = conditions[0].scan(rx)
 #                    puts 'condition_fields = ' + fields.inspect
-                    fields[0]
+                    fields.flatten
                 end
 
             end

data/lib/simple_record/attributes.rb

@@ -22,6 +22,17 @@ module SimpleRecord
         module ClassMethods
 
 
+            # Add configuration to this particular class.
+            #   :single_clob=> true/false. If true will store all clobs as a single object in s3. Default is false.
+            def sr_config(options={})
+                get_sr_config
+                @sr_config.merge!(options)
+            end
+
+            def get_sr_config
+                @sr_config ||= {}
+            end
+
             def defined_attributes
                 @attributes ||= {}
                 @attributes
@@ -329,15 +340,34 @@ module SimpleRecord
                 end
                 # get it from s3
                 unless new_record?
-                    begin
-                        ret                        = s3_bucket.get(s3_lob_id(name))
-#                        puts 'got from s3 ' + ret.inspect
-                        SimpleRecord.stats.s3_gets += 1
-                    rescue Aws::AwsError => ex
-                        if ex.include? /NoSuchKey/
-                            ret = nil
-                        else
-                            raise ex
+                    if self.class.get_sr_config[:single_clob]
+                        begin
+                            single_clob = s3_bucket(false, :new_bucket=>true).get(single_clob_id)
+                            single_clob = JSON.parse(single_clob)
+                            puts "single_clob=" + single_clob.inspect
+                            single_clob.each_pair do |name2,val|
+                                @lobs[name2.to_sym] = val
+                            end
+                            ret = @lobs[name]
+                            SimpleRecord.stats.s3_gets += 1
+                        rescue Aws::AwsError => ex
+                            if ex.include? /NoSuchKey/
+                                ret = nil
+                            else
+                                raise ex
+                            end
+                        end
+                    else
+                        begin
+                            ret = s3_bucket.get(s3_lob_id(name))
+                            # puts 'got from s3 ' + ret.inspect
+                            SimpleRecord.stats.s3_gets += 1
+                        rescue Aws::AwsError => ex
+                            if ex.include? /NoSuchKey/
+                                ret = nil
+                            else
+                                raise ex
+                            end
                         end
                     end
 

data/test/my_model.rb

@@ -8,7 +8,7 @@ class MyModel < MyBaseModel
     has_booleans :cool
     has_dates :birthday, :date1, :date2, :date3
 
-    has_clobs :clob1
+    has_clobs :clob1, :clob2
 
     #callbacks
     before_create :set_nickname
@@ -46,4 +46,15 @@ class MyModel < MyBaseModel
         @@attributes
     end
 
-end
+end
+
+
+
+class SingleClobClass < SimpleRecord::Base
+
+    sr_config :single_clob=>true
+
+    has_strings :name
+
+    has_clobs :clob1, :clob2
+end

data/test/test_lobs.rb

@@ -1,13 +1,12 @@
 require 'test/unit'
-require File.join(File.dirname(__FILE__), "/../lib/simple_record")
-require File.join(File.dirname(__FILE__), "./test_helpers")
-require File.join(File.dirname(__FILE__), "./test_base")
+require_relative "../lib/simple_record"
+require_relative "test_helpers"
+require_relative "test_base"
 require "yaml"
 require 'aws'
-require 'my_model'
-require 'my_child_model'
-require 'model_with_enc'
-require 'active_support'
+require_relative 'my_model'
+require_relative 'my_child_model'
+require_relative 'model_with_enc'
 
 # Tests for SimpleRecord
 #
@@ -15,36 +14,81 @@ require 'active_support'
 class TestLobs < TestBase
 
 
-    def test_blobs
+    def test_clobs
         mm = MyModel.new
 
         puts mm.clob1.inspect
         assert mm.clob1.nil?
-        
-        mm.name = "whatever"
-        mm.age = "1"
+
+        mm.name  = "whatever"
+        mm.age   = "1"
         mm.clob1 = "0" * 2000
         assert SimpleRecord.stats.s3_puts == 0
         puts mm.inspect
         mm.save
 
+        assert SimpleRecord.stats.s3_puts == 1
         sleep 2
 
+        mm.clob1 = "1" * 2000
+        mm.clob2 = "2" * 2000
+        mm.save
+        assert SimpleRecord.stats.s3_puts == 3
+
         mm2 = MyModel.find(mm.id)
         assert mm.id == mm2.id
         puts 'mm.clob1=' + mm.clob1.to_s
         puts 'mm2.clob1=' + mm2.clob1.to_s
         assert mm.clob1 == mm2.clob1
-        assert SimpleRecord.stats.s3_puts == 1, "puts is #{SimpleRecord.stats.s3_puts}"
+        assert SimpleRecord.stats.s3_puts == 3, "puts is #{SimpleRecord.stats.s3_puts}"
         assert SimpleRecord.stats.s3_gets == 1, "gets is #{SimpleRecord.stats.s3_gets}"
         mm2.clob1 # make sure it doesn't do another get
         assert SimpleRecord.stats.s3_gets == 1
 
+        assert mm.clob2 == mm2.clob2
+        assert SimpleRecord.stats.s3_gets == 2
+
         mm2.save
 
         # shouldn't save twice if not dirty
-         assert SimpleRecord.stats.s3_puts == 1
+        assert SimpleRecord.stats.s3_puts == 3
+
+    end
+
+    def test_single_clob
+        mm = SingleClobClass.new
+
+        puts mm.clob1.inspect
+        assert mm.clob1.nil?
+
+        mm.name  = "whatever"
+        mm.clob1 = "0" * 2000
+        mm.clob2 = "2" * 2000
+        assert SimpleRecord.stats.s3_puts == 0
+        puts mm.inspect
+        mm.save
 
+        assert SimpleRecord.stats.s3_puts == 1
+
+        sleep 2
+
+        mm2 = SingleClobClass.find(mm.id)
+        assert mm.id == mm2.id
+        puts 'mm.clob1=' + mm.clob1.to_s
+        puts 'mm2.clob1=' + mm2.clob1.to_s
+        assert_equal mm.clob1, mm2.clob1
+        assert SimpleRecord.stats.s3_puts == 1, "puts is #{SimpleRecord.stats.s3_puts}"
+        assert SimpleRecord.stats.s3_gets == 1, "gets is #{SimpleRecord.stats.s3_gets}"
+        mm2.clob1 # make sure it doesn't do another get
+        assert SimpleRecord.stats.s3_gets == 1
+
+        assert mm.clob2 == mm2.clob2
+        assert SimpleRecord.stats.s3_gets == 1
+
+        mm2.save
+
+        # shouldn't save twice if not dirty
+        assert SimpleRecord.stats.s3_puts == 1
     end
 
 end

metadata

@@ -5,8 +5,8 @@ version: !ruby/object:Gem::Version
   segments: 
   - 2
   - 0
-  - 0
-  version: 2.0.0
+  - 1
+  version: 2.0.1
 platform: ruby
 authors: 
 - Travis Reeder
@@ -16,7 +16,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-12-07 00:00:00 -08:00
+date: 2010-12-22 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency