redisted 0.0.1

data/readme.md

@@ -0,0 +1,643 @@
+PLEASE NOTE
+===========
+
+This is still ***UNDER DEVELOPMENT*** and not yet ready for prime-time. Feel free to fool around with it, file bug
+reports (via GitHub), and make suggestions. It is a work in progress. However, I do hope to have a fully
+flushed out implementation as soon as possible.
+
+
+Redisted
+========
+
+Redisted provides higher level model functionality for redis databases. It is not intended as an ActiveRecord
+plugin. If you want to use ActiveRecord, use a SQL database or something like Mongoid. Redisted is designed
+to provide model symatics but customized and optimized for redis data store.
+
+Do not expect ActiveRecord compatibility, but expect ActiveRecord-like capabilities highly taylored and customized to
+take best advantage of Redis.
+
+Installing
+==========
+
+TODO
+
+Configuring
+===========
+
+TODO
+
+Creating Models
+===============
+A basic Redisted model looks like this:
+
+    class MyModel < Redisted::Base
+    end
+
+Within the model, you specify fields, indices, relations, scopes, and additional model functionality, just like
+you do with ActiveRecord models.
+
+Instantiating Instances
+=======================
+Creating an instance is as simple as new or create:
+
+    # Creating an instance in memory only
+    obj=MyModel.new
+
+    # Creating and persisting an instance to Redis:
+    obj=MyModel.create
+    puts obj.id # <<<---An 'id' is created when it is persisted to Redis.
+
+    # Creating in memory, then later saving to Redis:
+    obj=MyModel.new
+    ...
+    obj.save
+    puts obj.id # <<<---The 'id' was created at time of 'save'
+
+Fields
+======
+Redis is a basic key/value store that is inheritantly schema-less. Redisted continues this by allowing the dynamic
+specification fields without having to create a static schema that must be migrated. Simply adding a field
+to the model specification makes it available. No database migration is necessary.
+
+To specify a field in Redisted, you use the field command. This command looks like this:
+
+    field :field_name,type: :field_type
+
+Field type can be one of :string, :integer, or :datetime. Here is an example class with fields defined:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :size, type: :integer
+      field :created, type: :datetime
+    end
+
+Instances of a Redisted model are stored in a single Redis hash. The name of the key is based on the name of the
+model and the ID associated with that model. For example, a MyModel instance (such as above), with an ID value
+of 1234 would be stored in a hash under the key:
+
+    mymodel:1234
+
+Individual fields within the hash would correspond to each field within the model. Each field is stored in Redis as
+a string, and Redisted deals with converting that string to/from the desired field types, defined above.
+
+Destroying an instance of a Redisted model is as simple as deleting the corresponding Redis key.
+
+Reading/Writing Fields
+----------------------
+Reading/writing a value to a field is as simple as using the included accessors:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :size, type: :integer
+      field :created, type: :datetime
+    end
+
+    obj=MyModel.create
+    obj.name="My Name"
+    puts "Name: #{obj.name}"
+
+If the object is persisted to Redis (using 'create' to make the object or a previous call to 'save', and therefore it has an 'id'), then
+by default reading/writing attributes read and write the value directly from Redis. For instance, using the above model:
+
+    obj=MyModel.create
+    obj.name="MyName" # Issues Redis call: HSET my_model[123] name "MyName"
+    puts "Name: #{obj.name}" # Issues Redis call: HGET my_model[123] name
+
+To save round trip latency to Redis when you are making several changes to an instance, you can cache the changes:
+
+    obj=MyModel.create
+    obj.cache do
+        obj.name="MyName"
+        obj.size=123
+        obj.size=obj.size+222
+    end # Issues Redis call: HMSET my_model[112334] name "MyName" size "345"
+
+Or:
+
+    obj=MyModel.create
+    obj.cache
+    obj.name="MyName"
+    obj.size=123
+    obj.size=obj.size+222
+    obj.save # Issues Redis call: HMSET my_model[112334] name "MyName" size "345"
+
+You can also pass in a hash to the create call:
+
+    obj=MyModel.create({name: "MyName", size: 123}) # Issue Redis call:HMSETmy_model[112334] name "MyName" size "123"
+
+When an object has not yet been persisted to disk (has no 'id'), this is the normal behavior:
+
+    obj=MyModel.new # <<<---Caching is on until the first save...
+    obj.name="MyName"
+    obj.size=123
+    obj.size=obj.size+222
+    obj.save # Issues Redis call: HMSET my_model[123] name "MyName" size "345"
+    obj.name="MyName2" # <<<---This now reverts to normal, non-caching functionality and issues immediately: HSET my_model[123] name "MyName2"
+
+You can change the default persisting strategy for an entire class within the class definition:
+
+    class MyModel < Redisted::Base
+      always_cache_until_save # <<<--- Tells Redisted to always cache each change until 'save' is called
+      field :name, type: :string
+      field :size, type: :integer
+      field :created, type: :datetime
+    end
+    obj=MyModel.create
+    obj.cache
+    obj.name="MyName"
+    obj.size=123
+    obj.size=obj.size+222
+    obj.save # Issues Redis call: HMSET my_model[123] name "MyName" size "345"
+
+For reading, the value is first checked to see if it has already been cached (by a previous read or write). If not, then
+it will be read directly from Redis:
+
+    obj=MyModel.find(123) #<<<--- Assumes an object with this 'id' was previously created
+    puts obj.name # Issues Redis call: HGET my_model[123] name
+    puts obj.name # Reads from cache...no Redis call made
+    obj.name="MyName" # Issues redis call: HSET my_model[123] name "MyName"
+    puts obj.name # Reads from cache...no Redis call made
+
+You can flush the cache and force a reread from Redis at any time:
+
+    obj=MyModel.find(123) #<<<--- Assumes an object with this 'id' was previously created
+    puts obj.name # Issues Redis call: HGET my_model[123] name
+    puts obj.name # Reads from cache...no Redis call made
+    obj.name="MyName" # Issues redis call: HSET my_model[123] name "MyName"
+    puts obj.name # Reads from cache...no Redis call made
+    obj.flush
+    puts obj.name # Issues Redis call: HGET my_model[123] name
+
+Cache Pre-Read
+--------------
+You can force an object to always pre-read all values into the cache at object create by specifying an object in the
+model class:
+
+    class MyModel < Redisted::Base
+      cache_on_first_get # <<<--- Tells Redisted to always cache when an object is instantiated from Redis
+      field :name, type: :string
+      field :size, type: :integer
+      field :created, type: :datetime
+    end
+    obj=MyModel.find(123) # Issues Redisc all: HMGET my_model[123] name size created
+    puts obj.name # Reads from cache...no Redis call made
+
+You can specify which keys you want to pre-read in a couple different ways:
+
+    pre_cache_all # Pre-read all keys
+    pre_cache_all keys: :all # Same as above
+    pre_cache_all keys: [:name,:size] # Only pre-read name &size, and not created (in this example)
+    pre_cache_all except: [:content] # Only pre-read name &size, and not created (in this example)
+
+You can specify when the pre-read occurs individually:
+
+    pre_cache_all when: :create        # Load the cache when the object is first created (via find, or any other way)
+    pre_cache_all when: :first_read    # Defer loading the cache until the first pre-cachable field is read from the object, then load the entire cache
+
+List of Fields
+--------------
+You can get a list of available fields from either a class or an object as follows:
+
+    obj=MyModel.new
+    field_list=MyModel.fields
+    or:
+    field_list=obj.fields
+
+This will return a hash with each key representing a field, and the value being a list of options provided during the
+field create. For instance:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :size, type: :integer, default: 0
+      field :created, type: :datetime
+    end
+    field_list=MyModel.fields
+
+This will produce the following hash:
+
+    {
+        name: {
+            type: :string
+        },
+        size: {
+            type: :integer,
+            default: 0
+        },
+        created: {
+            type: :datetime
+        },
+    }
+
+Note that there may be other values in field-specific hash, such as default option values, etc., and the list may
+change over time. You should only assume that values you specifically add to the field definition appear in this
+hash, and that other values may or may not be present.
+
+Basic Find
+==========
+
+The easiest way to locate and open an object from Redis is to use find. Find expects one parameter, the 'id' of the object
+you wish to open:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :size, type: :integer, default: 0
+      field :created, type: :datetime
+    end
+    obj=MyModel.find(1963)
+    puts obj.id # Returns 1963
+    puts obj.name # Returns the name of the object with 1963
+    ...
+
+You can also pass an array of 'id' values, and find will return an array of objects:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :size, type: :integer, default: 0
+      field :created, type: :datetime
+    end
+    objs=MyModel.find([1963,1982,1994])
+    puts objs[0].id # Returns 1963
+    puts objs[0].name # Returns the name of object with id 1963
+    puts objs[1].id # Returns 1982
+    puts objs[1].name # Returns the name of object with id 1982
+    puts objs[2].id # Returns 1994
+    puts objs[2].name # Returns the name of object with id 1994
+
+Delete and Destroy
+==================
+
+You can delete an object by calling the delete method on an instance:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :size, type: :integer, default: 0
+    end
+    obj=MyModel.create({name: "test", size: 15}) # Redis call: HMSET my_model:1122 name "test" size "15"
+    obj.delete # Redis call: DEL my_model:1122
+
+Destroy does the same thing, but you can specify standard callbacks to be run (see callbacks, below).
+
+Indices
+=======
+
+Indices in Redisted are different than in other similar packages. Indices are *required* in order to filter, sort, or
+determine uniqueness of items in a model. The entire filter/sort UI is dependent on the creation of indices.
+
+Unique Index
+------------
+
+A unique index is an index that specifies a field or group of fields who's value must be unique across all items
+of the given model. It is specified as such:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :integer
+
+      index :name, unique: true
+    end
+
+In this example, all instances of MyModel must maintain a unique "name" field. This information is persisted in
+Redis using a *SET*. The set is stored in a key specific to the model and the index. The redis key for the "name"
+unique index in Redis for the model above may look like this:
+
+    SET: my_model[name]
+
+Every time a new MyModel instance is persisted, a member is added to this SET. Everytime it is deleted, the corresponding
+member is deleted. Each time the value of "name" changes, the old member is removed and a new one is added. All of this
+occurs atomically using optimistic locking on this set key and the model hash key.
+
+If you try and create/persiste an instance of the model where the "name"is not unique, an exception will be raised.
+If you are performing saves one change at a time (the default), the check is made at the time when the assignment
+is made to the name field. If you are using cached writes, it is performed at the time of the save.
+
+The above syntax creates a unique index named "name" that is unique across the single field "name". You can specify
+unique indices in other ways as well:
+
+    index :ti, unique: :test_int
+      # Creates an index named "ti" that is unique across the field "test_int"
+    index :idxkey, unique: [:test_int,:name]
+      # Creates an index named "idxkey" that is unique across "test_int"/"name" field pair (the pair of values must be
+      # unique).
+
+The index name must be unique across all indexes for a given model.
+
+
+Filter/Sort Index
+-----------------
+A filter/sort index is an index that can be used to return a filtered set of items and/or sorted in a particular way.
+
+Here is a simple example:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :integer
+
+      index :odd, includes: ->(elem){(elem.test_int%2)!=0}
+      index :even, includes: ->(elem){(elem.test_int%2)==0}
+    end
+
+This model creates two indexes, one that contains a list of all model instances where test_int is odd, the other
+contains a list of all model instances where test_int is even.
+
+With the above specification, you can then use the following commands:
+
+    MyModel.by_odd.all    # Returns an array of all instances of MyModel where test_int is odd...
+    MyModel.by_even.all   # Returns an array of all instances of MyModel where test_int is even...
+
+The order of the returned results is not specified. Using this example, the indexes are maintained in SORTED
+SETS (ZADD/ZREM/etc.). The name of the keys are:
+
+    my_model[odd]
+    my_model[even]
+
+Each key will contain a member referring to the 'id' value of the instance that is contained in this index. For
+this example, the sort "score" is set to 1 (more on this later). Each time an instance is created or modified, the
+lambda function for the index specified in the model declation is executed. If the function returns true, then an
+entry for this instance is added to the index. If it is false, any existing entry for this instance is removed. When
+an instance is deleted, it's entry (if it exists) is deleted.
+
+You may specify a sort index as follows:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :integer
+
+      index :asc, order: ->(elem){elem.test_int},fields: :test_int
+    end
+
+This creates an index named asc, that is implemented as a SORTED SET with the following key:
+
+    my_model[asc]
+
+In this case, *all* instances of MyModel have their 'id' value inserted as a member in this set, and their sort
+"weight" is set to the value returned by the lambda function (order:).
+
+With the above specification, when you use the index like so:
+
+    MyModel.by_asc.all
+
+You will get an array of all objects in the model, sorted in ascending order by "test_int".
+
+Note that the lambda function can perform any calculation and return any result. For a reverse sort, you can simply
+put a minus sign in front of the equation.
+
+While the lambda can contain any information, the value of the Redis "weight" (as returned by the method) is only
+recalculated if one of the fields specified in the fields: parameter (which may be an array of fields) has changed
+values. Thus, it's important to include in this array all fields that are used for the order calculation.
+
+An index may be both a filter and a sort index, by combining the syntax:
+
+    index :sortedodd,includes: ->(elem){(elem.test_int%2)!=0}, order: ->(elem){elem.test_int}, fields: :test_int
+
+In this case, the entry will only be included in the index if the "includes" lambda returns true, and when inserted
+it will be given a sort "weight" of the value returned by the "order" lambda. The "weight"is only recalculated if
+the "test_int" field changes value.
+
+When using indexes, you can mix and match them in a single call. For instance, using all of the above indexes, you can
+perform the following lookups:
+
+    MyModel.by_odd.all
+      # Returns an array of all instances of MyModel where test_int is odd...
+    MyModel.by_asc.all
+      # Returns all instances of MyModel sorted by test_int.
+    MyModel.by_odd.by_asc.all
+      # Returns all instances of MyModel where test_int is odd, sorted by test_int.
+    MyModel.by_sortedodd.all
+      # Same as above: Returns all instances of MyModel where test_int is odd, sorted by test_int.
+
+Match Index
+-----------
+
+The above indexes provide a very high performant way of doing static filters. It works well when the specific
+queries are known in advance. This is because the indexes contain a specific set of static keys, and membership in the
+index is determined at the time the model is saved.
+
+When possible, these indexes are efficient and powerful. However, it's not always possible to do a query this way.
+
+That is why there are match indexes. A match index is used when a value needed for the query is not known ahead
+of time (at model save time). Here is the declaration of a match index:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :string
+      field :provider, type: :string
+
+      index :provider, match: ->(elem){(elem.provider)}
+    end
+
+With an index like this, you can perform queries such as the following:
+
+    a=MyModel.by_provider("cable").all
+    or:
+    a=MyModel.by_provider("satellite").all
+
+At model creation/save time, several SORTED SETS are created for this index, one for each unique value returned
+by the various objects when they call the "match" lambda. For instance, in the above example, if the only values
+that "provider" is ever set to are "cable", "satellite", or "overair", then the following keys are generated:
+
+    my_model[provider]:cable
+    my_model[provider]:satellite
+    my_model[provider]:overair
+
+Each key is a SORTED SET who's members are the 'id' values of objects where the "match" lambda returns the specified
+value. So, when you call 'by_provider("cable")', you include the 'my_model[provider]:cable' key in the query.
+
+The individual keys are created the first time a value is returned by the "match" function (i.e., when the first
+member is inserted). If a query is run for a value that does not yet have a key, such as the following:
+
+    MyModel.by_provider("xyzzy").all
+
+then the query will act as if it returns an empty set (which it is).
+
+
+
+Incomplete Queries
+------------------
+Just like in ActiveRecord, you can store intermediate results of a nested query, and use the results later. For
+instance, each of these does the exact same thing:
+
+In line:
+
+    MyModel.by_odd.by_asc.all
+
+Stored in a variable each step:
+
+    a=MyModel.by_odd
+    a=a.by_asc
+    a.all
+
+Storing the query and processing it later:
+
+    a=MyModel.by_odd.by_asc
+    ...
+    a.all
+
+Starting with a blank query:
+
+    a=MyModel.scoped
+    a=a.by_odd
+    a=a.by_asc
+    a.all
+
+The query is performed by calling ZINTERSTORE on each of the sorted sets. However, these ZINTERSTORE calls are not
+performed until the ".all" is executed. This way you can setup a query in a controller, for instance, and it only
+will execute if the view issues the ".all" on it.
+
+Besides ".all", you can also use ".each":
+
+    MyModel.by_odd.by_asc.each do |model|
+      # 'model' contains each instance that matches the query
+    end
+
+
+Updating An Index
+-----------------
+Given how indexes are created and handled, it is recommended that you do *not* modify the lambda function of an
+index after the index has been created and populated. Doing so is inviting problems, since the index is not
+regenerated when the function changes.
+
+If you must change the meaning of an index, we recommend you actually create a new index and begin using that, then
+obsolete the old one when it is not used any more. Using scopes (discussed below) rather than raw index can help
+make this easier.
+
+If you *must* change the lambda function, then you should rebuild the indexes for the model from scratch immediately
+after changing the function.
+
+From the command line:
+
+    rake redisted:recalculate_all
+
+Programatically (one model):
+
+    MyModel.recalculate
+
+Programatically (all models):
+
+    Redisted::recalculate_all
+
+For each index, this will cause the old index to be destroyed, and a new one to be regenerated. Depending on the
+number and complexity of the indexes, and the number of objects in the model, this could take some time. Since
+indexes in redisted are *required* to do queries, rather than simply providing performance enhancements, access
+to the query functionality of the model is offline while this reindex occurs, and hence should be performed during
+a maintenance window.
+
+TODO: Need a way to specify to *pause* query calls rather than failing them during an index recalcuate.
+
+TODO: Note that this entire recalculate interfact does not yet exist.
+
+Index Uniqueness
+----------------
+
+The names of the indexes must be unique across all indexes for this model. This goes for unique, filter/sort,
+and 'match' indexes.
+
+Scopes
+======
+
+Using scopes, you can specify a complex query and refer to it using a simple identifer. For example, combining
+much of the above:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :integer
+      field :provider, type: :string
+
+      index :odd, includes: ->(elem){(elem.test_int%2)!=0}
+      index :even, includes: ->(elem){(elem.test_int%2)==0}
+      index :provider, match: ->(elem){(elem.provider)}
+      index :asc, order: ->(elem){elem.test_int},fields: :test_int
+
+      scope :sorted_odd, ->{by_odd.by_asc}
+      scope :odd_providers, ->(name){by_odd.provider(name)}
+    end
+
+Then, you can use them in queries:
+
+    MyModel.sorted_odd.all
+    MyModel.odd_providers.all
+
+Validations
+===========
+
+Standard ActiveModel validations work with Redisted. Such as:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :integer
+      field :provider, type: :string
+
+      validates_length_of :name, :minumum=>5
+      validates_length_of :provider, :maximum=>20
+    end
+
+TODO:Note that "validates_uniqueness_of" is not yet supported...
+
+References (aka Relationships)
+==============================
+Redisted is great for creating models that are "in between" other non-Redisted models, such as ActiveRecord models or
+Mongoid models. Redis in general is great for creating integrated maps of values, such as object references. In fact,
+Redisted was created out of a need for a very fast way of assocating large number of 'tags' to a large number of
+'messages' stored in Mongoid very efficiently.
+
+As such, the relation mechanims in Redisted are optimized for creating not only references to other Redisted models,
+but to models of other backend stores, such as ActiveRecord and Mongoid.
+
+There are only two reference types, references_one and references_many. Here is an example:
+
+    class MyModel < Redisted::Base
+      field :name, type: :string
+      field :test_int, type: :integer
+      field :provider, type: :string
+
+      references_one :user
+      references_many :message
+
+      references_one :another_class, as: :aclass
+      references_many :another_class, as: :theclasses
+    end
+
+The "references_one" creates a reference to a single object of another model. The model does not have to be
+a redisted model (but it can be), it can be ActiveRecord, Mongoid, anything that accepts a ".find(id)" call.
+
+With a references_one field, you can:
+
+    u=ARUser.create
+    mm=MyModel.create
+
+    mm.user=u
+    or:
+    mm.user_id=u.id
+
+    u=mm.user # Returns an instance of ARUser
+    u=mm.user.xxx # Call method 'xxx' on the instance of ARUser
+
+With a refererences_many field, you can:
+
+    m1=ARMessage.create
+    m2=ARMessage.create
+    mm=MyModel.create
+
+    mm.message << m1
+    mm.message << m2
+
+    mm.message[0] # Returns an instance of ARMessage (m1)
+    mm.message[1] # Returns an instance of ARMessage (m2)
+
+This all will work as is, without any changes to the ARUser model. However, you will probably want to install
+a destroy callback in ARUser so that the reference is removed if 'u' is destroyed. You can do that using
+the following code:
+
+    class ARUser
+      Redisted::on_destroy :my_model,:user
+    end
+    class ARMessage
+      Redisted::on_destroy :my_model,:message
+
+
+Callbacks
+=========
+
+Callbacks work the same as ActiveRecord. Redisted supports callbacks on create, update, save, and destroy.
+For update and save, the callbacks are called anytime a value is written to Redisted. So, if the model is setup
+so that each field update forces a write to Redisted (the default), then these two callbacks are called on each
+field update.

data/redisted.gemspec

@@ -0,0 +1,23 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "redisted/version"
+
+Gem::Specification.new do |s|
+  s.name        = "redisted"
+  s.version     = Redisted::VERSION
+  s.authors     = ["Lee Atchison"]
+  s.email       = ["lee@leeatchison.com"]
+  s.homepage    = ""
+  s.summary     = %q{Rails models based on Redis.}
+  s.description = %q{Rails models based Redis.}
+
+  s.rubyforge_project = "redisted"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  # specify any dependencies here; for example:
+  s.add_runtime_dependency "redis"
+end