sadie 0.1.8 → 0.1.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 3d35237ce7365435d796e102a1e8d8e2455d9363
-  data.tar.gz: 03e110b3da7e3ab6d6ad0445e3a7c6ce357fdb55
+  metadata.gz: c2491a41d00ecae062b1fc3b7c4d0ae719b6cfeb
+  data.tar.gz: e041a331e9cbaaa65927628590191188d913d906
 SHA512:
-  metadata.gz: 6f148d1da7eb8101b19d167ffb72fa7b1b0918093254fe62eaea3c915322b8658646150bbd89239a6ad31e29d73569977dbd9a4d5a05716f4eb2b351a343e6a6
-  data.tar.gz: 8e849601b5ea45613827de33250b67f31cc980aed5d306f6b051bc9438416b9f5a48f0f8f57d6795c53a48f7e11604308a8ddcbc55b8dd39fd2e0d0fc6d69b02
+  metadata.gz: b884cc2dc571abb642daaf670dc264114d668b9799f108b32c1494e600f37f367fdb682872a48f999592ebc2a273f7cadefa789009dcfabc54e25eab7e42b919
+  data.tar.gz: 760fac48dbf4a93de8b9dc6d076c40e5219a1b5e68faa82ec59e05913adc5de888fdde220231b80397d9a33d0ed88234d51afdd46e6302bd0060201f4bf601fe

data/CHANGELOG

@@ -26,4 +26,5 @@
 [0.0.51] eacher bugfix. now correctly handles specific keys.
 [0.0.52] code cleanup
 [0.1.01] ** ground-up rewrite, ignore everything before this line if you're not a historian **
-[0.1.8] new version now at capability pairing with old version.
+[0.1.8] new version now at capability pairing with old version.
+[0.1.9] added metadata for keys, abstracted expiry and refresh data structs into timestamp queues, abstracted various mutexes in session into lock manager object in preparation for distributed behavior

data/README

@@ -1,12 +1,12 @@
 ==About Sadie
 
-Sadie is a general-purpose data server written in Ruby for other Rubyists.
+Sadie is a general-purpose data server written in Ruby for other Rubyists.  It is designed to be fast, but also resource efficient, storing data in memory or, optionally and on a key-by-key basis, in disk-based storage.
 
 ==Purpose
 Imagine you work in the IT department of a cell phone company and your boss asks you to design a system that will:
-  * build a status page that shows a map with the operational status of all the cell towers and, for each one, the distance to the nearest field technician
-  * generate a shapefile and a spreadsheet of the same data that will be available and up-to-date on the company's intranet, 24/7
-  * when there's an outtage, send an SMS alert to the nearest field tech to the tower with the problem
+* build a status page that shows a map with the operational status of all the cell towers and, for each one, the distance to the nearest field technician
+* generate a shapefile and a spreadsheet of the same data that will be available and up-to-date on the company's intranet, 24/7
+* when there's an outtage, send an SMS alert to the nearest field tech to the tower with the problem
 
 This is all inter-related data and, for a developer, it's not all that difficult to start thinking about how to approach these problems.
 
@@ -23,6 +23,8 @@ For data that changes often, it's possible to expire data after a set amount of
 
 Primers can also reference other primers so it's easy to hide complexity.
 
+For large, less frequently accessed data, it is possible to specfiy that the value be stored to disk instead of memory.
+
 ==How to use it
 Create a directory for sadie to operate in, say /var/sadie.
 
@@ -36,8 +38,8 @@ In the primers directory, create files that end in .rb that look like:
   prime ["test.expires.nsecs"] do
   
     expire :never
-    refresh 300 #generate this again every 5 minutes
-    
+    refresh 300       # generate this again every 5 minutes
+    store_in :memory  # optional, :memory is the default.  :file is also supported
     after "test.expires.nsecs" do |key, val|
       cool_notify 'ward@thecleavers.com', val
     end
@@ -75,8 +77,8 @@ Do it!  The github url for Sadie is at: [https://github.com/FredAtLandMetrics/sa
  
 ==Future versions
 Future version of Sadie will:
-  * index on key string patterns
-  * make use of a redis storage mechanism to become scalable and distributed
+* index on key string patterns
+* make use of a redis storage mechanism to become scalable and distributed
 
 ==Where to get it
 Sadie can be downloaded via its rubygems page[https://rubygems.org/gems/sadie] or from github[https://github.com/FredAtLandMetrics/sadie].

data/Rakefile

@@ -41,6 +41,16 @@ namespace :spec do
     system "rspec spec/primer.rb"
   end
   
+  desc "test timestamp queue"
+  task :timestamp_queue do
+    system "rspec spec/timestamp_queue.rb"
+  end
+  
+  desc "test lock_manager"
+  task :lock_manager do
+    system "rspec spec/lock_manager.rb"
+  end
+  
   desc "test storage manager"
   task :storage_manager do
     system "rspec spec/storage_manager.rb"

data/TODO

@@ -1,2 +1,25 @@
-1) rewrite everything
-2) re-document everything
+==Preparing for distributed-ness
+
+The main idea behind distributed Sadie is a bunch of Sadie servers that share an nfs volume and which make use of a Redis cluster.  So I need to be careful that no single sadie server need be "special".
+
+As it stands, the session has these things to consider:
+
+* mutexes
+* expiry data structure
+* refresh data structure
+* registered keys
+
+For mutexes, I'm thinking a locking abstraction will do.  Something like:
+
+  lock_id = acquire_lock( params )
+  release_lock( lock_id )
+
+Once abstracted, this should be relatively straightforward to extend as needed for distributed-ness.
+
+[ NOTE: lock abstraction is now done ]  
+
+The expiry and refresh data structures are currently implemented as red-black trees, but it would be ideal if there were a way to share this information amongst the sadie instances.  I'm thinking it might make sense to only a single sadie instance to be expiring at a time (and then only for a set amount of time).  Further, I'm thinking refresh could be synchronized such that only a single sadie instance can be finding a refreshable key at a time, but, once it's selected one, it releases its lock so that another instance can begin a search for a another refreshable.
+
+[ NOTE: the expiry and refresh data structures are now maintained by a 'timestamp_queue' class that will make it easier to swap in redis-based functionality ]
+
+The registered keys data structure can actually be maintained separately in each instance in as much as the primers should really be the same across the instances.

data/lib/lock_manager.rb

@@ -0,0 +1,52 @@
+require 'thread'
+
+class LockManager
+  
+  def initialize
+    @locks = {}
+  end
+  
+  def create( params )
+    systype = params[:systype].to_s
+    locktype = params[:locktype].to_s
+    lock_id = "#{systype}:#{locktype}"
+    @locks[lock_id] = Mutex.new unless @locks.has_key?( lock_id )
+    lock_id
+  end
+  
+  def acquire( lock_id )
+    if @locks[lock_id].try_lock
+      lock_id
+    else
+      nil
+    end
+  end
+  
+  def release( lock_id )
+    if @locks.has_key?( lock_id )
+      @locks[lock_id].unlock
+    end
+  end
+  
+  def critical_section_insist( lock_id )
+    if block_given?
+      if @locks.has_key?( lock_id )
+        @locks[lock_id].synchronize do
+          yield
+        end
+      end
+    end
+  end
+  
+  def critical_section_try( lock_id )
+    if block_given?
+      if @locks.has_key?( lock_id )
+        if acquire( lock_id )
+          yield
+          release( lock_id )
+        end
+      end
+    end
+  end
+
+end

data/lib/sadie/version.rb

@@ -1,3 +1,3 @@
 class Sadie
-  VERSION = "0.1.8"
+  VERSION = "0.1.9"
 end

data/lib/sadie_session.rb

@@ -3,26 +3,33 @@ require 'storage/memory'
 require 'storage/file'
 require 'primer'
 require 'thread'
-require 'rbtree'
+require 'lock_manager'
+require 'timestamp_queue'
 
 class SadieSession
   attr_accessor :primers_dirpath
   
   def initialize( params )
     
-    @expiry_mutex = Mutex.new
-    @expire_schedule = MultiRBTree.new
-    @expiry_thread = Thread.new do
-      _expiry_loop
-    end
+    # init lock manager
+    @lockmgr = LockManager.new
     
-    @refresh_mutex = Mutex.new
-    @refresh_schedule = MultiRBTree.new
-    @refresh_thread = Thread.new do
-      _refresh_loop
-    end
+    # init expiry and refresh threads
+    @expiry_lock = @lockmgr.create( :systype => :session,
+                                    :locktype => :expiry  )
+    @refresh_lock = @lockmgr.create( :systype => :session,
+                                     :locktype => :refresh  )
+
+    @expiry_queue,@refresh_queue = TimestampQueue.new,TimestampQueue.new
+    
+    _initialize_expiry_thread
+    _initialize_refresh_thread
     
+    
+    # init registered key hash
     @registered_key = {}
+    
+    # init session operating parameters
     @default_storage_mechanism = :memory
     @file_storage_mechanism_dirpath = nil
     unless params.nil?
@@ -30,7 +37,6 @@ class SadieSession
         
         if params.has_key?( :primers_dirpath )
           self.primers_dirpath = params[:primers_dirpath]
-          puts "initializing session with primer dirpath: #{self.primers_dirpath}"
           _register_primers
         end
         
@@ -45,18 +51,30 @@ class SadieSession
       end
     end
     
-    @storage_manager_thread_mutex = Mutex.new
+    # init storage manager
+    @storagemgr_lock = @lockmgr.create( :systype => :session,
+                                        :locktype => :expiry  )
     @storage_manager = SadieStorageManager.new
-    @storage_manager_thread_mutex.synchronize do
+    @lockmgr.critical_section_insist( @storagemgr_lock ) do
       @storage_manager.register_storage_mechanism :memory, SadieStorageMechanismMemory.new
       @storage_manager.register_storage_mechanism :file, SadieStorageMechanismFile.new(:key_storage_dirpath => @file_storage_mechanism_dirpath)
     end
+    
   end
   
-  def has_key?( key )
-    @storage_manager_thread_mutex.synchronize do
-      @storage_manager.has_key?( key )
+  def has_key?( key, params )
+    ret = @storage_manager.has_key?( key )
+    include_primers = true
+    
+    if ( params.is_a?( Hash ) ) && ( params.has_key?( :include_primers ) )
+      include_primers = params[:include_primers]
+    end      
+    
+    if ( ! ret ) && ( include_primers )
+      ret = primer_registered?( key )
     end
+    
+    ret
   end
   
   def primer_registered?( key )
@@ -64,23 +82,33 @@ class SadieSession
   end
   
   def unset( key )
-    @storage_manager_thread_mutex.synchronize do
+    @lockmgr.critical_section_insist( @storagemgr_lock ) do
       @storage_manager.unset( key )
     end
   end
   
+  def has_metadata?( key )
+    @storage_manager.has_metadata?( key )
+  end
+  
+  def metadata( key )
+    @storage_manager.metadata( key )
+  end
+  
   def set( keys, value, params=nil )
-    expires, mechanism = :never, @default_storage_mechanism
+    expires, mechanism, metadata = :never, @default_storage_mechanism, nil
     unless params.nil?
       if params.is_a? Hash
         expires = params[:expire] if params.has_key?( :expire )
         mechanism = params[:mechanism] if params.has_key?( :mechanism )
+        metadata = params[:metadata] if params.has_key?( :metadata )
       end
     end
-    @storage_manager_thread_mutex.synchronize do
+    @lockmgr.critical_section_insist( @storagemgr_lock ) do
       @storage_manager.set( :keys => Array( keys ),
                             :value => value,
-                            :mechanism => mechanism )
+                            :mechanism => mechanism,
+                            :metadata => metadata )
     end
     _manage_expiry( keys, expires ) unless expires == :never || expires == :on_get
   end
@@ -106,13 +134,25 @@ class SadieSession
   
   private
   
+  def _initialize_refresh_thread
+    @refresh_thread = Thread.new do
+      _refresh_loop
+    end
+  end
+  
+  def _initialize_expiry_thread
+    @expiry_thread = Thread.new do
+      _expiry_loop
+    end
+  end
+  
   def _manage_expiry( keys, expires_seconds )
     if ! expires_seconds.is_a?( Symbol ) && expires_seconds.to_i > 0
       expires = expires_seconds.to_i + _current_time
       unless Array(keys).empty?
         Array(keys).each do |key|
-          @expiry_mutex.synchronize do
-            @expire_schedule[expires] = key
+          @lockmgr.critical_section_insist( @expiry_lock ) do
+            @expiry_queue.insert( key, :timestamp => expires )
           end
         end
       end
@@ -124,8 +164,8 @@ class SadieSession
       refreshes = refresh_seconds.to_i + _current_time
       unless Array(keys).empty?
         Array(keys).each do |key|
-          @refresh_mutex.synchronize do
-            @refresh_schedule[refreshes] = key
+          @lockmgr.critical_section_insist( @refresh_lock ) do
+            @refresh_queue.insert( key, :timestamp => refreshes )
           end
         end
       end
@@ -159,18 +199,27 @@ class SadieSession
   
   def _expiry_pass
     time_now_in_seconds = _current_time
-    @expiry_mutex.synchronize do
-      loop do
-        break if @expire_schedule.empty?
-        ts,key = @expire_schedule.shift
-        if ts < time_now_in_seconds
+    
+    loop do
+      break if @expiry_queue.empty?
+      
+      ts,key = nil
+      
+      keys_to_unset = nil
+      @lockmgr.critical_section_insist( @expiry_lock ) do
+        
+        keys_to_unset = @expiry_queue.find( :all, :before => time_now_in_seconds )
+        
+      end
+      
+      unless keys_to_unset.nil?
+        keys_to_unset.each do |key|
           unset key
-        else
-          @expire_schedule[ts] = key
-          break
         end
       end
-    end    
+      
+    end
+      
   end
   
   def _refresh_loop
@@ -182,18 +231,18 @@ class SadieSession
   
   def _refresh_pass
     time_now_in_seconds = _current_time
-      loop do
-        break if @refresh_schedule.empty?
-        ts,key = @refresh_schedule.shift
-        if ts < time_now_in_seconds
+    unless @refresh_queue.empty?
+      keys = nil
+      @lockmgr.critical_section_insist( @refresh_lock ) do
+        keys = @refresh_queue.find(:all, :before => time_now_in_seconds)
+      end
+      
+      unless keys.nil?
+        keys.each do | key |
           _refresh key
-        else
-          @refresh_mutex.synchronize do
-            @refresh_schedule[ts] = key
-          end
-          break
         end
       end
+    end
   end
   
   def _refresh( key )

data/lib/sadie_storage_manager.rb

@@ -41,6 +41,14 @@ class SadieStorageManager
     end
   end
   
+  def has_metadata?( key )
+    @mechanisms[where_key?( key )].has_metadata?( key ) if has_key?( key )
+  end
+  
+  def metadata( key )
+    @mechanisms[where_key?( key )].metadata( key ) if has_key?( key )
+  end
+  
   def set( params )
     unless params.nil?
       
@@ -52,9 +60,16 @@ class SadieStorageManager
             
             if params.has_key?( :keys ) && params[:keys].is_a?( Array ) &&
                params.has_key?( :value )
-            
+              has_metadata = false
+              if params.has_key?(:metadata) && params[:metadata].is_a?( Hash )
+                has_metadata = true
+              end
               params[:keys].each do |key|
-                @mechanisms[params[:mechanism]].set( key, params[:value] )
+                if has_metadata
+                  @mechanisms[params[:mechanism]].set( key, params[:value], :metadata => params[:metadata] )
+                else
+                  @mechanisms[params[:mechanism]].set( key, params[:value] )
+                end
               end
             end
           end

data/lib/storage/file.rb

@@ -1,4 +1,6 @@
 require 'sadie_storage_mechanism'
+# require 'marshal'
+
 class SadieStorageMechanismFile < SadieStorageMechanism
   attr_accessor :key_storage_dirpath
   
@@ -13,9 +15,24 @@ class SadieStorageMechanismFile < SadieStorageMechanism
     end
   end
   
-  def set( key, value )
+  def set( key, value, params=nil )
     _validate_keystorage_directory
     File.open(_keyvalue_filepath(key), 'wb') { |file| file.write(value) }
+    unless params.nil?
+      if params.has_key? :metadata
+        _write_metadata_file( key, params[:metadata] )
+      end
+    end
+  end
+  
+  def metadata( key )
+    contents = File.open( _metadata_filepath( key ), 'rb') { |f| f.read }
+#     puts "contents: #{contents}"
+    Marshal.load( contents ) if has_metadata?( key )
+  end
+  
+  def has_metadata?( key )
+    File.exists? _metadata_filepath( key )
   end
   
   def get( key )
@@ -36,15 +53,31 @@ class SadieStorageMechanismFile < SadieStorageMechanism
   
 private
 
+  def _metadata_filepath( key )
+    _ensure_metadata_dirpath_exists
+    File.join( self.key_storage_dirpath, '.meta', key )
+  end
+  
+  def _write_metadata_file( key, metadata_hash )
+    raise 'metadata must be a hash' unless ( ! metadata_hash.nil? ) && ( metadata_hash.is_a?( Hash ) )
+    _ensure_metadata_dirpath_exists
+    File.open( _metadata_filepath( key ), 'wb' ) { |file| file.write( Marshal.dump( metadata_hash ) ) }
+  end
+
   def _validate_keystorage_directory
     raise "Key storage directory (#{self.key_storage_dirpath}) does not exist" unless _keystorage_directory_exists?
   end
 
   def _keyvalue_filepath(key)
-    File.join(self.key_storage_dirpath,key)
+    File.join( self.key_storage_dirpath, key )
   end
   
   def _keystorage_directory_exists?
     Dir.exists?( self.key_storage_dirpath )
   end
+  
+  def _ensure_metadata_dirpath_exists
+    dirpath = File.join( self.key_storage_dirpath, '.meta' )
+    Dir.mkdir( dirpath ) unless Dir.exists?( dirpath )
+  end
 end

data/lib/storage/memory.rb

@@ -3,10 +3,16 @@ class SadieStorageMechanismMemory < SadieStorageMechanism
   
   def initialize
     @storage_hash = {}
+    @metadata = {}
   end
   
-  def set( key, value )
+  def set( key, value, params=nil  )
     @storage_hash[key] = value
+    unless params.nil?
+      if params.has_key? :metadata
+        _write_metadata( key, params[:metadata] )
+      end
+    end
   end
   
   def get( key )
@@ -21,4 +27,19 @@ class SadieStorageMechanismMemory < SadieStorageMechanism
     @storage_hash.has_key?( key )
   end
   
+  def metadata( key )
+    @metadata[key] if has_metadata?( key )
+  end
+  
+  def has_metadata?( key )
+    @metadata.has_key?( key )
+  end
+  
+  private
+  
+  def _write_metadata( key, metadata_hash )
+    raise 'metadata must be a hash' unless ( ! metadata_hash.nil? ) && ( metadata_hash.is_a?( Hash ) )
+    @metadata[key] = metadata_hash
+  end
+  
 end

data/lib/timestamp_queue.rb

@@ -0,0 +1,103 @@
+require 'rbtree'
+class TimestampQueue
+  
+  def initialize
+    @queue = MultiRBTree.new
+  end
+  
+  def insert( key, params=nil )
+    ts = nil
+    if ( params.is_a?( Hash ) ) &&
+       ( params.has_key?( :timestamp ) )
+      ts = params[:timestamp]
+    else
+      ts = _current_time
+    end      
+    @queue[ts] = key
+  end
+  
+  def find( which, params=nil )
+    if which == :first
+      _find_first( params )
+    elsif which == :all
+      _find_all( params )
+    end
+  end
+  
+  def empty?
+    @queue.empty?
+  end
+  
+  private
+  
+  def _get_functions( params )
+    testfunc,getfunc = nil,nil,nil
+    if params.is_a? Hash
+      if params.has_key? :before
+        thresh = params[:before]
+        testfunc = Proc.new { |x| (x < thresh) }
+        getfunc = Proc.new { |q| q.shift }
+      end
+    end
+    [testfunc,getfunc]
+  end
+  
+  def _find_first( params )
+    
+    ret = nil
+    testfunc,getfunc = _get_functions(params)
+    unless testfunc.nil?
+      ts,key = getfunc.call(@queue)
+#       puts "testing: #{ts}, #{key}"
+      if testfunc.call(ts)
+        ret = _package_rec(ts,key,params)
+      else
+        @queue[ts] = key
+      end
+    end
+    
+    ret
+  end
+  
+  def _find_all( params )
+    ret = nil
+    testfunc,getfunc = _get_functions(params)
+    unless testfunc.nil?
+      
+      retarray = []
+      loop do
+        break if @queue.empty?
+        ts,key = getfunc.call(@queue)
+        if testfunc.call(ts)
+          retarray.push _package_rec(ts,key,params)
+        else
+          @queue[ts] = key
+          break
+        end
+      end
+      ret = retarray unless retarray.empty?
+    end
+    ret
+  end
+  
+  
+  def _package_rec(timestamp,key,params)
+    if params.nil? || ! params.is_a?( Hash )
+      key
+    elsif params.has_key?( :as )
+      if params[:as] == :hash
+        { :timestamp => timestamp,
+          :key => key }
+      else
+        key
+      end
+    else
+      key
+    end
+  end
+  
+  def _current_time
+    Time.now.to_i
+  end
+  
+end

data/spec/lock_manager.rb

@@ -0,0 +1,147 @@
+$:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'lock_manager'
+require 'thread'
+
+describe LockManager do
+  
+  before :each do
+    @lockmgr = LockManager.new
+  end
+  
+  it "should properly protect critical sections with acquire and release" do
+    @total = 0
+    @max = 0
+    t1 = Thread.new do
+      @total += 1
+      @max = @total if @total > @max
+      sleep rand(3).to_i
+      @total -= 1
+    end
+    t2 = Thread.new do
+      @total += 1
+      @max = @total if @total > @max
+      sleep rand(3).to_i
+      @total -= 1
+    end
+    t3 = Thread.new do
+      @total += 1
+      @max = @total if @total > @max
+      sleep rand(3).to_i
+      @total -= 1
+    end
+    sleep 5
+    t1.join
+    t2.join
+    t3.join
+    ( @max > 1 ).should be_true
+    
+    lockid = @lockmgr.create( :systype => 'test', :locktype => 'test' )
+    @total = 0
+    @max = 0
+    t1 = Thread.new do
+      3.times do
+        unless @lockmgr.acquire( lockid ).nil?
+          @total += 1
+          @max = @total if @total > @max
+          sleep rand(3).to_i
+          @total -= 1
+          @lockmgr.release( lockid )
+        end
+      end
+    end
+    t2 = Thread.new do
+      3.times do
+        unless @lockmgr.acquire( lockid ).nil?
+          @total += 1
+          @max = @total if @total > @max
+          sleep rand(3).to_i
+          @total -= 1
+          @lockmgr.release( lockid )
+        end
+      end
+    end
+    t3 = Thread.new do
+      3.times do
+        unless @lockmgr.acquire( lockid ).nil?
+          @total += 1
+          @max = @total if @total > @max
+          sleep rand(3).to_i
+          @total -= 1
+          @lockmgr.release( lockid )
+        end
+      end
+    end
+    sleep 5
+    t1.join
+    t2.join
+    t3.join
+    ( @max > 1 ).should be_false
+  end
+  
+  it "should properly protect critical sections with critical section insist" do
+    @total = 0
+    @max = 0
+    t1 = Thread.new do
+      @total += 1
+      @max = @total if @total > @max
+      sleep rand(3).to_i
+      @total -= 1
+    end
+    t2 = Thread.new do
+      @total += 1
+      @max = @total if @total > @max
+      sleep rand(3).to_i
+      @total -= 1
+    end
+    t3 = Thread.new do
+      @total += 1
+      @max = @total if @total > @max
+      sleep rand(3).to_i
+      @total -= 1
+    end
+    sleep 5
+    t1.join
+    t2.join
+    t3.join
+    ( @max > 1 ).should be_true
+    
+    lockid = @lockmgr.create( :systype => 'test', :locktype => 'test' )
+    @total = 0
+    @max = 0
+    t1 = Thread.new do
+      3.times do
+        @lockmgr.critical_section_insist( lockid ) do
+          @total += 1
+          @max = @total if @total > @max
+          sleep rand(3).to_i
+          @total -= 1
+        end
+      end
+    end
+    t2 = Thread.new do
+      3.times do
+        @lockmgr.critical_section_insist( lockid ) do
+          @total += 1
+          @max = @total if @total > @max
+          sleep rand(3).to_i
+          @total -= 1
+        end
+      end
+    end
+    t3 = Thread.new do
+      3.times do
+        @lockmgr.critical_section_insist( lockid ) do
+          @total += 1
+          @max = @total if @total > @max
+          sleep rand(3).to_i
+          @total -= 1
+        end
+      end
+    end
+    sleep 5
+    t1.join
+    t2.join
+    t3.join
+    ( @max > 1 ).should be_false
+  end
+end

data/spec/sadie_session.rb

@@ -1,4 +1,5 @@
 $:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'timestamp_queue'
 require 'sadie_session'
 require 'pp'
 describe SadieSession do
@@ -38,7 +39,7 @@ describe SadieSession do
   end
   
   it "should not execute primer assign directives" do
-    @session.has_key?("minimal.primer").should be_false
+    @session.has_key?("minimal.primer", :include_primers => false).should be_false
   end
   
   it "should be possible to _get_ registered keys" do
@@ -51,16 +52,23 @@ describe SadieSession do
   
   it "should be possible to expire on get" do
     @session.get("test.expires.onget").should == "testval"
-    @session.has_key?("test.expires.onget").should be_false
+    @session.has_key?("test.expires.onget", :include_primers => false).should be_false
   end
   
   it "should put keys in the expire schedule" do
-    def @session.in_expire_schedule?( key )
-      ( ! @expire_schedule.values.index(key).nil? )
+    def @session.get_expiry_queue
+      @expiry_queue
     end
+    q = @session.get_expiry_queue
+    def q.in_expire_schedule?( key )
+      ! @queue.values.index(key).nil?
+    end
+#     def @session.in_expire_schedule?( key )
+#       ( ! @expire_schedule.values.index(key).nil? )
+#     end
     
     @session.get("test.expires.nsecs").should == "testval"
-    @session.in_expire_schedule?("test.expires.nsecs").should be true
+    q.in_expire_schedule?("test.expires.nsecs").should be true
   end
   
   it "should expire keys using _expire_pass" do
@@ -71,9 +79,9 @@ describe SadieSession do
     @session.stub(:_current_time).and_return(2,5,8,11,14)
     @session.stub(:_expiry_loop).and_return(false)
     @session.get("test.expires.nsecs").should == "testval"
-    @session.has_key?("test.expires.nsecs").should be_true
+    @session.has_key?("test.expires.nsecs", :include_primers => false).should be_true
     @session.run_expiry_pass
-    @session.has_key?("test.expires.nsecs").should be_false
+    @session.has_key?("test.expires.nsecs", :include_primers => false).should be_false
   end
   
   it "should refresh keys" do
@@ -124,6 +132,34 @@ describe SadieSession do
     @session.detect_storage_mechanism('key2').should == :file
   end
   
+  it "should be possible for primer files to choose the storage mechanism" do
+    def @session.detect_storage_mechanism(key)
+      @storage_manager.where_key?( key )
+    end
+    val_mem = @session.get("minimal.primer")
+    val_file = @session.get("minimal.primer.file")
+    val_mem.should == "testval"
+    val_file.should == "testval_file"
+    @session.detect_storage_mechanism("minimal.primer").should == :memory
+    @session.detect_storage_mechanism("minimal.primer.file").should == :file
+  end
+  
+  it "should be able to set and retrieve metadata" do
+    test_metadata_hash = {  :type => :string,
+                            :importance => :huge  }
+    test_metadata_hash_right = {  :type => :string,
+                                  :importance => :huge  }
+    test_metadata_hash_wrong = {  :type => :string,
+                                  :importance => :mild  }
+    @session.set( "test.key1", "test.value1", :metadata => test_metadata_hash )
+    @session.set( "test.key2", "test.value2" )
+    @session.has_metadata?( "test.key1" ).should be_true
+    @session.has_metadata?( "test.key2" ).should be_false
+    ( @session.metadata( "test.key1" ) == test_metadata_hash_right ).should be_true
+    ( @session.metadata( "test.key1" ) == test_metadata_hash_wrong ).should be_false
+    
+  end
+  
   # --- SLOW!
   if ENV.has_key?('SADIE_SESSION_TEST_TIMERS') && ENV['SADIE_SESSION_TEST_TIMERS'].to_i == 1
     

data/spec/storage_manager.rb

@@ -4,55 +4,65 @@ require 'storage_mechanisms/memory'
 
 describe SadieStorageManager do
   
+  before :each do
+    @storage = SadieStorageManager.new
+    @mech = SadieStorageMechanismMemory.new
+    @storage.register_storage_mechanism :memory, @mech
+  end
+  
   it "should be able to get and set using the memory storage mechanism" do
-    
-    storage = SadieStorageManager.new
-    mech = SadieStorageMechanismMemory.new
-    storage.register_storage_mechanism :memory, mech
-    storage.set( :mechanism => :memory,
-                 :keys => [ "simple.test" ],
-                 :value => "test.value" )
-    mech.get( "simple.test" ).should == "test.value"
+    @storage.set( :mechanism => :memory,
+                  :keys => [ "simple.test" ],
+                  :value => "test.value" )
+    @mech.get( "simple.test" ).should == "test.value"
   end
   
   it "should report a mechanism is registered after it has been registered" do
-    storage = SadieStorageManager.new
-    mech = SadieStorageMechanismMemory.new
-    storage.register_storage_mechanism :memory, mech
-    storage.mechanism_is_registered?( :memory ).should be_true
+    @storage.mechanism_is_registered?( :memory ).should be_true
   end
   
   it "should have a functional has_key? method" do
-    storage = SadieStorageManager.new
-    mech = SadieStorageMechanismMemory.new
-    storage.register_storage_mechanism :memory, mech
-    storage.has_key?( "test.key" ).should be_false
-    storage.set( :keys => ["test.key"],
-                 :value => "test.value",
-                 :mechanism => :memory )
-    storage.has_key?( "test.key" ).should be_true
+    @storage.has_key?( "test.key" ).should be_false
+    @storage.set( :keys => ["test.key"],
+                  :value => "test.value",
+                  :mechanism => :memory )
+    @storage.has_key?( "test.key" ).should be_true
   end
   
   it "should have a functional get method" do
-    storage = SadieStorageManager.new
-    mech = SadieStorageMechanismMemory.new
-    storage.register_storage_mechanism :memory, mech
-    storage.set( :keys => ["test.key"],
-                 :value => "test.value",
-                 :mechanism => :memory )
-    storage.get( "test.key" ).should == "test.value"
+    @storage.set( :keys => ["test.key"],
+                  :value => "test.value",
+                  :mechanism => :memory )
+    @storage.get( "test.key" ).should == "test.value"
   end
   
   it "should have a functional unset method" do
-    storage = SadieStorageManager.new
-    mech = SadieStorageMechanismMemory.new
-    storage.register_storage_mechanism :memory, mech
-    storage.set( :keys => ["test.key"],
-                 :value => "test.value",
-                 :mechanism => :memory )
-    storage.has_key?( "test.key" ).should be_true
-    storage.unset "test.key"
-    storage.has_key?( "test.key" ).should be_false
+    @storage.set( :keys => ["test.key"],
+                  :value => "test.value",
+                  :mechanism => :memory )
+    @storage.has_key?( "test.key" ).should be_true
+    @storage.unset "test.key"
+    @storage.has_key?( "test.key" ).should be_false
   end
   
+  it "should be able to set metadata for a key" do
+    @storage.set( :keys => ["test.key1"],
+                  :value => "test.value1",
+                  :mechanism => :memory )
+    test_metadata_hash = {  :type => :string,
+                            :importance => :huge  }
+    test_metadata_hash_right = {  :type => :string,
+                                  :importance => :huge  }
+    test_metadata_hash_wrong = {  :type => :string,
+                                  :importance => :mild  }
+    @storage.set( :keys => ["test.key2"],
+                  :value => "test.value2",
+                  :mechanism => :memory,
+                  :metadata => test_metadata_hash )
+    @storage.has_metadata?( "test.key1" ).should be_false
+    @storage.has_metadata?( "test.key2" ).should be_true
+    ( @storage.metadata( "test.key2" ) == test_metadata_hash_right ).should be_true
+    ( @storage.metadata( "test.key2" ) == test_metadata_hash_wrong ).should be_false
+  end    
+  
 end

data/spec/storage_mechanisms/file.rb

@@ -28,4 +28,41 @@ describe SadieStorageMechanismFile do
     @mech.has_key?( "somekey.test" ).should be_false
   end
   
+  it "should create a file when set is called" do
+    @mech.set 'somekey.test','some_value'
+    File.exists?( File.join( '/tmp/stormech-file-test','somekey.test' ) ).should be_true
+  end
+  
+  it "should delete a file when unset is called" do
+    @mech.set 'somekey.test','some_value'
+    File.exists?( File.join( '/tmp/stormech-file-test','somekey.test' ) ).should be_true
+    @mech.unset 'somekey.test'
+    File.exists?( File.join( '/tmp/stormech-file-test','somekey.test' ) ).should be_false
+  end
+  
+  it "should write metadata to a file" do
+    @mech.set 'somekey.test','some_value', :metadata => { :type => :string }
+    File.exists?( File.join( '/tmp/stormech-file-test','somekey.test' ) ).should be_true
+    File.exists?( File.join( '/tmp/stormech-file-test/.meta','somekey.test' ) ).should be_true
+  end
+  
+  it "should return the same metadata that was given to it" do
+    metadata_to_give = {
+      :type => :string,
+      :awesomeness_level => :excrutiatingly
+    }
+    metadata_to_test = {
+      :type => :string,
+      :awesomeness_level => :excrutiatingly
+    }
+    wrong_metadata_to_test = {
+      :type => :integer,
+      :awesomeness_level => :excrutiatingly
+    }
+    @mech.set 'somekey.test','some_value', :metadata => metadata_to_give
+    fetched_meta = @mech.metadata( 'somekey.test' )
+    ( fetched_meta == metadata_to_test ).should be_true
+    ( fetched_meta == wrong_metadata_to_test ).should be_false
+  end
+  
 end

data/spec/storage_mechanisms/memory.rb

@@ -1,28 +1,46 @@
 require 'storage/memory'
 describe SadieStorageMechanismMemory do
   
+  before :each do
+    @mech = SadieStorageMechanismMemory.new
+  end
   it "should successfully return a set value" do
-    
-    mech = SadieStorageMechanismMemory.new
-    mech.set 'somekey.test','some_value'
-    mech.get( 'somekey.test' ).should == 'some_value'
+    @mech.set 'somekey.test','some_value'
+    @mech.get( 'somekey.test' ).should == 'some_value'
     
   end
   
   it "should have a functional has_key? method" do
-    mech = SadieStorageMechanismMemory.new
-    mech.has_key?( "somekey.test" ).should be_false
-    mech.set 'somekey.test','some_value'
-    mech.has_key?( "somekey.test" ).should be_true
+    @mech.has_key?( "somekey.test" ).should be_false
+    @mech.set 'somekey.test','some_value'
+    @mech.has_key?( "somekey.test" ).should be_true
   end
   
   it "should have a functional unset method" do
-    mech = SadieStorageMechanismMemory.new
-    mech.set 'somekey.test','some_value'
-    mech.has_key?( "somekey.test" ).should be_true
-    mech.unset 'somekey.test'
-    mech.has_key?( "somekey.test" ).should be_false
+    @mech.set 'somekey.test','some_value'
+    @mech.has_key?( "somekey.test" ).should be_true
+    @mech.unset 'somekey.test'
+    @mech.has_key?( "somekey.test" ).should be_false
     
   end
   
+  it "should return the same metadata that was given to it" do
+    metadata_to_give = {
+      :type => :string,
+      :awesomeness_level => :excrutiatingly
+    }
+    metadata_to_test = {
+      :type => :string,
+      :awesomeness_level => :excrutiatingly
+    }
+    wrong_metadata_to_test = {
+      :type => :integer,
+      :awesomeness_level => :excrutiatingly
+    }
+    @mech.set 'somekey.test','some_value', :metadata => metadata_to_give
+    fetched_meta = @mech.metadata( 'somekey.test' )
+    ( fetched_meta == metadata_to_test ).should be_true
+    ( fetched_meta == wrong_metadata_to_test ).should be_false
+  end
+  
 end

data/spec/timestamp_queue.rb

@@ -0,0 +1,39 @@
+$:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'timestamp_queue'
+
+describe TimestampQueue do
+  
+  before :each do
+    @tsq = TimestampQueue.new
+  end
+  
+  it "should store a key using the current time if one is not provided" do
+    @tsq.stub(:_current_time).and_return(99)
+    @tsq.insert( 'testkey' )
+    rec = @tsq.find :first, :before => 100, :as => :hash
+    rec[:key].should == 'testkey'
+    rec[:timestamp].should == 99
+  end
+  
+  it "should return a key if the as parameter is not provided" do
+    @tsq.stub(:_current_time).and_return(99)
+    @tsq.insert( 'testkey' )
+    key = @tsq.find :first, :before => 100
+    key.should == 'testkey'
+  end
+  
+  it "should return an array of records using :all" do
+    @tsq.stub(:_current_time).and_return(99,100,101,104)
+    @tsq.insert( 'testkey1' )
+    @tsq.insert( 'testkey2' )
+    @tsq.insert( 'testkey3' )
+    @tsq.insert( 'testkey4' )
+    keys = @tsq.find :all, :before => 102
+    keys.is_a?( Array ).should be_true
+    keys.empty?.should be_false
+    keys.length.should == 3
+    keys[0].should == 'testkey1'
+    keys[1].should == 'testkey2'
+    keys[2].should == 'testkey3'
+  end
+end

data/test/v2/test_installation/primers/minimal_filestormech.rb

@@ -0,0 +1,6 @@
+prime ["minimal.primer.file"] do
+  store_in :file
+  assign do
+    set "testval_file"
+  end
+end

data/test/v2/test_installation/primers/test_refresh.rb

@@ -1,7 +1,7 @@
 prime ["test.refresh"] do
   refresh 1
   assign do
-    if session.has_key?( "test.refresh" )
+    if session.has_key?( "test.refresh", :include_primers => false )
       set session.get("test.refresh").gsub(/^r/,"rr")
     else
       set "refresh"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sadie
 version: !ruby/object:Gem::Version
-  version: 0.1.8
+  version: 0.1.9
 platform: ruby
 authors:
 - Fred McDavid
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-07 00:00:00.000000000 Z
+date: 2014-01-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sinatra
@@ -96,6 +96,7 @@ files:
 - doc/planning/brainstorm.txt
 - doc/planning/todo.examples.txt
 - lib/.gitignore
+- lib/lock_manager.rb
 - lib/primer.rb
 - lib/sadie/version.rb
 - lib/sadie_server.rb
@@ -104,6 +105,7 @@ files:
 - lib/sadie_storage_mechanism.rb
 - lib/storage/file.rb
 - lib/storage/memory.rb
+- lib/timestamp_queue.rb
 - rdoc/classes/Sadie.html
 - rdoc/created.rid
 - rdoc/files/README.html
@@ -116,6 +118,7 @@ files:
 - rdoc/index.html
 - rdoc/rdoc-style.css
 - sadie.gemspec
+- spec/lock_manager.rb
 - spec/primer.rb
 - spec/sadie_server.rb
 - spec/sadie_server_lib.rb
@@ -123,10 +126,12 @@ files:
 - spec/storage_manager.rb
 - spec/storage_mechanisms/file.rb
 - spec/storage_mechanisms/memory.rb
+- spec/timestamp_queue.rb
 - test/v2/another_test_installation/config/sadie.yml
 - test/v2/another_test_installation/primers/minimal.rb
 - test/v2/test_installation/config/sadie.yml
 - test/v2/test_installation/primers/minimal.rb
+- test/v2/test_installation/primers/minimal_filestormech.rb
 - test/v2/test_installation/primers/onelevel/twolevel/test_subdir.rb
 - test/v2/test_installation/primers/test_after_each.rb
 - test/v2/test_installation/primers/test_after_key.rb