sadie 0.0.52 → 0.1.6

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 8e344594b19d7b7541c54600a79644071396458c
+  data.tar.gz: db4f2671448cea713c637315067a3449e9cd675a
+SHA512:
+  metadata.gz: ab2e088b58e59f8fbd6ae4fb4446954a73ff4bb8afa12084f92abecb0f7a09cbaa34531ba73df491a4aeafe72fe2ff481a90320a6a10cacf3f717925642f7e61
+  data.tar.gz: 56a214771d30d04eb93a59dc6b4bdfab76663be7578408a9fbf59c5b8161f682533dc95471bca3e5962eafa44aa324732d7657f4187dc81bd29ac48501935f42

data/CHANGELOG

@@ -24,4 +24,5 @@
 [0.0.49] botched the upload...resubmitting
 [0.0.50] fixed isset? bug (was only working for non-expensive keys)
 [0.0.51] eacher bugfix. now correctly handles specific keys.
-[0.0.52] code cleanup
+[0.0.52] code cleanup
+[0.1.01] ** ground-up rewrite, ignore everything before this line if you're not a historian **

data/README

@@ -1,34 +1,88 @@
 ==About Sadie
 
-Sadie is a general-purpose data server written in Ruby and it's intended to be used by a Rubie developer to simplify access to inter-related data sets.
+Sadie is a general-purpose data server written in Ruby for other Rubyists.
 
-It is designed to provide handle-based access to data of all types, much like a simple hash.  Unlike a hash, a sadie instance is created with a pathname which points to a directory of primers which tell sadie how to determine what the value of a given hash key should be, but values are not assigned until the hash keys are requested.  In this way, only necessary data transformations and/or accesses are performed.
+==Purpose
+Imagine you work in the IT department of a cell phone company and your boss asks you to design a system that will:
+  * build a status page that shows a map with the operational status of all the cell towers and, for each one, the distance to the nearest field technician
+  * generate a shapefile and a spreadsheet of the same data that will be available and up-to-date on the company's intranet, 24/7
+  * when there's an outtage, send an SMS alert to the nearest field tech to the tower with the problem
 
-Valid primers currently are:
+This is all inter-related data and, for a developer, it's not all that difficult to start thinking about how to approach these problems.
 
-  .ini - provide access to section, key, value data in ini-style-files
-  .dbi.conx - simple access to database handles
-  .sql2ar - simple access to database data
-  .res, .res.rb - provides access to general-purpose, programatic manipulation of data (i.e. just use sadie instance for input and output of data)
-  .tmpl - simple templating engine for text-based file types
+With that said, if your boss a week later decides that he also needs to have google map imagery on the alert page and charts showing tower availability over time and average response time of technicians over the past several weeks, then things get a little more complicated.
 
-There's also another file type that can be found in the primer directory called "eacher files".  These are simple files which allow for certain tasks to be performed before, after and during the priming of a key/value pair.  Currently, the only plugin that makes effective use of eachers is sql2ar and it provides a way for developers to iterate on the rows of a query result as it is initially processed into the resultant array.
+And if two weeks after that, he says he needs the technician availability data updated every ten minutes and the cell tower availability every five minutes, and, oh, the number of calls that come into customer care regarding a each tower's primary coverage area, it gets tough to imagine that the initial software engineering effort will have been robust enough to keep things sane.
+
+This is the sort of problem that Sadie is intended to address.
+
+==How it works
+Sadie is a key/value store that uses <em>primers</em>--which are really just ruby files written using Sadie's primer DSL--to set the key/value pairs.  These primers can be called on-demand (so no computational resources get expended until a request for a value is made) or they can be refreshed at certain intervals.
+
+For data that changes often, it's possible to expire data after a set amount of time or, if necessary, just after it's accessed for a just-in-time computation.
+
+Primers can also reference other primers so it's easy to hide complexity.
+
+==How to use it
+Create a directory for sadie to operate in, say /var/sadie.
+
+Then create a directory in that called primers.  <em>So, now there's a /var/sadie/primers</em>
+
+In the primers directory, create files that end in .rb that look like:
+
+  require 'intersting_lib_that_does_neato_things'
+  require 'cool_notifier'
+  
+  prime ["test.expires.nsecs"] do
   
-It's very simple to add new primer types, so it would be simple to create primers for html, xml, csv, xls, pointers to all of the above on remote machines, or any- and everything else.
+    expire :never
+    refresh 300 #generate this again every 5 minutes
+    
+    after "test.expires.nsecs" do |key, val|
+      cool_notify 'ward@thecleavers.com', val
+    end
+    
+    assign do
+    
+      someothervar = session.get( "some.other.var" )
+      andanother = session.get( "yet.another.var" )
+      
+      neat_val = awesome_whatzit( someothervar, andanother )
+      
+      set neat_val
+    end
+  end
 
-At Landmetrics, we use a predecessor to Sadie to power perimetercomps[http://perimetercomps.com], producing each 50+ page report with charts, graphs and tables with only 12 queries to the database.
+Have a look at other primers in test/v2/test_installation/primers for more information.
 
-Sadie can be downloaded via its rubygems page[https://rubygems.org/gems/sadie] or from github[https://github.com/FredAtLandMetrics/sadie].
+Now run the sadie server on whatever directory you set it up on:
+
+  bin/sadie_server.rb --framework-dirpath=/var/sadie
+  
+And you can make get requests on the keys,
+
+  wget http://localhost:4567/test.expires.nsecs -O /tmp/outputfile
 
-USE CASES
+Or, you can use the sadie session in your Ruby code like this:
+
+  @sadie_session = SadieSession.new( :primers_dirpath => '/var/sadie/primers' )
+  puts "test.expires.nsecs: #{@sadie_session.get('test.expires.nsecs')}"
+ 
+That's it!
+
+==Contribute
+Do it!  The github url for Sadie is at: [https://github.com/FredAtLandMetrics/sadie]
+ 
+==Future versions
+Future version of Sadie will:
+  * index on key string patterns
+  * make use of a redis storage mechanism to become scalable and distributed
+
+==Where to get it
+Sadie can be downloaded via its rubygems page[https://rubygems.org/gems/sadie] or from github[https://github.com/FredAtLandMetrics/sadie].
 
-[templating]    As noted above, perimetercomps[http://perimetercomps.com] is an example of how one might
-                use sadie to to create a templating engine wherein many pieces of related information
-                from many different sources are made into charts, graphs, tables and text before being
-                assembled into a single deliverable document.  We used LaTeX, but html, xml, css, txt, etc.
-                would be just as easy.
+==Support
+Feel free to email me at fred@landmetrics.com
 
-[data import]   Tasks involving grabbing data from different places in different formats and
-                massaging the data into a consistent and useful format.  A few simple primer plugins
-                would make this very easy.
-                
+==How to say thanks
+I accept thanks via email at fred@landmetrics.com.  Beer from afar and lucrative consulting gigs are also welcomed.

data/Rakefile

@@ -11,10 +11,50 @@ task :test do
     ruby "test/tc_sadie_twodeep.rb"
 end
 
+namespace :spec do
+  
+  desc "test sadie server library"
+  task :sadie_server_lib do
+    system "rspec spec/sadie_server_lib.rb"
+  end
+  
+  namespace :storage_mechanism do
+    desc "test the memory based storage mechanism"
+    task :memory do
+      system "rspec spec/storage_mechanisms/memory.rb"
+    end
+  end
+  
+  desc "test primer"
+  task :primer do
+    system "rspec spec/primer.rb"
+  end
+  
+  desc "test storage manager"
+  task :storage_manager do
+    system "rspec spec/storage_manager.rb"
+  end
+  
+  desc "test session"
+  task :session do
+    system "rspec spec/sadie_session.rb"
+  end
+  
+  desc "test session with timers (slow)"
+  task :session_with_timers do
+    system "SADIE_SESSION_TEST_TIMERS=1 rspec spec/sadie_session.rb"
+  end
+  
+  desc "test RESTful server"
+  task :server do
+    system "rspec spec/sadie_server.rb"
+  end
+end
+
 # increment version
+desc "deploy a new gem"
 task :deploy => 'inc_version' do
     version = current_sadie_version
-    sh "git push"
     sh "gem build sadie.gemspec"
     sh "gem push sadie-#{version}.gem"
 end

data/TODO

@@ -1,31 +1,2 @@
-[setDestroyTimer]  this will accept either a quantity of time or a specific date and
-                   will destroy the data at that time, causing a "reprime" event
-                   
-[threadedness]     split getSadieInstance(sessionid] into getSadieInstanceSlave[sessionid]
-                   where getSadieInstance will continute to bring up a normal Sadie
-                   instance and getSadieInstanceSlave will perform reads in its own thread
-                   but will prime in sync with the main instance thread (such that two
-                   primers for the same key would never execute at the same time, rather,
-                   the second one would simply block until the first completed).  This
-                   would allow for some concurrency within a session without.  Rather than
-                   calling getSadieInstanceSlave directly, a prgram could simple call a
-                   new method, consider( key ) which would fire off a get in an instance
-                   slave to allow for some background processing
-                   
-[keyregexpmatch]   instead of requiring that all keys must be known either as having been set
-                   or having a known primer, a key could be matched against a regex.
-                   This would make it possible to encode some useful priming information
-                   in the key itself, enough that the primer could use it and divine the
-                   desired result.  So, where Sadie::prime takes an arg of containing
-                   a list of keys that this primer "provides", it will alternatively take an
-                   arg of "key-match" which will fire off the primer when the key matches.
-                   
-[sampleapps]       make a few simple apps with primer directories 
-
-[primer_path]      currently defined via a path to a single directory, the toplevel primer
-                   directory idea should be made into a collection of directories as spec'ed
-                   in a colon-separated list, and it should check the environment, if
-                   otherwise undefined.  would make it easy to have multiple primer
-                   repositories.
-                   
-                   
+1) rewrite everything
+2) re-document everything

data/bin/sadie

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+require 'sinatra'
+require 'sadie_server'
+
+get '/:key' do
+  sadie_server_get params[:key]
+end
+
+post '/:key' do
+  sadie_server_set params[:key], params[:value]
+end
+
+post '/query' do
+  sadie_server_query params[:query]
+end
+

data/bin/sadie_query_server

@@ -0,0 +1,15 @@
+#!/usr/bin/env ruby
+
+require 'sinatra'
+require 'sadie_server'
+
+sadie_server_proc_args ARGV
+
+get '/:key' do
+  sadie_server_get params[:key]
+end
+
+post '/query' do
+  sadie_server_query params[:query]
+end
+

data/bin/sadie_server

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+
+require 'sinatra'
+require 'server'
+
+server = SadieServer.new( SadieServer::proc_args( ARGV ) )
+
+get '/:key' do
+  server.get params[:key]
+end
+
+post '/:key' do
+  server.set params[:key], params[:value]
+end
+
+post '/setmultiple' do
+  server.set_multiple params[:data]
+end
+
+post '/getmultiple' do
+  server.set_multiple params[:keys]
+end
+
+post '/query' do
+  server.query params[:query]
+end
+

data/bin/sadie_server.rb

@@ -0,0 +1,23 @@
+#!/usr/bin/env ruby
+$:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'sadie_server'
+require 'pp'
+arghash = SadieServer::proc_args( ARGV )
+puts "arghash: #{arghash.pretty_inspect}"
+require 'sinatra'
+
+server = SadieServer.new( arghash )
+
+get '/:key' do
+  server.get params[:key]
+end
+
+post '/:key' do
+  server.set params[:key], params[:value]
+end
+
+# FUTURE
+# post '/query' do
+#   server.query params[:query]
+# end
+

data/doc/v2

@@ -0,0 +1,161 @@
+BIG UGLY PLANNING DOC
+
+So, now that Sadie's been written and reasonably well tested, I'd like to see it change to something more ruby-esque.
+
+I liked the idea of pluggable types, but I want to add RESTful services and they're just not going to work, so I think I'll go with constructors/destructors as an alternative.
+
+I'm also going to move to the more rubyesque style of using method calls in primer definitions instead of parameters...it just looks better.
+
+So, where in the old version I've got a primer that looks like:
+
+  Sadie::prime( "provides" =>     %w{ toplevel_double.oneprime
+                                      toplevel_double.twoprime } ) do |sadie|
+      sadie.set( "toplevel_double.oneprime", "primedthem" )
+      sadie.set( "toplevel_double.twoprime", "primedthem" )
+  end
+
+and an eacher in a separate file that looks like:
+
+
+  Sadie::eacher( :when => Sadie::BEFORE   ) do |sadie|
+      @testeach_topleveldouble = "blah"
+      sadie.debug! 10, ">>>>HERE!!!"
+  end
+
+  Sadie::eacher( :when => Sadie::AFTER,
+                :provides => ["toplevel_double.eachtest"]   ) do |sadie,val|
+                
+      @testeach_topleveldouble = "#{@testeach_topleveldouble}bloo"
+      sadie.set "toplevel_double.eachtest", @testeach_topleveldouble 
+      sadie.debug! 10, ">>>>THERE!!!"
+      
+  end
+
+I'll now have a single file that looks like:
+
+  prime %w{ toplevel_double.oneprime toplevel_double.twoprime toplevel_double.eachtest } do
+
+    before :each do
+      @testeach_topleveldouble = "blah"
+    end
+    
+    after :each do |val|
+    
+      @testeach_topleveldouble = "#{@testeach_topleveldouble}bloo"
+      
+      assign "toplevel_double.eachtest" do
+        set @testeach_topleveldouble
+      end
+      
+    end
+    
+    assign "toplevel_double.oneprime" do
+      set "primedthem"
+    end
+    
+    assign "toplevel_double.twoprime" do
+      set "primedthem"
+    end
+    
+  end
+  
+But for a more interesting example of things to come:
+
+  prime %w{ shapefile.url } do
+  
+    expire   "shapefile.url",     :after => 300
+    
+    after "shapefile.url" do |value|
+      notify_admin_of_new_shapefile value
+    end
+    
+    assign "shapefile.url" do
+      output = generate_shapefile(get('gis.points.array'))
+      set "http://#{GISHOST}/shapefiles/#{output}"
+    end
+  
+  end
+  
+  prime %w{ webview.image.url } do
+  
+    expire :after => 30                                   # note that, if unspecified, expire applies to all names in prime array
+    
+    assign do                                             # note that, if unspecified, assign applies to all names in prime array
+      set build_webview_image(get('gis.points.array'))
+    end
+  
+  end
+  
+  prime %w{ gis.points.array } do
+  
+    expire "gis.points.array", :immediately
+    
+    assign "gis.points.array" do
+      set get_badguy_locations_from_drone
+    end
+  
+  end
+  
+What's happening here is we have a bit of continually refreshable data called gis.points.array that is new every time we retrieve it.  We're building an image using the gis points array for the web page that only changes every 60 seconds and we're building a new shapefile using the gis points array that only changes every 300 seconds.  No look at this:
+
+  prime %w{ shapefile.url } do
+  
+    expire   "shapefile.url",     :never
+    refresh 300
+    
+    after "shapefile.url" do |value|
+      notify_admin_of_new_shapefile value
+    end
+    
+    assign "shapefile.url" do
+      output = generate_shapefile(get('gis.points.array'))
+      set "http://#{GISHOST}/shapefiles/#{output}"
+    end
+  
+  end
+  
+So, now, instead of being generated on demand, this becomes a background process via the refresh statement and a new shapefile will be generated every 300 seconds.
+
+*** Index by key name such that a.b.c creates an entry in three indexes, a., b., and a.b.c.
+
+       this will facilitate an aggregate getter
+       
+*** Allow for a read-only query language that executes ruby code in safe mode level = 4 (and maybe runs in some sort of jailed process sandbox that only has access to the local sadie server via RESTful interface?)
+
+It should also be possible to expire when some other key is set, like:
+
+  prime %w{ shapefile.url } do
+  
+    expire   "shapefile.url",     :after => key_is_set("gis.points.array")
+    
+    after "shapefile.url" do |value|
+      notify_admin_of_new_shapefile value
+    end
+    
+    assign "shapefile.url" do
+      output = generate_shapefile(get('gis.points.array'))
+      set "http://#{GISHOST}/shapefiles/#{output}"
+    end
+  
+  end
+  
+It should also be possible to expire when a file on the filesystem has changed:
+
+  prime ["gis.points.list_string"] do
+    expire :after => file_has_changed("/var/gisdata/someguys_points_file.csv")
+    assign do
+      set "gis.points.list_string", File.open("/var/gisdata/someguys_points_file.csv", 'rb') { |f| f.read }
+    end
+  end
+  
+It should be possible to choose the method of storage for an item
+
+  prime ["gis.points.list_string"] do
+    expire :after => file_has_changed("/var/gisdata/someguys_points_file.csv")
+    store_in :redis (or :memory, :file)
+    assign do
+      set "gis.points.list_string", File.open("/var/gisdata/someguys_points_file.csv", 'rb') { |f| f.read }
+    end
+  end
+
+  

data/lib/primer.rb

@@ -0,0 +1,201 @@
+class Primer
+  
+  attr_accessor :keys, :mode, :session, :storage_mechanism, :assign_keys, :filepath, :refresh_rate
+  
+  def initialize( params=nil )
+    self.storage_mechanism = :memory
+    self.refresh_rate = nil
+    @before_block = {}
+    @after_block = {}
+    expire(:never)
+    unless params.nil?
+      if params.is_a? Hash
+        if params.has_key?( :session )
+          self.session = params[:session]
+        end
+      end      
+    end
+  end
+  
+  def refresh( rsecs )
+    self.refresh_rate = rsecs
+  end
+  
+  def refreshes?
+    ( ! refresh_rate.nil? )
+  end
+  
+  def expire( val=nil )
+    unless val.nil?
+      @expire = val
+    end
+    @expire
+  end
+  
+  def decorate( primer_filepath )
+    if File.exists?( primer_filepath )
+      self.instance_eval File.open(primer_filepath, 'rb') { |f| f.read } 
+      self.filepath = primer_filepath
+    else
+      raise ArgumentError, "#{primer_filepath} not found"
+    end
+  end
+  
+  def before( arg, &block )
+    if arg == :each
+      if block_given?
+        @before_block[:each] = block
+      end
+    elsif arg.is_a? String
+      if self.keys.index( arg ).nil?
+        raise 'key passed as argument to before must be declared in the prime directive'
+      else
+        @before_block[arg] = block
+      end
+    end
+  end
+  
+  def after( arg, &block )
+    if arg == :each
+      if block_given?
+        @after_block[:each] = block
+      end
+    elsif arg.is_a? String
+      if self.keys.index( arg ).nil?
+        raise 'key passed as argument to after must be declared in the prime directive'
+      else
+        @after_block[arg] = block
+      end
+    end
+  end
+  
+  def prime( k=nil )
+    self.keys = Array(k) if _validate_key_arg(k)
+    yield if block_given?
+  end
+  
+  def assign( keys = nil )
+    if keys.nil?
+      self.assign_keys = self.keys
+    elsif _validate_key_arg(keys)
+      unless (Array( keys ) - self.keys).empty?
+        raise 'assigning keys that are not given as arguments to prime is not permitted'
+      end
+      self.assign_keys = Array(keys)
+    end
+    unless mode == :registration
+      yield if block_given?
+    end
+  end
+  
+  def set( keys=nil, value=nil)
+    if value.nil?
+      _set1(keys)
+    else
+      _set2(keys,value)
+    end
+  end
+  
+  def store_in( mech )
+    self.storage_mechanism = mech
+  end
+  
+  private
+  
+  def _set2( keys=nil, value )
+    if keys.nil?
+      _set1(value)
+    elsif _validate_key_arg(keys)
+      unless (Array( keys ) - self.assign_keys).empty?
+        raise 'assigning keys that are not given as arguments to assign (or to prime, if assign was given no keys as arguments) is not permitted'
+      end
+      
+      if @before_block.has_key?(:each) && ! @before_block[:each].nil?
+        
+        Array(keys).each do |key|
+          @before_block[:each].call(key)
+        end
+        
+      end
+      
+      Array(keys).each do |key|
+        if @before_block.has_key?(key) && ! @before_block[key].nil?
+          
+          @before_block[key].call(key)
+          
+        end
+      end
+      
+      self.session.set( Array(keys), value, :mechanism => self.storage_mechanism,
+                                            :expire => self.expire                )
+      
+      if @after_block.has_key?(:each) && ! @after_block[:each].nil?
+        
+        Array(keys).each do |key|
+          @after_block[:each].call(key,value)
+        end
+        
+      end
+      
+      Array(keys).each do |key|
+        if @after_block.has_key?(key) && ! @after_block[key].nil?
+          
+          @after_block[key].call(key,value)
+          
+        end
+      end
+    end
+  end
+  
+  def _set1( value )
+    unless @before_block[:each].nil?
+      
+      Array(keys).each do |key|
+        @before_block[:each].call(key)
+      end
+      
+    end
+    Array(self.assign_keys).each do |key|
+      if @before_block.has_key?(key) && ! @before_block[key].nil?
+        
+        @before_block[key].call(key)
+        
+      end
+    end
+    
+    self.session.set( Array(keys), value, :mechanism => self.storage_mechanism,
+                                          :expire => self.expire                )
+    
+    if @after_block.has_key?(:each) && ! @after_block[:each].nil?
+      
+      Array(keys).each do |key|
+        @after_block[:each].call(key,value)
+      end
+      
+    end
+    
+    Array(keys).each do |key|
+      if @after_block.has_key?(key) && ! @after_block[key].nil?
+        
+        @after_block[key].call(key,value)
+        
+      end
+    end
+  end
+  
+  def _validate_key_arg( k=nil )
+    if k.is_a?(String)
+      true
+    elsif k.is_a?( Array )
+      k.each do |key|
+        unless key.is_a?( String )
+          raise 'keys must be string or array or strings'
+        end
+      end
+    else
+      raise 'keys must be string or array or strings'
+    end
+    true
+  end
+  
+end

data/lib/sadie/version.rb

@@ -1,3 +1,3 @@
 class Sadie
-  VERSION = "0.0.52"
+  VERSION = "0.1.6"
 end