epo 0.0.1

data/README

@@ -0,0 +1,157 @@
+
+= EPO
+
+EPO is a no-brainer, plain-ruby file system database. If you have objects and
+need to store them in a clean hierarchy, then EPO is a good choice.
+EPO is not a good choice if you want to perform optimized queries or if you
+operate on big datasets.
+In EPO, each object has a directory, so, it is easy to use programs which write
+their output at a given file without using temp dirs.  Similarly, if you do
+lots of batch operations on your EPO's resources, the one-directory per
+resource approach scales well.
+
+== Philosophy
+
+EPO only is a small library, so summarizing it's philosophy takes only a few
+lines:
+* My filesystem already is a database
+* My database should be plain ruby and
+* But my database should be easy to use from non-ruby programs
+
+== The EPO hierarchy
+
+EPO tries to build from the lessons of KISS and REST routes.
+If you're familiar with REST applications, you will find EPO's hierarchy pretty
+straightforward.
+
+=== Important Rules
+
+Here are the simple rules for EPO's databases:
+* EPO operates on Welo::Resources
+* each resource has one directory
+* the directory's path identifies the resource stored
+* there is one serialization file per (resource, perspective, extension) tuple
+* any other file in the directory should not impact your ruby application, but may have meaning to other programs (e.g., a thumbnail created by your file-viewer)
+
+=== Examples
+
+If you have a resource "user" identified by its "login", if you have three
+users: peter, jon, and marc. 
+The database directories may look like the following. Between brackets are reference to further explanations.
+
+$ tree db/
+db/
+└── user
+    ├── peter
+    |   ├── resource-default.json  [1a]
+    |   └── picture.png            [1b]
+    ├── jon
+    |   ├── resource-default.json  [2a]
+    |   ├── picture.png            [2b]
+    |   └── pubkey.rsa.txt         [2c]
+    └── marc
+        ├── resource-default.json  [3a]
+        ├── picture.png            [3b]
+        └── thumbnail.dat          [3c]
+
+In the previous hierarchy, [1a, 2a, 3a] are the serialization of the user
+resources under the 'default' (or :default) perspective in the JSON format.
+The perspective is a concept of Welo's resource, which is basically a list of
+fields of an object that you want to dump or observe (an administrator may have
+access to more details than a mere, non-registered user).
+
+The other objects may or may not be related to your ruby application.  A
+convenient explanation may be: [1b, 2b, 3b] are pictures, most likely your
+user's headshots, uploaded by your users through your application.  User [2b]
+has given his public key, and we see that your filebrowser has created a
+thumbnail in [3c], which has nothing to do with your application.
+
+=== Attention
+
+The main thing to keep in mind is that filesystems come with a slight problem:
+file paths are strings.
+As a result, when designing your EPO hierarchy, you must be aware of:
+- encoding issues
+- case sensitivity
+- ambiguities with path separators 
+
+== Benefits
+
+Example of things you can do easily with EPO:
+* use a filesystem explorer to visualize/explore your DB
+* organize your documents without hassle
+* use bash scripts for batch processing
+* use rsync/NFS/git on all or part of your database's content
+* use ftp/http servers to expose a branch of your DB's hierarchy 
+* have other programs use your DB easily without configuring databases
+
+Say you have photos to sort. You may sort them by year, by place, by subject or
+other things. Some software propose you to tag your photos and build a database
+with these photos for you. Unfortunately, these softwares are often closed, you
+cannot re-use their database, or it requires lots of effort to script the
+software to do something it is not ready for (e.g., resize your pictures in a
+batch). Moreover, the file-viewer of your OS may be good enough to view your
+pictures.  Filesystems have solved many database issues since ages.  So, why
+not just rely on your file system to store your pictures? 
+
+== Usage (please have a look at the examples directory)
+
+=== EPO is simple
+There is no complex things in EPO, as an example, there is no:
+- index
+- transaction
+- thread/multiprocess safety
+- lifecycle hook
+If you want to add a missing feature, feel free to fork/subclass/add a module,
+or implement it directly on your filesystem.
+
+=== Theory
+
+EPO uses:
+* Derailleur's paths routing to quickly map paths to resources
+  (we may want to remove this dependency later)
+* Welo's observers to hook events when iterating on the filesystem
+
+EPO::DB are just simple, in-memory Ruby objects. 
+They have no connection to take care of, no credentials.
+
+An EPO::DB may understand several formats (json or yaml are standard choices).
+You may modify this by changing EPO::DB#extensions .
+
+EPO::DB are headless, in the sense that they don't store their hierarchy's root
+in a variable. As a result, an EPO::DB contains the concepts of the models but
+are not actually tied to the data on the filesystem.
+You may have two EPO::DB on the same filesystem's root (each understanding
+different resources or formats).
+
+=== Commented code bits
+
+Say you have two models (which are Welo::Resources): Person and Item.
+Both models must have a "identifying" named :flat_db (this is just a default convention).
+  class Person 
+    include Welo::Resource
+    identify :flat_db, [:name]
+    ...
+
+If you want to create a database only able to handle persons
+  EPO::DB.new([Person])
+
+Creates a database able to handle persons and items
+  db = EPO::DB.new([Person, Item])
+
+Saves a person with the default perspective, in all format
+  person = Person.new(...)
+  db.save(root, person)
+
+Iterates on all the DB items (as observations)
+  db.each_resource(root) { |observation| ... }
+
+== Dependencies
+
+welo >= 0.1.0
+derailleur >= 0.5.0
+a JSON library if you use json formatting
+
+== License
+
+The MIT license

data/Rakefile

@@ -0,0 +1,44 @@
+
+require 'rubygems'
+require 'rake/gempackagetask'
+
+$LOAD_PATH.unshift('lib')
+require 'epo'
+
+spec = Gem::Specification.new do |s|
+
+        s.name = 'epo'
+        s.rubyforge_project = 'epo'
+        s.version = EPO::VERSION
+        s.author = EPO::AUTHORS.first
+        s.homepage = EPO::WEBSITE
+        s.summary = "A no-brainer, plain-ruby database"
+        s.email = "crapooze@gmail.com"
+        s.platform = Gem::Platform::RUBY
+
+        s.files = [
+          'Rakefile', 
+          'TODO', 
+          'README', 
+          'lib/epo.rb',
+          'lib/epo/core/db.rb',
+          'lib/epo/core/observer.rb',
+          'lib/epo/core/dispatch_observations.rb',
+        ]
+
+        s.require_path = 'lib'
+        s.bindir = 'bin'
+        s.executables = []
+        s.has_rdoc = true
+
+        s.add_dependency('derailleur', '>= 0.0.5')
+        s.add_dependency('welo', '>= 0.1.0')
+end
+
+Rake::GemPackageTask.new(spec) do |pkg|
+        pkg.need_tar = true
+end
+
+task :gem => ["pkg/#{spec.name}-#{spec.version}.gem"] do
+        puts "generated #{spec.version}"
+end

data/TODO

@@ -0,0 +1,9 @@
+* delete record/observation
+* maybe with some modifications to Derailleur:
+  - graceful not found case
+  - clever pruning when finding:
+    - when path is a directory and there is no node, prune
+    - when path is a directory but there is a node, continue
+  - cache observation structs somewhere in the db or in the observer
+* fiber-friendly batch operations
+* register on inotify events

data/lib/epo/core/db.rb

@@ -0,0 +1,189 @@
+
+require 'derailleur/core/application'
+require 'fileutils'
+
+module EPO
+  class DB
+    include Derailleur::Application
+
+    # The list of understood models, models must behave like Welo::Resources
+    attr_reader :models
+
+    # A list of understood extensions (e.g., '.json', '.yaml'), default is '.json'
+    attr_reader :extensions
+
+    # The name of the identifiers to use to map Welo::Resources to directories
+    attr_reader :identifying_sym
+
+    # Creates a new DB
+    # models: the list of Welo::Resource models to use (i.e., classes)
+    def initialize(models, params={})
+      @models = models
+      @extensions = params[:extensions] || ['.json']
+      @identifying_sym = params[:identifying_sym] || :flat_db
+      build!
+    end
+
+    private
+
+    # creates the tree to map the DB structure to classes paths
+    # does not register the model because it uses existing one
+    def build!
+      models.each do |model|
+        build_route_for_model(model)
+      end
+    end
+
+    public
+
+    # Registers a model on a derailleur node at its path_model
+    # Does not modify self.models
+    def build_route_for_model(model)
+      path = model.path_model(identifying_sym) #XXX allow to change the db identifying name
+      node = build_route(path)
+      node.content = model
+    end
+
+    # Updates self.models and build a route for the model
+    # raise an error if the model is already known
+    def register_model(model)
+      raise ArgumentError, "already know this model" if models.include?(model)
+      models << model
+      build_route_for_model(model)
+    end
+
+    # Loads the file at path and turn it to a ruby object based on the file extension.
+    # If the file extension is not supported, will raise an error.
+    # uses JSON.parse(File.read(path)) and YAML.load_file(path) for '.json' and '.yaml'
+    # otherwise, will forward the handling such that 
+    # '.foobaar' maps to :read_foobar(path)
+    def read_file(path)
+      raise ArgumentError, "don't know extension #{ext}, use from [#{extensions.join(', ')}]" unless understands_ext?(path)
+      ext = File.extname(path)
+      case ext
+      when '.json'
+        JSON.parse(File.read(path))
+      when '.yaml'
+        YAML.load_file(path)
+      else
+        self.send "read_#{ext.tr('.','')}", path
+      end
+    end
+
+    # Wether or not this DB understands the extension of the file at path.
+    def understands_ext?(path)
+      extensions.find{|ext| ext == File.extname(path)}
+    end
+
+    # Returns the perspective name and extension name (a 2 items array)
+    # for the given path.
+    # By default the blobs for resources content are stored in files named
+    # 'resource-<persp><ext>' with ext starting with a dot
+    def persp_and_ext_for_basename(path)
+      base = File.basename(path).sub('resource-','').sub(/\.[^\.]+$/,'')
+      [base, File.extname(path)]
+    end
+
+    # Returns the basename of a resource blob for a perspective named persp and
+    # in a format with extension ext (including the leading dot).  
+    # see also persp_and_ext_for_basename
+    def basename_for_persp_and_ext(persp, ext)
+      "resource-#{persp}#{ext}"
+    end
+
+    # Saves one or more resource at the filesystem path given at root
+    # This method is mainly an handy helper around batch_save, look at the
+    # source and at batch_save's doc
+    def save(root, resource, perspective=:default, exts=nil)
+      exts ||= extensions
+      batch_save(root, [resource].flatten, [perspective].flatten, [exts].flatten)
+    end
+
+    # Saves all the resources, under all the perspectives persps, and all format
+    # given by extensions exts at the filesystem path root.
+    # resources, perps, exts, must respond to :each, like for Enumerable
+    def batch_save(root, resources, persps, exts)
+      batch_save_actions(root, resources, persps, exts) do |action|
+        action.perform
+      end
+    end
+
+    # Yields all the action needed to store all the resources
+    # at perspectives persps and with formats exts.
+    # All actions respond to :perform (it is when they're executed).
+    # If no block given, returns an Enumerator with these actions.
+    def batch_save_actions(root, resources, persps, exts)
+      if block_given?
+        resources.each do |resource|
+          db_path = File.join(root, resource.path(identifying_sym))
+          yield PrepareDirAction.new(db_path)
+          exts.each do |ext|
+            persps.each do |persp|
+              basename = basename_for_persp_and_ext(persp, ext)
+              resource_path = File.join(db_path, basename) 
+              yield StoreResourceAction.new(resource_path, resource, persp, ext)
+            end
+          end
+        end
+      else
+        Enumerator.new(self, root, resources, persps, exts)
+      end
+    end
+
+    # An action to prepare the directory for a resource
+    PrepareDirAction = Struct.new(:path) do
+      def perform
+        FileUtils.mkdir_p(path)
+      end
+    end
+
+    # An action to serialize and store a resource seen in a given perspective
+    # into a file at path, under the format with a format given by its
+    # extension name.
+    StoreResourceAction = Struct.new(:path, :resource, :persp, :ext) do
+      def perform
+        data = resource.to_ext(ext.to_s, persp)
+        store_data_at_path(data, path)
+      end
+
+      def store_data_at_path(data, path, enum=:each, mode='w')
+        File.open(path, mode) do |f|
+          write_data_to_io(data, f, enum)
+        end
+      end
+
+      def write_data_to_io(data, io, enum=:each)
+        if data.respond_to?(enum)
+          data.each do |buf|
+            io << buf
+          end
+        else
+          io << data
+        end
+      end
+    end
+
+    # Returns a new EPO::Observer for itself
+    def observer
+      Observer.for_db(self)
+    end
+
+    # Iterates on every resource found and understood in the filesystem
+    # directory root.
+    # If no block is given, returns an iterator.
+    def each_resource(root) 
+      if block_given?
+        xp = observer
+        models.each do |model|
+          xp.on(model) do |obs|
+            yield obs
+          end
+        end
+        xp.read_tree(root)
+      else
+        Enumerator.new(self, :each_resource, root)
+      end
+    end
+
+  end
+end

data/lib/epo/core/dispatch_observations.rb

@@ -0,0 +1,21 @@
+
+module EPO
+  module DispatchObservations
+    # Register an event when an observation of a resource
+    # matching a given model is made
+    def on(model,&blk)
+      register(model,&blk)
+    end
+
+    private
+
+    # Call each blocks for an observation registering from the 
+    # observation's resource model
+    def dispatch(observation)
+      blks = registrations[observation.class.resource] || []
+      blks.each do |blk|
+        blk.call(observation)
+      end
+    end
+  end
+end

data/lib/epo/core/observer.rb

@@ -0,0 +1,81 @@
+
+require 'welo'
+require 'find'
+require 'epo/core/db'
+require 'epo/core/dispatch_observations'
+
+module EPO
+  # An Observer is a object able to look at the filesystem and 
+  # observe resources in it.
+  class Observer < Welo::Observer
+    include DispatchObservations
+    # in EPO, the source of an observation is:
+    # - a db object able to make a ruby object from a file path
+    # - a file path
+    Source = Struct.new(:db, :path) do
+      def observe
+        db.read_file(path)
+      end
+    end
+
+    # The DB able to tell if a path is understandable or not
+    attr_accessor :db
+
+    # Creates and return a new observer from a DB. 
+    # It will take the models from the DB.
+    def self.for_db(db)
+      obj = self.new(db.models)
+      obj.db = db
+      obj
+    end
+
+    # Creates a new observer for given models.
+    # Will instanciate a new DB.
+    def initialize(models=[])
+      super(models)
+      @db = DB.new(models)
+      register(:observation) do |o|
+        dispatch(o)
+      end
+    end
+
+    # Read a single path, the root is the part of the path corresponding
+    # to where on the filesystem the database is rooted.
+    # e.g.  for a path in a photo collection:
+    #       /home/crapooze/project/foobar/db/photo/1
+    #       the root is likely to be:
+    #       /home/crapooze/project/foobar/db
+    # If a successful observation happens, then it will call the relevant hooks.
+    def read_path(path, root=nil)
+      full_dirname, base = File.split(path)
+      dirname = if root
+                  full_dirname.sub(root,'')
+                else
+                  full_dirname
+                end
+      #XXX may raise an exception for unknown path, we should rescue this/use a
+      #silent method and test for nil
+      node = db.get_route(dirname)
+
+      persp_str = db.persp_and_ext_for_basename(path).first
+      persp = node.content.perspectives.keys.find{|k| k.to_s == persp_str}
+      #XXX no need to create this many time, should cache it
+      st = Welo::ObservationStruct.new_for_resource_in_perspective(node.content, persp)
+      source = Source.new(db, path)
+      observe_source(source, st)
+    end
+
+    # Recursively reads the files in the filesystem (with Find).
+    # For each path, will try to read_path
+    # Currently, there is no pruning, or control possible.
+    def read_tree(root)
+      Find.find(root) do |path|
+        if File.directory?(path)
+          #XXX maybe prune the branch if valid but has no content
+        elsif db.understands_ext?(path)
+          read_path(path, root) 
+        end
+      end
+    end
+  end
+end

data/lib/epo.rb

@@ -0,0 +1,10 @@
+
+module EPO
+  VERSION = "0.0.1"
+  AUTHORS = ["crapooze"]
+  WEBSITE = "https://github.com/crapooze/epo"
+  LICENCE = "MIT"
+
+  require 'epo/core/observer'
+  require 'epo/core/db'
+end

metadata

@@ -0,0 +1,99 @@
+--- !ruby/object:Gem::Specification 
+name: epo
+version: !ruby/object:Gem::Version 
+  prerelease: false
+  segments: 
+  - 0
+  - 0
+  - 1
+  version: 0.0.1
+platform: ruby
+authors: 
+- crapooze
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-03-19 00:00:00 -04:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: derailleur
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        - 0
+        - 5
+        version: 0.0.5
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: welo
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        segments: 
+        - 0
+        - 1
+        - 0
+        version: 0.1.0
+  type: :runtime
+  version_requirements: *id002
+description: 
+email: crapooze@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- Rakefile
+- TODO
+- README
+- lib/epo.rb
+- lib/epo/core/db.rb
+- lib/epo/core/observer.rb
+- lib/epo/core/dispatch_observations.rb
+has_rdoc: true
+homepage: https://github.com/crapooze/epo
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: epo
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: A no-brainer, plain-ruby database
+test_files: []
+