epo 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README ADDED
@@ -0,0 +1,157 @@
1
+
2
+ = EPO
3
+
4
+ EPO is a no-brainer, plain-ruby file system database. If you have objects and
5
+ need to store them in a clean hierarchy, then EPO is a good choice.
6
+ EPO is not a good choice if you want to perform optimized queries or if you
7
+ operate on big datasets.
8
+ In EPO, each object has a directory, so, it is easy to use programs which write
9
+ their output at a given file without using temp dirs. Similarly, if you do
10
+ lots of batch operations on your EPO's resources, the one-directory per
11
+ resource approach scales well.
12
+
13
+ == Philosophy
14
+
15
+ EPO only is a small library, so summarizing it's philosophy takes only a few
16
+ lines:
17
+ * My filesystem already is a database
18
+ * My database should be plain ruby and
19
+ * But my database should be easy to use from non-ruby programs
20
+
21
+ == The EPO hierarchy
22
+
23
+ EPO tries to build from the lessons of KISS and REST routes.
24
+ If you're familiar with REST applications, you will find EPO's hierarchy pretty
25
+ straightforward.
26
+
27
+ === Important Rules
28
+
29
+ Here are the simple rules for EPO's databases:
30
+ * EPO operates on Welo::Resources
31
+ * each resource has one directory
32
+ * the directory's path identifies the resource stored
33
+ * there is one serialization file per (resource, perspective, extension) tuple
34
+ * any other file in the directory should not impact your ruby application, but may have meaning to other programs (e.g., a thumbnail created by your file-viewer)
35
+
36
+ === Examples
37
+
38
+ If you have a resource "user" identified by its "login", if you have three
39
+ users: peter, jon, and marc.
40
+ The database directories may look like the following. Between brackets are reference to further explanations.
41
+
42
+ $ tree db/
43
+ db/
44
+ └── user
45
+ ├── peter
46
+ | ├── resource-default.json [1a]
47
+ | └── picture.png [1b]
48
+ ├── jon
49
+ | ├── resource-default.json [2a]
50
+ | ├── picture.png [2b]
51
+ | └── pubkey.rsa.txt [2c]
52
+ └── marc
53
+ ├── resource-default.json [3a]
54
+ ├── picture.png [3b]
55
+ └── thumbnail.dat [3c]
56
+
57
+ In the previous hierarchy, [1a, 2a, 3a] are the serialization of the user
58
+ resources under the 'default' (or :default) perspective in the JSON format.
59
+ The perspective is a concept of Welo's resource, which is basically a list of
60
+ fields of an object that you want to dump or observe (an administrator may have
61
+ access to more details than a mere, non-registered user).
62
+
63
+ The other objects may or may not be related to your ruby application. A
64
+ convenient explanation may be: [1b, 2b, 3b] are pictures, most likely your
65
+ user's headshots, uploaded by your users through your application. User [2b]
66
+ has given his public key, and we see that your filebrowser has created a
67
+ thumbnail in [3c], which has nothing to do with your application.
68
+
69
+ === Attention
70
+
71
+ The main thing to keep in mind is that filesystems come with a slight problem:
72
+ file paths are strings.
73
+ As a result, when designing your EPO hierarchy, you must be aware of:
74
+ - encoding issues
75
+ - case sensitivity
76
+ - ambiguities with path separators
77
+
78
+ == Benefits
79
+
80
+ Example of things you can do easily with EPO:
81
+ * use a filesystem explorer to visualize/explore your DB
82
+ * organize your documents without hassle
83
+ * use bash scripts for batch processing
84
+ * use rsync/NFS/git on all or part of your database's content
85
+ * use ftp/http servers to expose a branch of your DB's hierarchy
86
+ * have other programs use your DB easily without configuring databases
87
+
88
+ Say you have photos to sort. You may sort them by year, by place, by subject or
89
+ other things. Some software propose you to tag your photos and build a database
90
+ with these photos for you. Unfortunately, these softwares are often closed, you
91
+ cannot re-use their database, or it requires lots of effort to script the
92
+ software to do something it is not ready for (e.g., resize your pictures in a
93
+ batch). Moreover, the file-viewer of your OS may be good enough to view your
94
+ pictures. Filesystems have solved many database issues since ages. So, why
95
+ not just rely on your file system to store your pictures?
96
+
97
+ == Usage (please have a look at the examples directory)
98
+
99
+ === EPO is simple
100
+ There is no complex things in EPO, as an example, there is no:
101
+ - index
102
+ - transaction
103
+ - thread/multiprocess safety
104
+ - lifecycle hook
105
+ If you want to add a missing feature, feel free to fork/subclass/add a module,
106
+ or implement it directly on your filesystem.
107
+
108
+ === Theory
109
+
110
+ EPO uses:
111
+ * Derailleur's paths routing to quickly map paths to resources
112
+ (we may want to remove this dependency later)
113
+ * Welo's observers to hook events when iterating on the filesystem
114
+
115
+ EPO::DB are just simple, in-memory Ruby objects.
116
+ They have no connection to take care of, no credentials.
117
+
118
+ An EPO::DB may understand several formats (json or yaml are standard choices).
119
+ You may modify this by changing EPO::DB#extensions .
120
+
121
+ EPO::DB are headless, in the sense that they don't store their hierarchy's root
122
+ in a variable. As a result, an EPO::DB contains the concepts of the models but
123
+ are not actually tied to the data on the filesystem.
124
+ You may have two EPO::DB on the same filesystem's root (each understanding
125
+ different resources or formats).
126
+
127
+ === Commented code bits
128
+
129
+ Say you have two models (which are Welo::Resources): Person and Item.
130
+ Both models must have a "identifying" named :flat_db (this is just a default convention).
131
+ class Person
132
+ include Welo::Resource
133
+ identify :flat_db, [:name]
134
+ ...
135
+
136
+ If you want to create a database only able to handle persons
137
+ EPO::DB.new([Person])
138
+
139
+ Creates a database able to handle persons and items
140
+ db = EPO::DB.new([Person, Item])
141
+
142
+ Saves a person with the default perspective, in all format
143
+ person = Person.new(...)
144
+ db.save(root, person)
145
+
146
+ Iterates on all the DB items (as observations)
147
+ db.each_resource(root) { |observation| ... }
148
+
149
+ == Dependencies
150
+
151
+ welo >= 0.1.0
152
+ derailleur >= 0.5.0
153
+ a JSON library if you use json formatting
154
+
155
+ == License
156
+
157
+ The MIT license
data/Rakefile ADDED
@@ -0,0 +1,44 @@
1
+
2
+ require 'rubygems'
3
+ require 'rake/gempackagetask'
4
+
5
+ $LOAD_PATH.unshift('lib')
6
+ require 'epo'
7
+
8
+ spec = Gem::Specification.new do |s|
9
+
10
+ s.name = 'epo'
11
+ s.rubyforge_project = 'epo'
12
+ s.version = EPO::VERSION
13
+ s.author = EPO::AUTHORS.first
14
+ s.homepage = EPO::WEBSITE
15
+ s.summary = "A no-brainer, plain-ruby database"
16
+ s.email = "crapooze@gmail.com"
17
+ s.platform = Gem::Platform::RUBY
18
+
19
+ s.files = [
20
+ 'Rakefile',
21
+ 'TODO',
22
+ 'README',
23
+ 'lib/epo.rb',
24
+ 'lib/epo/core/db.rb',
25
+ 'lib/epo/core/observer.rb',
26
+ 'lib/epo/core/dispatch_observations.rb',
27
+ ]
28
+
29
+ s.require_path = 'lib'
30
+ s.bindir = 'bin'
31
+ s.executables = []
32
+ s.has_rdoc = true
33
+
34
+ s.add_dependency('derailleur', '>= 0.0.5')
35
+ s.add_dependency('welo', '>= 0.1.0')
36
+ end
37
+
38
+ Rake::GemPackageTask.new(spec) do |pkg|
39
+ pkg.need_tar = true
40
+ end
41
+
42
+ task :gem => ["pkg/#{spec.name}-#{spec.version}.gem"] do
43
+ puts "generated #{spec.version}"
44
+ end
data/TODO ADDED
@@ -0,0 +1,9 @@
1
+ * delete record/observation
2
+ * maybe with some modifications to Derailleur:
3
+ - graceful not found case
4
+ - clever pruning when finding:
5
+ - when path is a directory and there is no node, prune
6
+ - when path is a directory but there is a node, continue
7
+ - cache observation structs somewhere in the db or in the observer
8
+ * fiber-friendly batch operations
9
+ * register on inotify events
@@ -0,0 +1,189 @@
1
+
2
+ require 'derailleur/core/application'
3
+ require 'fileutils'
4
+
5
+ module EPO
6
+ class DB
7
+ include Derailleur::Application
8
+
9
+ # The list of understood models, models must behave like Welo::Resources
10
+ attr_reader :models
11
+
12
+ # A list of understood extensions (e.g., '.json', '.yaml'), default is '.json'
13
+ attr_reader :extensions
14
+
15
+ # The name of the identifiers to use to map Welo::Resources to directories
16
+ attr_reader :identifying_sym
17
+
18
+ # Creates a new DB
19
+ # models: the list of Welo::Resource models to use (i.e., classes)
20
+ def initialize(models, params={})
21
+ @models = models
22
+ @extensions = params[:extensions] || ['.json']
23
+ @identifying_sym = params[:identifying_sym] || :flat_db
24
+ build!
25
+ end
26
+
27
+ private
28
+
29
+ # creates the tree to map the DB structure to classes paths
30
+ # does not register the model because it uses existing one
31
+ def build!
32
+ models.each do |model|
33
+ build_route_for_model(model)
34
+ end
35
+ end
36
+
37
+ public
38
+
39
+ # Registers a model on a derailleur node at its path_model
40
+ # Does not modify self.models
41
+ def build_route_for_model(model)
42
+ path = model.path_model(identifying_sym) #XXX allow to change the db identifying name
43
+ node = build_route(path)
44
+ node.content = model
45
+ end
46
+
47
+ # Updates self.models and build a route for the model
48
+ # raise an error if the model is already known
49
+ def register_model(model)
50
+ raise ArgumentError, "already know this model" if models.include?(model)
51
+ models << model
52
+ build_route_for_model(model)
53
+ end
54
+
55
+ # Loads the file at path and turn it to a ruby object based on the file extension.
56
+ # If the file extension is not supported, will raise an error.
57
+ # uses JSON.parse(File.read(path)) and YAML.load_file(path) for '.json' and '.yaml'
58
+ # otherwise, will forward the handling such that
59
+ # '.foobaar' maps to :read_foobar(path)
60
+ def read_file(path)
61
+ raise ArgumentError, "don't know extension #{ext}, use from [#{extensions.join(', ')}]" unless understands_ext?(path)
62
+ ext = File.extname(path)
63
+ case ext
64
+ when '.json'
65
+ JSON.parse(File.read(path))
66
+ when '.yaml'
67
+ YAML.load_file(path)
68
+ else
69
+ self.send "read_#{ext.tr('.','')}", path
70
+ end
71
+ end
72
+
73
+ # Wether or not this DB understands the extension of the file at path.
74
+ def understands_ext?(path)
75
+ extensions.find{|ext| ext == File.extname(path)}
76
+ end
77
+
78
+ # Returns the perspective name and extension name (a 2 items array)
79
+ # for the given path.
80
+ # By default the blobs for resources content are stored in files named
81
+ # 'resource-<persp><ext>' with ext starting with a dot
82
+ def persp_and_ext_for_basename(path)
83
+ base = File.basename(path).sub('resource-','').sub(/\.[^\.]+$/,'')
84
+ [base, File.extname(path)]
85
+ end
86
+
87
+ # Returns the basename of a resource blob for a perspective named persp and
88
+ # in a format with extension ext (including the leading dot).
89
+ # see also persp_and_ext_for_basename
90
+ def basename_for_persp_and_ext(persp, ext)
91
+ "resource-#{persp}#{ext}"
92
+ end
93
+
94
+ # Saves one or more resource at the filesystem path given at root
95
+ # This method is mainly an handy helper around batch_save, look at the
96
+ # source and at batch_save's doc
97
+ def save(root, resource, perspective=:default, exts=nil)
98
+ exts ||= extensions
99
+ batch_save(root, [resource].flatten, [perspective].flatten, [exts].flatten)
100
+ end
101
+
102
+ # Saves all the resources, under all the perspectives persps, and all format
103
+ # given by extensions exts at the filesystem path root.
104
+ # resources, perps, exts, must respond to :each, like for Enumerable
105
+ def batch_save(root, resources, persps, exts)
106
+ batch_save_actions(root, resources, persps, exts) do |action|
107
+ action.perform
108
+ end
109
+ end
110
+
111
+ # Yields all the action needed to store all the resources
112
+ # at perspectives persps and with formats exts.
113
+ # All actions respond to :perform (it is when they're executed).
114
+ # If no block given, returns an Enumerator with these actions.
115
+ def batch_save_actions(root, resources, persps, exts)
116
+ if block_given?
117
+ resources.each do |resource|
118
+ db_path = File.join(root, resource.path(identifying_sym))
119
+ yield PrepareDirAction.new(db_path)
120
+ exts.each do |ext|
121
+ persps.each do |persp|
122
+ basename = basename_for_persp_and_ext(persp, ext)
123
+ resource_path = File.join(db_path, basename)
124
+ yield StoreResourceAction.new(resource_path, resource, persp, ext)
125
+ end
126
+ end
127
+ end
128
+ else
129
+ Enumerator.new(self, root, resources, persps, exts)
130
+ end
131
+ end
132
+
133
+ # An action to prepare the directory for a resource
134
+ PrepareDirAction = Struct.new(:path) do
135
+ def perform
136
+ FileUtils.mkdir_p(path)
137
+ end
138
+ end
139
+
140
+ # An action to serialize and store a resource seen in a given perspective
141
+ # into a file at path, under the format with a format given by its
142
+ # extension name.
143
+ StoreResourceAction = Struct.new(:path, :resource, :persp, :ext) do
144
+ def perform
145
+ data = resource.to_ext(ext.to_s, persp)
146
+ store_data_at_path(data, path)
147
+ end
148
+
149
+ def store_data_at_path(data, path, enum=:each, mode='w')
150
+ File.open(path, mode) do |f|
151
+ write_data_to_io(data, f, enum)
152
+ end
153
+ end
154
+
155
+ def write_data_to_io(data, io, enum=:each)
156
+ if data.respond_to?(enum)
157
+ data.each do |buf|
158
+ io << buf
159
+ end
160
+ else
161
+ io << data
162
+ end
163
+ end
164
+ end
165
+
166
+ # Returns a new EPO::Observer for itself
167
+ def observer
168
+ Observer.for_db(self)
169
+ end
170
+
171
+ # Iterates on every resource found and understood in the filesystem
172
+ # directory root.
173
+ # If no block is given, returns an iterator.
174
+ def each_resource(root)
175
+ if block_given?
176
+ xp = observer
177
+ models.each do |model|
178
+ xp.on(model) do |obs|
179
+ yield obs
180
+ end
181
+ end
182
+ xp.read_tree(root)
183
+ else
184
+ Enumerator.new(self, :each_resource, root)
185
+ end
186
+ end
187
+
188
+ end
189
+ end
@@ -0,0 +1,21 @@
1
+
2
+ module EPO
3
+ module DispatchObservations
4
+ # Register an event when an observation of a resource
5
+ # matching a given model is made
6
+ def on(model,&blk)
7
+ register(model,&blk)
8
+ end
9
+
10
+ private
11
+
12
+ # Call each blocks for an observation registering from the
13
+ # observation's resource model
14
+ def dispatch(observation)
15
+ blks = registrations[observation.class.resource] || []
16
+ blks.each do |blk|
17
+ blk.call(observation)
18
+ end
19
+ end
20
+ end
21
+ end
@@ -0,0 +1,81 @@
1
+
2
+ require 'welo'
3
+ require 'find'
4
+ require 'epo/core/db'
5
+ require 'epo/core/dispatch_observations'
6
+
7
+ module EPO
8
+ # An Observer is a object able to look at the filesystem and
9
+ # observe resources in it.
10
+ class Observer < Welo::Observer
11
+ include DispatchObservations
12
+ # in EPO, the source of an observation is:
13
+ # - a db object able to make a ruby object from a file path
14
+ # - a file path
15
+ Source = Struct.new(:db, :path) do
16
+ def observe
17
+ db.read_file(path)
18
+ end
19
+ end
20
+
21
+ # The DB able to tell if a path is understandable or not
22
+ attr_accessor :db
23
+
24
+ # Creates and return a new observer from a DB.
25
+ # It will take the models from the DB.
26
+ def self.for_db(db)
27
+ obj = self.new(db.models)
28
+ obj.db = db
29
+ obj
30
+ end
31
+
32
+ # Creates a new observer for given models.
33
+ # Will instanciate a new DB.
34
+ def initialize(models=[])
35
+ super(models)
36
+ @db = DB.new(models)
37
+ register(:observation) do |o|
38
+ dispatch(o)
39
+ end
40
+ end
41
+
42
+ # Read a single path, the root is the part of the path corresponding
43
+ # to where on the filesystem the database is rooted.
44
+ # e.g. for a path in a photo collection:
45
+ # /home/crapooze/project/foobar/db/photo/1
46
+ # the root is likely to be:
47
+ # /home/crapooze/project/foobar/db
48
+ # If a successful observation happens, then it will call the relevant hooks.
49
+ def read_path(path, root=nil)
50
+ full_dirname, base = File.split(path)
51
+ dirname = if root
52
+ full_dirname.sub(root,'')
53
+ else
54
+ full_dirname
55
+ end
56
+ #XXX may raise an exception for unknown path, we should rescue this/use a
57
+ #silent method and test for nil
58
+ node = db.get_route(dirname)
59
+
60
+ persp_str = db.persp_and_ext_for_basename(path).first
61
+ persp = node.content.perspectives.keys.find{|k| k.to_s == persp_str}
62
+ #XXX no need to create this many time, should cache it
63
+ st = Welo::ObservationStruct.new_for_resource_in_perspective(node.content, persp)
64
+ source = Source.new(db, path)
65
+ observe_source(source, st)
66
+ end
67
+
68
+ # Recursively reads the files in the filesystem (with Find).
69
+ # For each path, will try to read_path
70
+ # Currently, there is no pruning, or control possible.
71
+ def read_tree(root)
72
+ Find.find(root) do |path|
73
+ if File.directory?(path)
74
+ #XXX maybe prune the branch if valid but has no content
75
+ elsif db.understands_ext?(path)
76
+ read_path(path, root)
77
+ end
78
+ end
79
+ end
80
+ end
81
+ end
data/lib/epo.rb ADDED
@@ -0,0 +1,10 @@
1
+
2
+ module EPO
3
+ VERSION = "0.0.1"
4
+ AUTHORS = ["crapooze"]
5
+ WEBSITE = "https://github.com/crapooze/epo"
6
+ LICENCE = "MIT"
7
+
8
+ require 'epo/core/observer'
9
+ require 'epo/core/db'
10
+ end
metadata ADDED
@@ -0,0 +1,99 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: epo
3
+ version: !ruby/object:Gem::Version
4
+ prerelease: false
5
+ segments:
6
+ - 0
7
+ - 0
8
+ - 1
9
+ version: 0.0.1
10
+ platform: ruby
11
+ authors:
12
+ - crapooze
13
+ autorequire:
14
+ bindir: bin
15
+ cert_chain: []
16
+
17
+ date: 2011-03-19 00:00:00 -04:00
18
+ default_executable:
19
+ dependencies:
20
+ - !ruby/object:Gem::Dependency
21
+ name: derailleur
22
+ prerelease: false
23
+ requirement: &id001 !ruby/object:Gem::Requirement
24
+ none: false
25
+ requirements:
26
+ - - ">="
27
+ - !ruby/object:Gem::Version
28
+ segments:
29
+ - 0
30
+ - 0
31
+ - 5
32
+ version: 0.0.5
33
+ type: :runtime
34
+ version_requirements: *id001
35
+ - !ruby/object:Gem::Dependency
36
+ name: welo
37
+ prerelease: false
38
+ requirement: &id002 !ruby/object:Gem::Requirement
39
+ none: false
40
+ requirements:
41
+ - - ">="
42
+ - !ruby/object:Gem::Version
43
+ segments:
44
+ - 0
45
+ - 1
46
+ - 0
47
+ version: 0.1.0
48
+ type: :runtime
49
+ version_requirements: *id002
50
+ description:
51
+ email: crapooze@gmail.com
52
+ executables: []
53
+
54
+ extensions: []
55
+
56
+ extra_rdoc_files: []
57
+
58
+ files:
59
+ - Rakefile
60
+ - TODO
61
+ - README
62
+ - lib/epo.rb
63
+ - lib/epo/core/db.rb
64
+ - lib/epo/core/observer.rb
65
+ - lib/epo/core/dispatch_observations.rb
66
+ has_rdoc: true
67
+ homepage: https://github.com/crapooze/epo
68
+ licenses: []
69
+
70
+ post_install_message:
71
+ rdoc_options: []
72
+
73
+ require_paths:
74
+ - lib
75
+ required_ruby_version: !ruby/object:Gem::Requirement
76
+ none: false
77
+ requirements:
78
+ - - ">="
79
+ - !ruby/object:Gem::Version
80
+ segments:
81
+ - 0
82
+ version: "0"
83
+ required_rubygems_version: !ruby/object:Gem::Requirement
84
+ none: false
85
+ requirements:
86
+ - - ">="
87
+ - !ruby/object:Gem::Version
88
+ segments:
89
+ - 0
90
+ version: "0"
91
+ requirements: []
92
+
93
+ rubyforge_project: epo
94
+ rubygems_version: 1.3.7
95
+ signing_key:
96
+ specification_version: 3
97
+ summary: A no-brainer, plain-ruby database
98
+ test_files: []
99
+