georgi-git_store 0.1.1 → 0.2

data/.gitignore

@@ -1,3 +1,4 @@
 *~
 doc/*
 pkg/*
+test/repo

data/README.md

@@ -1,21 +1,25 @@
 Git Store - using Git as versioned data store in Ruby
 =====================================================
 
-GitStore is a small Ruby library, providing an easy interface to the
-version control system [Git][1]. It aims to use Git as a versioned
-data store much like the well known PStore. Basically GitStore checks
-out the repository into a in-memory representation, which can be
-modified and finally committed. In this way your data is stored in a
-folder structure and can be checked out and examined, but the
-application may access the data in a convenient hash-like way. This
-library is based on [Grit][2], the main technology behind [GitHub][3].
+GitStore implements a versioned data store based on the revision
+management system [Git][1]. You can store object hierarchies as nested
+hashes, which will be mapped on the directory structure of a git
+repository. Basically GitStore checks out the repository into a
+in-memory representation, which can be modified and finally committed.
 
+GitStore supports transactions, so that updates to the store either
+fail or succeed completely.
 
-## Installation
+GitStore manages concurrent access by a file locking scheme. So only
+one process can start a transaction at one time. This is implemented
+by locking the `refs/head/<branch>.lock` file, which is also
+respected by the git binary.
+
+### Installation
 
 GitStore can be installed as gem easily, if you have RubyGems 1.2.0:
 
-    $ gem sources -a http://gems.github.com you only have to do this once)
+    $ gem sources -a http://gems.github.com
     $ sudo gem install georgi-git_store
 
 If you don't have RubyGems 1.2.0, you may download the package on the
@@ -25,7 +29,7 @@ If you don't have RubyGems 1.2.0, you may download the package on the
     $ sudo gem install git_store
 
 
-## Usage Example
+### Usage Example
 
 First thing you should do, is to initialize a new git repository.
 
@@ -37,29 +41,90 @@ Now you can instantiate a GitStore instance and store some data. The
 data will be serialized depending on the file extension. So for YAML
 storage you can use the 'yml' extension:
 
-    class WikiPage < Struct.new(:author, :title, :body); end
-    class User < Struct.new(:name); end
+    @@ruby
 
-    store = GitStore.new('.')
+    store = GitStore.new('/path/to/repo')
 
     store['users/matthias.yml'] = User.new('Matthias')
-    store['pages/home.yml'] = WikiPage.new('matthias', 'Home', 'This is the home page...')
+    store['pages/home.yml'] = Page.new('matthias', 'Home')
 
     store.commit 'Added user and page'
 
-Note that directories will be created automatically.
+    # Note, that directories will be created automatically.
+    # Another way to access a path is:
+
+    store['config', 'wiki.yml'] = { 'name' => 'My Personal Wiki' }
+
+    # Finally you can access the git store as a Hash of Hashes, but in
+    # this case you have to create the Tree objects manually:
+
+    puts store['users']['wiki.yml']['name']
+
+
+### Transactions
+
+If you access the repository from different processes, you should
+write to your store using transactions. If something goes wrong inside
+a transaction, all changes will be rolled back to the original state.
+
+    @@ruby
+
+    store = GitStore.new('/path/to/repo')
+
+    store.transaction do
+      # If an exception happens here, the transaction will be aborted.
+      store['pages/home.yml'] = Page.new('matthias', 'Home')
+    end
+
+    # transaction without a block
+
+    store.start_transaction
+ 
+    store['pages/home.yml'] = Page.new('matthias', 'Home')
+
+    store.rollback # This will restore the original state
+
+
+### Performance
+
+Maintaining 1000 objects in one folder seems to yield quite usable
+results. If I run the following benchmark:
 
-Another way to access a path is:
+    @@ruby
+
+    Benchmark.bm 20 do |x|
+      x.report 'store 1000 objects' do
+        store.transaction { 'aaa'.upto('jjj') { |key| store[key] = rand.to_s } }
+      end
+      x.report 'commit one object' do
+        store.transaction { store['aa'] = rand.to_s }
+      end
+      x.report 'load 1000 objects' do
+        GitStore.new('.')
+      end
+      x.report 'load 1000 with grit' do
+        Grit::Repo.new('.').tree.contents.each { |e| e.data }
+      end  
+    end
 
-    store[config', 'wiki.yml'] = { 'name' => 'My Personal Wiki' }
 
-Finally you can access the git store as a Hash of Hashes, but in this
-case you have to create the Tree objects manually:
+I get following results:
 
-    store['users'] = GitStore::Tree.new
-    store['users']['matthias.yml'] = User.new('Matthias')
+                              user     system      total        real
+    store 1000 objects    4.150000   0.880000   5.030000 (  5.035804)
+    commit one object     0.070000   0.020000   0.090000 (  0.082252)
+    load 1000 objects     0.630000   0.120000   0.750000 (  0.750765)
+    load 1000 with grit   1.960000   0.260000   2.220000 (  2.228583)
 
-## Where is my data?
+
+In a real world scenario, you should partition your data. For example,
+my blog engine [Shinmun][7], stores posts in folders by month.
+
+One nice thing about the results is, that GitStore loads large
+directories three times faster than [Grit][2].
+
+
+### Where is my data?
 
 When you call the `commit` method, your data is written back straight
 into the git repository. No intermediate file representation. So if
@@ -69,33 +134,58 @@ you want to look into your data, you can use some git browser like
     $ git checkout
 
 
-## Iteration
+### Development Mode
+
+There is also some kind of development mode, which is convenient to
+use. Imagine you are tweaking the design of your blog, which is
+storing its pages in a GitStore. You don't want to commit each change
+to some change in your browser. FileStore helps you here:
+
+    @@ruby
+
+    store = GitStore::FileStore.new('.')
+
+    # Access the file 'posts/2009/1/git-store.md'
+
+    p store['posts', 2009, 1, 'git-store.md']
+
+
+FileStore forbids you to write to the disk, as this makes no sense. If
+you want to store something programmatically, you have to use the real
+GitStore.
+
+
+### Iteration
 
 Iterating over the data objects is quite easy. Furthermore you can
 iterate over trees and subtrees, so you can partition your data in a
 meaningful way. For example you may separate the config files and the
 pages of a wiki:
 
-    store['pages/home.yml'] = WikiPage.new('matthias', 'Home', 'This is the home page...')
-    store['pages/about.yml'] = WikiPage.new('matthias', About', 'About this site...')
-    store['pages/links.yml'] = WikiPage.new('matthias', 'Links', 'Some useful links...')
+    @@ruby
+
+    store['pages/home.yml'] = Page.new('matthias', 'Home')
+    store['pages/about.yml'] = Page.new('matthias', 'About')
+    store['pages/links.yml'] = WikiPage.new('matthias', 'Links')
     store['config/wiki.yml'] = { 'name' => 'My Personal Wiki' }
 
-    store.each { |obj| ... } # yields all pages and the config hash
+    store.each { |obj| ... } # yields all pages and the config file
     store['pages'].each { |page| ... } # yields only the pages
 
 
-## Serialization
+### Serialization
 
 Serialization is dependent on the filename extension. You can add more
 handlers if you like, the interface is like this:
 
+    @@ruby
+
     class YAMLHandler
-      def read(id, name, data)
+      def read(path, data)
         YAML.load(data)
       end
    
-      def write(data)
+      def write(path, data)
         data.to_yaml
       end    
     end
@@ -105,28 +195,35 @@ handlers if you like, the interface is like this:
 
 Shinmun uses its own handler for files with `md` extension:
 
+    @@ruby
+
     class PostHandler
-      def read(name, data)
-        Post.new(:filename => name, :src => data)
+      def read(path, data)
+        Post.new(:path => path, :src => data)
       end
    
-      def write(post)
+      def write(path, post)
         post.dump
       end    
     end
 
-    GitStore::Handler[md'] = PostHandler.new
+    GitStore::Handler['md'] = PostHandler.new
+
+
+### GitStore on GitHub
+
+Download or fork the project on its [Github page][5]
+
+
+### Related Work
 
+John Wiegley already has done [something similar for Python][4].
 
-## Related Work
 
-John Wiegley already has done [something similar for Python][4]. His
-implementation has its own git interface, GitStore uses the wonderful
-[Grit][2] library.
 
 [1]: http://git.or.cz/
 [2]: http://github.com/mojombo/grit
-[3]: http://github.com/
 [4]: http://www.newartisans.com/blog_files/git.versioned.data.store.php
 [5]: http://github.com/georgi/git_store
 [6]: http://www.kernel.org/pub/software/scm/git/docs/git-gui.html
+[7]: http://www.matthias-georgi.de/shinmun

data/git_store.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'git_store'
-  s.version = '0.1.1'
+  s.version = '0.2'
   s.date = '2008-12-17'
   s.summary = 'a simple data store based on git'
   s.author = 'Matthias Georgi'
@@ -16,7 +16,12 @@ LICENSE
 README.md
 git_store.gemspec
 lib/git_store.rb
-spec/git_store_spec.rb
+lib/git_store/blob.rb
+lib/git_store/tree.rb
+lib/git_store/handlers.rb
+lib/git_store/pack.rb
+test/git_store_spec.rb
+test/benchmark.rb
 }
 end
 

data/lib/git_store/blob.rb

@@ -0,0 +1,64 @@
+class GitStore
+
+  # This class stores the raw string data of a blob, but also the
+  # deserialized data object.
+  class Blob
+
+    attr_accessor :store, :id, :mode, :path, :data
+
+    # Initialize a Blob with default mode of '100644'.
+    def initialize(store)
+      @store = store
+      @mode = '100644'
+    end
+
+    # Set all attributes at once.
+    def set(id, mode = nil, path = nil, data = nil, object = nil)
+      @id, @mode, @path, @data, @object = id, mode, path, data, object
+    end
+
+    # Returns the extension of the filename.
+    def extname
+      File.extname(path)[1..-1]
+    end
+
+    # Returns the handler for serializing the blob data.
+    def handler
+      Handler[extname]
+    end
+
+    # Returns true if data is new or hash value is different from current id.
+    def modified?
+      id.nil? || @modified
+    end
+
+    # Returns the data object.
+    def object
+      @object ||= handler.read(path, data)
+    end
+
+    # Set the data object.
+    def object=(value)
+      @modified = true
+      @object = value
+      @data = handler.respond_to?(:write) ? handler.write(path, value) : value
+    end
+
+    def load_from_disk
+      @object = nil
+      @data = open("#{store.path}/#{path}", 'rb') { |f| f.read }
+    end
+
+    # Write the data to the git object store
+    def write_to_store      
+      if modified?
+        @modified = false
+        @id = store.put_object(data, 'blob')
+      else
+        @id
+      end
+    end   
+
+  end
+
+end

data/lib/git_store/handlers.rb

@@ -0,0 +1,57 @@
+
+# This fix ensures sorted yaml maps.
+class Hash
+	def to_yaml( opts = {} )
+		YAML::quick_emit( object_id, opts ) do |out|
+      out.map( taguri, to_yaml_style ) do |map|
+        sort_by { |k, v| k.to_s }.each do |k, v|
+          map.add( k, v )
+        end
+      end
+    end
+	end
+end
+
+class GitStore
+
+  class DefaultHandler
+    def read(path, data)
+      data
+    end
+    
+    def write(path, data)
+      data.to_s
+    end
+  end
+  
+  class YAMLHandler    
+    def read(path, data)
+      YAML.load(data)
+    end
+
+    def write(path, data)
+      data.to_yaml
+    end    
+  end
+
+  class RubyHandler
+    def read(path, data)
+      Object.module_eval(data)
+    end
+  end
+
+  class ERBHandler
+    def read(path, data)
+      ERB.new(data)
+    end
+  end
+
+  Handler = {
+    'yml' => YAMLHandler.new,
+    'rhtml' => ERBHandler.new,
+    'rxml' => ERBHandler.new,
+    'rb' => RubyHandler.new
+  }
+
+  Handler.default = DefaultHandler.new
+end