georgi-git_store 0.2.4 → 0.3

data/README.md

@@ -10,25 +10,13 @@ in-memory representation, which can be modified and finally committed.
 GitStore supports transactions, so that updates to the store either
 fail or succeed completely.
 
-GitStore manages concurrent access by a file locking scheme. So only
-one process can start a transaction at one time. This is implemented
-by locking the `refs/head/<branch>.lock` file, which is also
-respected by the git binary.
-
 ### Installation
 
-GitStore can be installed as gem easily, if you have RubyGems 1.2.0:
+GitStore can be installed as gem easily:
 
     $ gem sources -a http://gems.github.com
     $ sudo gem install georgi-git_store
 
-If you don't have RubyGems 1.2.0, you may download the package on the
-[github page][4] and build the gem yourself:
-
-    $ gem build git_store.gemspec
-    $ sudo gem install git_store
-
-
 ### Usage Example
 
 First thing you should do, is to initialize a new git repository.
@@ -41,8 +29,6 @@ Now you can instantiate a GitStore instance and store some data. The
 data will be serialized depending on the file extension. So for YAML
 storage you can use the 'yml' extension:
 
-    @@ruby
-
     store = GitStore.new('/path/to/repo')
 
     store['users/matthias.yml'] = User.new('Matthias')
@@ -50,24 +36,17 @@ storage you can use the 'yml' extension:
 
     store.commit 'Added user and page'
 
-    # Note, that directories will be created automatically.
-    # Another way to access a path is:
-
-    store['config', 'wiki.yml'] = { 'name' => 'My Personal Wiki' }
-
-    # Finally you can access the git store as a Hash of Hashes, but in
-    # this case you have to create the Tree objects manually:
-
-    puts store['users']['wiki.yml']['name']
-
-
 ### Transactions
 
-If you access the repository from different processes, you should
-write to your store using transactions. If something goes wrong inside
-a transaction, all changes will be rolled back to the original state.
+GitStore manages concurrent access by a file locking scheme. So only
+one process can start a transaction at one time. This is implemented
+by locking the `refs/head/<branch>.lock` file, which is also
+respected by the git binary.
 
-    @@ruby
+If you access the repository from different processes or threads, you
+should write to the store using transactions. If something goes wrong
+inside a transaction, all changes will be rolled back to the original
+state.
 
     store = GitStore.new('/path/to/repo')
 
@@ -76,7 +55,8 @@ a transaction, all changes will be rolled back to the original state.
       store['pages/home.yml'] = Page.new('matthias', 'Home')
     end
 
-    # transaction without a block
+
+A transaction without a block looks like this:
 
     store.start_transaction
  
@@ -85,76 +65,16 @@ a transaction, all changes will be rolled back to the original state.
     store.rollback # This will restore the original state
 
 
-### Performance
-
-Maintaining 1000 objects in one folder seems to yield quite usable
-results. If I run the following benchmark:
-
-    @@ruby
-
-    Benchmark.bm 20 do |x|
-      x.report 'store 1000 objects' do
-        store.transaction { 'aaa'.upto('jjj') { |key| store[key] = rand.to_s } }
-      end
-      x.report 'commit one object' do
-        store.transaction { store['aa'] = rand.to_s }
-      end
-      x.report 'load 1000 objects' do
-        GitStore.new('.')
-      end
-      x.report 'load 1000 with grit' do
-        Grit::Repo.new('.').tree.contents.each { |e| e.data }
-      end  
-    end
-
-
-I get following results:
-
-                              user     system      total        real
-    store 1000 objects    4.150000   0.880000   5.030000 (  5.035804)
-    commit one object     0.070000   0.020000   0.090000 (  0.082252)
-    load 1000 objects     0.630000   0.120000   0.750000 (  0.750765)
-    load 1000 with grit   1.960000   0.260000   2.220000 (  2.228583)
-
-
-In a real world scenario, you should partition your data. For example,
-my blog engine [Shinmun][7], stores posts in folders by month.
-
-One nice thing about the results is, that GitStore loads large
-directories three times faster than [Grit][2].
-
-
-### Where is my data?
+### Data Storage
 
 When you call the `commit` method, your data is written back straight
 into the git repository. No intermediate file representation. So if
-you want to look into your data, you can use some git browser like
-[git-gui][6] or just checkout the files:
+you want to have a look at your data, you can use a git browser like
+[git-gui][6] or checkout the files:
 
     $ git checkout
 
 
-### Development Mode
-
-There is also some kind of development mode, which is convenient to
-use. Imagine you are tweaking the design of your blog, which is
-storing its pages in a GitStore. You don't want to commit each change
-to some change in your browser. FileStore helps you here:
-
-    @@ruby
-
-    store = GitStore::FileStore.new('.')
-
-    # Access the file 'posts/2009/1/git-store.md'
-
-    p store['posts', 2009, 1, 'git-store.md']
-
-
-FileStore forbids you to write to the disk, as this makes no sense. If
-you want to store something programmatically, you have to use the real
-GitStore.
-
-
 ### Iteration
 
 Iterating over the data objects is quite easy. Furthermore you can
@@ -162,15 +82,15 @@ iterate over trees and subtrees, so you can partition your data in a
 meaningful way. For example you may separate the config files and the
 pages of a wiki:
 
-    @@ruby
-
     store['pages/home.yml'] = Page.new('matthias', 'Home')
     store['pages/about.yml'] = Page.new('matthias', 'About')
-    store['pages/links.yml'] = WikiPage.new('matthias', 'Links')
     store['config/wiki.yml'] = { 'name' => 'My Personal Wiki' }
 
-    store.each { |obj| ... } # yields all pages and the config file
-    store['pages'].each { |page| ... } # yields only the pages
+    # Enumerate all objects
+    store.each { |obj| ... } 
+
+    # Enumerate only pages
+    store['pages'].each { |page| ... }
 
 
 ### Serialization
@@ -178,42 +98,39 @@ pages of a wiki:
 Serialization is dependent on the filename extension. You can add more
 handlers if you like, the interface is like this:
 
-    @@ruby
-
     class YAMLHandler
-      def read(path, data)
+      def read(data)
         YAML.load(data)
       end
    
-      def write(path, data)
+      def write(data)
         data.to_yaml
       end    
     end
 
-    GitStore::Handler['yml'] = YAMLHandler.new
-
-
 Shinmun uses its own handler for files with `md` extension:
 
-    @@ruby
-
     class PostHandler
-      def read(path, data)
-        Post.new(:path => path, :src => data)
+      def read(data)
+        Post.new(:src => data)
       end
    
-      def write(path, post)
+      def write(post)
         post.dump
       end    
     end
 
-    GitStore::Handler['md'] = PostHandler.new
+    store = GitStore.new('.')
+    store.handler['md'] = PostHandler.new
 
 
 ### GitStore on GitHub
 
 Download or fork the project on its [Github page][5]
 
+### Mailing List
+
+Please join the [GitStore Google Group][3] for further discussion.
 
 ### Related Work
 
@@ -223,6 +140,7 @@ John Wiegley already has done [something similar for Python][4].
 
 [1]: http://git.or.cz/
 [2]: http://github.com/mojombo/grit
+[3]: http://groups.google.com/group/gitstore
 [4]: http://www.newartisans.com/blog_files/git.versioned.data.store.php
 [5]: http://github.com/georgi/git_store
 [6]: http://www.kernel.org/pub/software/scm/git/docs/git-gui.html

data/git_store.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name = 'git_store'
-  s.version = '0.2.4'
+  s.version = '0.3'
   s.summary = 'a simple data store based on git'
   s.author = 'Matthias Georgi'
   s.email = 'matti.georgi@gmail.com'
@@ -17,9 +17,13 @@ README.md
 git_store.gemspec
 lib/git_store.rb
 lib/git_store/blob.rb
+lib/git_store/commit.rb
+lib/git_store/diff.rb
 lib/git_store/tree.rb
 lib/git_store/handlers.rb
 lib/git_store/pack.rb
+test/tree_spec.rb
+test/commit_spec.rb
 test/git_store_spec.rb
 test/benchmark.rb
 }

data/lib/git_store.rb

@@ -2,11 +2,14 @@ require 'rubygems'
 require 'zlib'
 require 'digest/sha1'
 require 'yaml'
+require 'fileutils'
 
 require 'git_store/blob'
+require 'git_store/diff'
 require 'git_store/tree'
-require 'git_store/handlers'
 require 'git_store/pack'
+require 'git_store/commit'
+require 'git_store/handlers'
 
 # GitStore implements a versioned data store based on the revision
 # management system git. You can store object hierarchies as nested
@@ -36,23 +39,41 @@ require 'git_store/pack'
 class GitStore
   include Enumerable
 
-  attr_reader :path, :index, :root, :branch, :lock_file, :head, :packs
+  TYPE_CLASS = {
+    'tree' => Tree,
+    'blob' => Blob,
+    'commit' => Commit
+  }     
+
+  attr_reader :path, :index, :root, :branch, :user, :lock_file, :head, :packs, :handler
 
   # Initialize a store.
   def initialize(path, branch = 'master')
-    @path   = path.chomp('/')
-    @branch = branch
-    @root   = Tree.new(self)
-    @packs  = {}
+    if not File.exists?("#{path}/.git")
+      raise ArgumentError, "first argument must be a valid Git repository: `#{path}'"
+    end
+    
+    @path    = path.chomp('/')
+    @branch  = branch
+    @root    = Tree.new(self)
+    @packs   = {}
+    
+    init_handler
+    
+    name = IO.popen("git config user.name")  { |io| io.gets.chomp }
+    email = IO.popen("git config user.email") { |io| io.gets.chomp }
+    
+    @user = "#{name} <#{email}>"
     
-    raise(ArgumentError, "first argument must be a valid Git repository") unless valid?
-
     load_packs("#{path}/.git/objects/pack")
     load
   end
 
-  def valid?
-    File.exists?("#{path}/.git")
+  def init_handler
+    @handler = {
+      'yml' => YAMLHandler.new
+    }    
+    @handler.default = DefaultHandler.new
   end
 
   # The path to the current head file.
@@ -68,30 +89,43 @@ class GitStore
   # Read the id of the head commit.
   #
   # Returns the object id of the last commit.
-  def read_head
+  def read_head_id
     File.read(head_path).strip if File.exists?(head_path)
   end
 
+  def handler_for(path)
+    handler[ path.split('.').last ]
+  end
+
   # Read an object for the specified path.
-  #
-  # Use multiple arguments or a string with slashes.
-  def [](*args)
-    root[*args]
+  def [](path)
+    root[path] 
   end
 
   # Write an object to the specified path.
-  #
-  # Use multiple arguments or a string with slashes.
-  def []=(*args)
-    value = args.pop
-    root[*args] = value
+  def []=(path, data)
+    root[path] = data
   end
 
-  # Delete the specified path.
-  #
-  # Use multiple arguments or a string with slashes.
-  def delete(*args)
-    root.delete(*args)
+  # Iterate over all key-values pairs found in this store.
+  def each(&block)
+    root.each(&block)
+  end
+
+  def paths
+    root.paths
+  end
+
+  def values
+    root.values
+  end
+
+  def delete(path)
+    root.delete(path)
+  end
+
+  def tree(name)
+    root.tree(name)
   end
 
   # Returns the store as a hash tree.
@@ -101,30 +135,38 @@ class GitStore
 
   # Inspect the store.
   def inspect
-    "#<GitStore #{path} #{branch} #{root.to_hash.inspect}>"
-  end
-
-  # Iterate over all values found in this store.
-  def each(&block)
-    root.each(&block)
+    "#<GitStore #{path} #{branch}>"
   end
 
   # Has our store been changed on disk?
   def changed?
-    head != read_head
-  end
-
-  def refresh!
-    load if changed?
+    head.nil? or head.id != read_head_id
   end
 
   # Load the current head version from repository. 
-  def load
-    if @head = read_head
-      commit = get_object(head)[0]
-      root.id = commit.split(/[ \n]/, 3)[1].strip
-      root.data = get_object(root.id)[0]
-      root.load_from_store
+  def load(from_disk = false)
+    if id = read_head_id
+      @head = get(id)
+      @root = get(@head.tree)
+    end
+    
+    load_from_disk if from_disk
+  end
+  
+  def load_from_disk
+    @mtime ||= {}
+    
+    root.each_blob do |path, blob|
+      file = "#{self.path}/#{path}"
+      
+      if File.file?(file)
+        mtime = File.mtime(file)
+        
+        if @mtime[path] != mtime
+          @mtime[path] = mtime
+          blob.data = File.read(file)
+        end
+      end
     end
   end
 
@@ -133,7 +175,7 @@ class GitStore
     load if changed?
   end
 
-  # Do we have a current transacation?
+  # Is there any transaction going on?
   def in_transaction?
     Thread.current['git_store_lock']
   end
@@ -174,7 +216,7 @@ class GitStore
   #
   # Any changes made to the store are discarded.
   def rollback
-    root.load_from_store
+    load
     finish_transaction
   end
   
@@ -188,25 +230,56 @@ class GitStore
     File.unlink("#{head_path}.lock") rescue nil    
   end
 
+  def user_info(user, time)
+    "#{ user } #{ time.to_i } #{ time.to_s.split[4] }"
+  end
+
   # Write the commit object to disk and set the head of the current branch.
   #
   # Returns the id of the commit object
-  def commit(message = '', author = 'ruby', committer = 'ruby')
-    time = "#{ Time.now.to_i } #{ Time.now.to_s.split[4] }"
-    tree = root.write_to_store
+  def commit(message = '', author = "#{user_info user, Time.now}", committer = "#{user_info user, Time.now}")
+    commit = Commit.new(self)
+    commit.tree = root.write
+    commit.parent << head.id if head
+    commit.author = author
+    commit.committer = committer
+    commit.message = message
+    commit.write
 
-    contents = [ "tree #{tree}", (head and "parent #{head}"),
-                 "author #{author} #{time}",
-                 "committer #{committer} #{time}", '', message
-                 ].compact.join("\n")
+    open(head_path, "wb") do |file|
+      file.write(commit.id)
+    end
 
-    id = put_object(contents, 'commit')
+    @head = commit
+  end
 
-    open(head_path, "wb") do |file|
-      file.write(id)
+  def commits(limit = 10, start = head)
+    entries = []
+    current = start
+    
+    while current and entries.size < limit
+      entries << current
+      current = get(current.parent.first)
     end
 
-    @head = id
+    entries
+  end
+
+  def get(id)
+    return nil if id.nil?
+    type, content = get_object(id)
+
+    klass = TYPE_CLASS[type]
+    klass.new(self, id, content)
+  end
+
+  # Returns the hash value of an object string.
+  def sha(str)
+    Digest::SHA1.hexdigest(str)[0, 40]
+  end
+
+  def id_for(type, content)
+    sha "#{type} #{content.length}\0#{content}"
   end
 
   # Read the raw object with the given id from the repository.
@@ -218,32 +291,26 @@ class GitStore
     if File.exists?(path)
       buf = open(path, "rb") { |f| f.read }
 
-      raise if not legacy_loose_object?(buf)
+      raise "not a loose object: #{id}" if not legacy_loose_object?(buf)
       
       header, content = Zlib::Inflate.inflate(buf).split(/\0/, 2)
       type, size = header.split(/ /, 2)
+      
+      raise "bad object: #{id}" if content.length != size.to_i
     else
       content, type = get_object_from_pack(id)
     end
     
-    return content, type
+    return type, content
   end
-
-  # Returns the hash value of an object string.
-  def sha(str)
-    Digest::SHA1.hexdigest(str)[0, 40]
-  end
-
+  
   # Write a raw object to the repository.
   #
   # Returns the object id.
-  def put_object(content, type)
-    size = content.length.to_s    
-    header = "#{type} #{size}\0"
-    data = header + content
-    
+  def put_object(type, content)
+    data = "#{type} #{content.length}\0#{content}"    
     id = sha(data)
-    path = object_path(id)    
+    path = object_path(id)
     
     unless File.exists?(path)
       FileUtils.mkpath(File.dirname(path))
@@ -263,7 +330,7 @@ class GitStore
 
   def get_object_from_pack(id)
     pack, offset = @packs[id]
-        
+    
     pack.parse_object(offset) if pack
   end      
 
@@ -281,53 +348,5 @@ class GitStore
       end
     end
   end
-  
-  # FileStore reads a working copy out of a directory. Changes made to
-  # the store will not be written to a repository. This is useful, if
-  # you want to read a filesystem without having a git repository.
-  class FileStore < GitStore
-
-    def initialize(path)
-      @mtime = {}
-      super
-    rescue ArgumentError
-    end
-
-    def load
-      root.load_from_disk
-      
-      each_blob_in(root) do |blob|
-        @mtime[blob.path] = File.mtime("#{path}/#{blob.path}")
-      end
-    end
-
-    def each_blob_in(tree, &blob)
-      tree.table.each do |name, entry|
-        case entry
-        when Blob; yield entry
-        when Tree; each_blob_in(entry, &blob)
-        end
-      end
-    end        
-
-    def refresh!
-      each_blob_in(root) do |blob|
-        path = "#{self.path}/#{blob.path}"
-        if File.exist?(path)
-          mtime = File.mtime(path)
-          if @mtime[blob.path] != mtime
-            @mtime[blob.path] = mtime
-            blob.load_from_disk
-          end
-        else
-          delete blob.path
-        end
-      end
-    end
-
-    def commit(message="")
-    end
-    
-  end
 
 end