rod 0.6.0 → 0.6.1

data/README

@@ -10,17 +10,25 @@ fast access for data, which rarely changes.
 == FEATURES/PROBLEMS:
 
 * nice Ruby interface which mimicks Active Record
-* Ruby-to-C on-the-fly translation based on mmap
-* optimized for speed
+* Ruby-to-C on-the-fly translation based on mmap and RubyInline
+* optimized for (reading) speed
 * weak reference collections for easy memory reclaims
-* segmented indices for short start-up time
+* segmented indices for short start-up times
+* compatibility check of library version
+* compatibility check of data model
+* autogeneration of model (based on the database metadata)
+* automatic model migrations (addition/removal of properties so far)
+* append of the database (new objects, new elements in plural associations)
+* databases interlinking (via direct links or inverted indices)
 
 * doesn't work on Windows
+* some space is wasted during when database is appended
 
 == SYNOPSIS:
 
 ROD is designed for storing and accessing data which rarely changes.
-It is an opposite of RDBMS as the data is not normalized.
+It is an opposite of RDBMS as the data is not normalized, while
+"joins" are much faster.
 It is an opposite of in-memory databases, since it is designed to cover
 out of core data sets (10 GB and more).
 
@@ -30,17 +38,15 @@ is interconnected in many ways, thus the relational model (joins) introduces
 unacceptable performance hit. The size of corpora forces them to be kept
 on disks. The in-memory data bases are unacceptable for larg corpora and
 would require the data to be kept mostly in the operational memory,
-which is not needed, while accessing dictionaries (in most cases only a friction
-of the data is needed). That's why a storage facility which minimizes the
-number of disk reads was designed. The Ruby interface facilitates it's
-usage.
+which is not needed, while accessing dictionaries (in most cases only a fraction
+of the data is used at the same time). That's why a storage facility which minimizes the
+number of disk reads was designed. The Ruby interface facilitates it's usage.
 
 == REQUIREMENTS:
 
 * RubyInline
 * english
 * ActiveModel
-* weak_hash
 
 == INSTALL
 
@@ -77,23 +83,21 @@ Grab from rubygems:
  end
 
  MyDatabase.create_database("data")
- user = User.new
- user.name = 'Fred'
- user.surname = 'Smith'
- user.age = 22
- account = Account.new
- account.email = "fred@smith.org"
- account.login = "fred"
- account.password = "password"
- file1 = File.new
- file1.title = "Lady Gaga video"
+ user = User.new(:name => 'Fred',
+                 :surname => 'Smith',
+                 :age => 22)
+ account = Account.new(:email => "fred@smith.org",
+                       :login => "fred",
+                       :password => "password")
+ file1 = File.new(:title => "Lady Gaga video")
  file2.data = "0012220001..."
- file2 = File.new
- file2.title = "Pink Floyd video"
+ file2 = File.new(:title => "Pink Floyd video")
  file2.data = "0012220001..."
+
  user.account = account
  user.files << file1
  user.files << file2
+
  user.store
  account.store
  file1.store
@@ -118,7 +122,7 @@ Grab from rubygems:
 
 (The MIT License)
 
-Copyright (c) 2008-2010 Aleksander Pohl
+Copyright (c) 2008-2011 Aleksander Pohl
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/Rakefile

@@ -1,7 +1,7 @@
 $:.unshift "lib"
 require 'rod/constants'
 
-task :default => [:install]
+task :default => [:all_tests]
 
 $gem_name = "rod"
 
@@ -22,7 +22,7 @@ task :uninstall do
   sh "sudo gem uninstall #$gem_name"
 end
 
-task :all_tests => [:test,:spec,:regression_test] do
+task :all_tests => [:test,:regression_test,:spec] do
 end
 
 desc "Run performence tests"
@@ -36,9 +36,18 @@ desc "Run tests and specs"
 task :test do
   sh "ruby tests/save_struct.rb"
   sh "ruby tests/load_struct.rb"
+  sh "ruby tests/class_compatibility_create.rb"
+  sh "ruby tests/class_compatibility_verify.rb"
+  sh "ruby tests/generate_classes_create.rb"
+  sh "ruby tests/generate_classes_rewrite.rb"
+  sh "ruby tests/generate_classes_rewrite.rb"
+  sh "ruby tests/generate_classes_verify.rb"
+  sh "ruby tests/migration_create.rb"
+  sh "ruby tests/migration_migrate.rb"
+  sh "ruby tests/migration_verify.rb"
   sh "ruby tests/unit/model.rb"
   sh "ruby tests/unit/model_tests.rb"
-  sh "ruby tests/unit/abstract_database.rb"
+  sh "ruby tests/unit/database.rb"
 end
 
 # Should be removed some time -- specs should cover all these cases
@@ -48,7 +57,12 @@ task :regression_test do
 end
 
 task :spec do
-  sh "bundle exec cucumber features/*"
+  sh "bundle exec cucumber --tags ~@ignore features/*"
+end
+
+# Work in progress
+task :wip do
+  sh "bundle exec cucumber --tags @wip features/*"
 end
 
 desc "Clean"

data/changelog.txt

@@ -1,3 +1,36 @@
+0.6.1
+- Change order of tests in all_tests
+- Update DB 'created_at' if empty
+- #112 fix: indices are rewritten during migration
+- #110 bulk allocate for objects durign migration
+- #111 back-up of database.yml during migration
+- Fix: uninitilized module name
+- Make incompatible model version message more inteligible
+- Make index rewrite/convert option more meaningful
+- #108 fix: indices are destroyed for read/write mode
+- #81 automatic schema transformation
+- #104 remove ruby inline generated files when model is generated
+- tests/read_on_create.rb remove ruby inline and data files during creation
+- #100 fix: removal of RubyInline generated files
+- #43 generate classes from metadata
+- #43 generate classes without module embedding
+-  #96 fix: find_by for assocs
+- #95 remove weak_hash dependency
+- #25 check compatibility of class structure
+- #87 store supreclass of a class in metadata
+- #26 check version of library: fix revision check - Add ~@ignore tag as default for specs
+- #93 fix: storage of has_many appended objects - Turn on full object referential integrity
+- #65 store DB creation and update time in metadata
+- #89 Replace SimpleWeakHash with Cache
+- #90 Remove blocks from collection proxy
+- #80 tests and proper fix for segfault on _allocate_polymorphic
+- Add check for index of already stored object
+- #73 append of has many relationship
+- #82 index (flat -> segmented) transformation tool
+- #86 Cache indexed properties (optimiz.)
+- Remove legacy examples
+- #80 sefault on _allocate_polymorphic_join_elements
+- #79 fix: reading indices during DB creation causes an error
 0.6.0
 - #64 index for associations
 - Update legacy test to new API

data/lib/rod.rb

@@ -1,6 +1,5 @@
 require 'inline'
 require 'english/inflect'
-require 'simple_weak_hash'
 require 'active_model'
 require 'active_support/dependencies'
 
@@ -10,10 +9,12 @@ require 'active_support/dependencies'
 ActiveSupport::Dependencies.mechanism = :require
 
 require 'rod/abstract_database'
+require 'rod/abstract_model'
 require 'rod/constants'
 require 'rod/database'
 require 'rod/exception'
 require 'rod/join_element'
+require 'rod/cache'
 require 'rod/collection_proxy'
 require 'rod/model'
 require 'rod/string_element'

data/lib/rod/abstract_database.rb

@@ -1,6 +1,7 @@
 require 'singleton'
 require 'yaml'
 require 'rod/segmented_index'
+require 'fileutils'
 
 module Rod
   # This class implements the database abstraction, i.e. it
@@ -12,9 +13,12 @@ module Rod
     # a given model (set of classes).
     include Singleton
 
+    # The meta-data of the DataBase.
+    attr_reader :metadata
+
     # Initializes the classes linked with this database and the handler.
     def initialize
-      @classes ||= self.class.special_classes
+      @classes ||= self.special_classes
       @handler = nil
     end
 
@@ -44,54 +48,127 @@ module Rod
     def create_database(path)
       raise DatabaseError.new("Database already opened.") unless @handler.nil?
       @readonly = false
-      self.classes.each{|s| s.send(:build_structure)}
       @path = canonicalize_path(path)
-      # XXX maybe should be more careful?
       if File.exist?(@path)
-        Dir.glob("#{@path}**/*").each do |file_name|
-          File.delete(file_name) unless File.directory?(file_name)
-        end
+        remove_file("#{@path}database.yml")
       else
         Dir.mkdir(@path)
       end
+      self.classes.each do |klass|
+        klass.send(:build_structure)
+        remove_file(klass.path_for_data(@path))
+        klass.indexed_properties.each do |property,options|
+          path = klass.path_for_index(@path,property,options)
+          if test(?d,path)
+            remove_files(path + "*")
+          elsif test(?f,path)
+            remove_file(path)
+          end
+        end
+        next if special_class?(klass)
+        remove_files_but(klass.inline_library)
+      end
       generate_c_code(@path, classes)
+      remove_files_but(self.inline_library)
+      @metadata = {}
+      @metadata["Rod"] = {}
+      @metadata["Rod"][:created_at] = Time.now
       @handler = _init_handler(@path)
       _create(@handler)
     end
 
-    # Opens the database at +path+ for readonly mode. This allows
+    # Opens the database at +path+ with +options+. This allows
     # for Rod::Model.count, Rod::Model.each, and similar calls.
-    #
-    # By default the database is opened in +readonly+ mode. You
-    # can change it by passing +false+ as the second argument.
-    def open_database(path,readonly=true)
+    # Options:
+    # * +:readonly+ - no modifiaction (append of models and has many association)
+    #   is allowed (defaults to +true+)
+    # * +:generate+ - value could be true or a module. If present, generates
+    #   the classes from the database metadata. If module given, the classes
+    #   are generated withing the module.
+    def open_database(path,options={:readonly => true})
       raise DatabaseError.new("Database already opened.") unless @handler.nil?
-      @readonly = readonly
-      self.classes.each{|s| s.send(:build_structure)}
+      options = convert_options(options)
+      @readonly = options[:readonly]
       @path = canonicalize_path(path)
-      generate_c_code(@path, classes)
-      metadata = {}
+      @metadata = {}
       File.open(@path + DATABASE_FILE) do |input|
-        metadata = YAML::load(input)
+        @metadata = YAML::load(input)
       end
-      unless valid_version?(metadata["Rod"][:version])
-        raise RodException.new("Incompatible versions - library #{VERSION} vs. file #{metatdata["Rod"][:version]}")
+      unless valid_version?(@metadata["Rod"][:version])
+        raise IncompatibleVersion.new("Incompatible versions - library #{VERSION} vs. " +
+                                      "file #{metatdata["Rod"][:version]}")
       end
+      if options[:generate]
+        module_instance = (options[:generate] == true ? Object : options[:generate])
+        generate_classes(module_instance)
+      elsif options[:migrate]
+        create_legacy_classes
+        FileUtils.cp(@path + DATABASE_FILE, @path + DATABASE_FILE + LEGACY_DATA_SUFFIX)
+      end
+      self.classes.each do |klass|
+        klass.send(:build_structure)
+        next if special_class?(klass)
+        if options[:generate] && module_instance != Object
+          remove_files_but(klass.inline_library)
+        end
+      end
+      generate_c_code(@path, self.classes)
       @handler = _init_handler(@path)
       self.classes.each do |klass|
-        meta = metadata[klass.name]
+        meta = @metadata[klass.name]
         if meta.nil?
           # new class
           next
         end
+        unless klass.compatible?(meta,self) || options[:generate] || options[:migrate]
+            raise IncompatibleVersion.
+              new("Incompatible definition of '#{klass.name}' class.\n" +
+                  "Database and runtime versions are different:\n" +
+                  "  #{meta}\n  #{klass.metadata(self)}")
+        end
         set_count(klass,meta[:count])
         file_size = File.new(klass.path_for_data(@path)).size
         unless file_size % _page_size == 0
           raise DatabaseError.new("Size of data file of #{klass} is invalid: #{file_size}")
         end
         set_page_count(klass,file_size / _page_size)
+        if options[:migrate]
+          next unless klass.name =~ LEGACY_RE
+          new_class = klass.name.sub(LEGACY_RE,"").constantize
+          set_count(new_class,meta[:count])
+          pages = (meta[:count] * new_class.struct_size / _page_size.to_f).ceil
+          set_page_count(new_class,pages)
+        end
       end
       _open(@handler)
+      if options[:migrate]
+        empty_data = "\0" * _page_size
+        self.classes.each do |klass|
+          next unless klass.to_s =~ LEGACY_RE
+          new_class = klass.name.sub(LEGACY_RE,"").constantize
+          old_metadata = klass.metadata(self)
+          old_metadata.merge!({:superclass => old_metadata[:superclass].sub(LEGACY_RE,"")})
+          unless new_class.compatible?(old_metadata,self)
+            File.open(new_class.path_for_data(@path),"w") do |out|
+              send("_#{new_class.struct_name}_page_count",@handler).
+                times{|i| out.print(empty_data)}
+            end
+            klass.migrate
+            current_file_name = klass.path_for_data(@path)
+            legacy_file_name = current_file_name + LEGACY_DATA_SUFFIX
+            new_file_name = new_class.path_for_data(@path)
+            FileUtils.mv(current_file_name,legacy_file_name)
+            FileUtils.mv(new_file_name,current_file_name)
+          end
+          @classes.delete(klass)
+          new_class.model_path = nil
+        end
+        close_database(false,true)
+        options.delete(:migrate)
+        readonly = options.delete(:old_readonly)
+        options[:readonly] = readonly
+        open_database(path,options)
+      end
     end
 
     # Closes the database.
@@ -99,58 +176,36 @@ module Rod
     # If the +purge_classes+ flag is set to true, the information about the classes
     # linked with this database is removed. This is important for testing, when
     # classes with same names have different definitions.
-    def close_database(purge_classes=false)
+    #
+    # If the +skip_indeces+ flat is set to true, the indices are not written.
+    def close_database(purge_classes=false,skip_indices=false)
       raise DatabaseError.new("Database not opened.") if @handler.nil?
 
       unless readonly_data?
         unless referenced_objects.select{|k, v| not v.empty?}.size == 0
           raise DatabaseError.new("Not all associations have been stored: #{referenced_objects.size} objects")
         end
-        metadata = {}
-        rod_data = metadata["Rod"] = {}
-        rod_data[:version] = VERSION
-        self.classes.each do |klass|
-          meta = metadata[klass.name] = {}
-          meta[:count] = count(klass)
-          next if special_class?(klass)
-          # fields
-          fields = meta[:fields] = {} unless klass.fields.empty?
-          klass.fields.each do |field,options|
-            fields[field] = {}
-            fields[field][:options] = options
-            write_index(klass,field,options) if options[:index]
-          end
-          # singular_associations
-          has_one = meta[:has_one] = {} unless klass.singular_associations.empty?
-          klass.singular_associations.each do |name,options|
-            has_one[name] = {}
-            has_one[name][:options] = options
-            write_index(klass,name,options) if options[:index]
-          end
-          # plural_associations
-          has_many = meta[:has_many] = {} unless klass.plural_associations.empty?
-          klass.plural_associations.each do |name,options|
-            has_many[name] = {}
-            has_many[name][:options] = options
-            write_index(klass,name,options) if options[:index]
+        unless skip_indices
+          self.classes.each do |klass|
+            klass.indexed_properties.each do |property,options|
+              write_index(klass,property,options)
+            end
           end
         end
-        File.open(@path + DATABASE_FILE,"w") do |out|
-          out.puts(YAML::dump(metadata))
-        end
+        write_metadata
       end
       _close(@handler)
       @handler = nil
       # clear cached data
       self.clear_cache
       if purge_classes
-        @classes = self.class.special_classes
+        @classes = self.special_classes
       end
     end
 
     # Clears the cache of the database.
     def clear_cache
-      classes.each{|c| c.cache.send(:__get_hash__).clear}
+      classes.each{|c| c.cache.clear }
     end
 
     #########################################################################
@@ -264,19 +319,20 @@ module Rod
     def write_index(klass,property,options)
       raise DatabaseError.new("Readonly database.") if readonly_data?
       class_index = klass.index_for(property,options)
-      class_index.each do |key,ids|
-        unless ids.is_a?(CollectionProxy)
-          proxy = CollectionProxy.new(ids[1]) do |index|
-            [join_index(ids[0],index), klass]
+      # Only convert the index, without (re)storing the values.
+      unless options[:convert]
+        class_index.each do |key,ids|
+          unless ids.is_a?(CollectionProxy)
+            proxy = CollectionProxy.new(ids[1],self,ids[0],klass)
+          else
+            proxy = ids
           end
-        else
-          proxy = ids
-        end
-        offset = _allocate_join_elements(proxy.size,@handler)
-        proxy.each_id.with_index do |rod_id,index|
-          set_join_element_id(offset, index, rod_id)
+          offset = _allocate_join_elements(proxy.size,@handler)
+          proxy.each_id.with_index do |rod_id,index|
+            set_join_element_id(offset, index, rod_id)
+          end
+          class_index[key] = [offset,proxy.size]
         end
-        class_index[key] = [offset,proxy.size]
       end
       case options[:index]
       when :flat,true
@@ -301,17 +357,21 @@ module Rod
     # Store the object in the database.
     def store(klass,object)
       raise DatabaseError.new("Readonly database.") if readonly_data?
-      send("_store_" + klass.struct_name,object,@handler)
-      # set fields' values
-      object.class.fields.each do |name,options|
-        # rod_id is set during _store
-        object.update_field(name) unless name == "rod_id"
-      end
-      # set ids of objects referenced via singular associations
-      object.class.singular_associations.each do |name,options|
-        object.update_singular_association(name,object.send(name))
+      new_object = (object.rod_id == 0)
+      if new_object
+        send("_store_" + klass.struct_name,object,@handler)
+        # set fields' values
+        object.class.fields.each do |name,options|
+          # rod_id is set during _store
+          object.update_field(name) unless name == "rod_id"
+        end
+        # set ids of objects referenced via singular associations
+        object.class.singular_associations.each do |name,options|
+          object.update_singular_association(name,object.send(name))
+        end
       end
       # set ids of objects referenced via plural associations
+      # TODO should be disabled, when there are no new elements
       object.class.plural_associations.each do |name,options|
         elements = object.send(name) || []
         if options[:polymorphic]
@@ -345,7 +405,7 @@ module Rod
       library = VERSION.split(".")
       return false if file[0] != library[0] || file[1] != library[1]
       if library[1].to_i.even?
-        return true
+        return file[2].to_i <= library[2].to_i
       else
         return file[2] == library[2]
       end
@@ -362,8 +422,141 @@ module Rod
     end
 
     # Special classes used by the database.
-    def self.special_classes
+    def special_classes
       [JoinElement, PolymorphicJoinElement, StringElement]
     end
+
+    def convert_options(options)
+      result = {}
+      case options
+      when true,false
+        result[:readonly] = options
+      when Hash
+        result = options
+        if options[:migrate]
+          result[:old_readonly] = options[:readonly]
+          result[:readonly] = false
+        end
+      else
+        raise RodException.new("Invalid options for open_database: #{options}!")
+      end
+      result[:readonly] = true if result[:readonly].nil?
+      result
+    end
+
+    # Generates the classes for the data using the metadata from database.yml
+    # +module_instance+ is the module in which the classes are generated.
+    # This allows for embedding them in a separate namespace and use the same model
+    # with different databases in the same time.
+    def generate_classes(module_instance)
+      special_names = special_classes.map{|k| k.name}
+      special_names << "Rod"
+      superclasses = {}
+      @metadata.reject{|k,o| special_names.include?(k)}.each do |k,o|
+        superclasses[k] = o[:superclass]
+      end
+      superclass_tree = {}
+      superclasses.each do |klass,superclass|
+        superclass_tree[klass] = []
+        current_superclass = superclass
+        loop do
+          break if current_superclass.nil?
+          superclass_tree[klass] << current_superclass
+          break if current_superclass == "Rod::Model"
+          current_superclass = superclasses[current_superclass]
+        end
+      end
+      superclasses.keys.sort do |klass1,klass2|
+        if superclass_tree[klass1].include?(klass2)
+          1
+        elsif superclass_tree[klass2].include?(klass1)
+          -1
+        else
+          klass1 <=> klass2
+        end
+      end.each do |klass_name|
+        metadata = @metadata[klass_name]
+        original_name = klass_name
+        if module_instance != Object
+          prefix = module_instance.name + "::"
+          if superclasses.keys.include?(metadata[:superclass])
+            metadata[:superclass] = prefix + metadata[:superclass]
+          end
+          [:fields,:has_one,:has_many].each do |property_type|
+            next if metadata[property_type].nil?
+            metadata[property_type].each do |property,options|
+              if superclasses.keys.include?(options[:options][:class_name])
+                metadata[property_type][property][:options][:class_name] =
+                  prefix + options[:options][:class_name]
+              end
+            end
+          end
+          # klass name
+          klass_name = prefix + klass_name
+          @metadata.delete(original_name)
+          @metadata[klass_name] = metadata
+        end
+        klass = Model.generate_class(klass_name,metadata)
+        klass.model_path = Model.struct_name_for(original_name)
+        @classes << klass
+        klass.database_class(self.class)
+      end
+    end
+
+    # During migration it creats the classes which are used to read
+    # the legacy data. It also changes the path for the
+    # actual classes not to conflict with paths of legacy data.
+    def create_legacy_classes
+      legacy_module = nil
+      begin
+        legacy_module = Object.const_get(LEGACY_MODULE)
+      rescue NameError
+        legacy_module = Module.new
+        Object.const_set(LEGACY_MODULE,legacy_module)
+      end
+      self.classes.each do |klass|
+        next if special_class?(klass)
+        klass.model_path = Model.struct_name_for(klass.name) + NEW_DATA_SUFFIX
+      end
+      generate_classes(legacy_module)
+    end
+
+    # Removes single file.
+    def remove_file(file_name)
+      if test(?f,file_name)
+        File.delete(file_name)
+        puts "Removing #{file_name}" if $ROD_DEBUG
+      end
+    end
+
+    # Remove all files matching the +pattern+.
+    # If +skip+ given, the file with the given name is not deleted.
+    def remove_files(pattern,skip=nil)
+      Dir.glob(pattern).each do |file_name|
+        remove_file(file_name) unless file_name == skip
+      end
+    end
+
+    # Removes all files which are similar (i.e. are generated
+    # by RubyInline for the same class) to +name+
+    # excluding the file with exactly the name given.
+    def remove_files_but(name)
+      remove_files(name.sub(INLINE_PATTERN_RE,"*"),name)
+    end
+
+    # Writes the metadata to the database.yml file.
+    def write_metadata
+      metadata = {}
+      rod_data = metadata["Rod"] = {}
+      rod_data[:version] = VERSION
+      rod_data[:created_at] = self.metadata["Rod"][:created_at] || Time.now
+      rod_data[:updated_at] = Time.now
+      self.classes.each do |klass|
+        metadata[klass.name] = klass.metadata(self)
+      end
+      File.open(@path + DATABASE_FILE,"w") do |out|
+        out.puts(YAML::dump(metadata))
+      end
+    end
   end
 end