xapian_db 1.2.5.1 → 1.3

data/CHANGELOG.md

@@ -1,4 +1,8 @@
-##1.2.5.1 (January 7th, 2013)
+##1.3 (February 19th, 2013)
+
+Breaking Changes:
+  - changed the internal serialization format for attributes; the new default is string, new codecs were added. YAML is not supported
+    anymore, although you could implement a YAML-Codec on your own if you need it (see examples/custom_serialization.rb)
 
 Changes:
 

data/README.rdoc

@@ -4,7 +4,10 @@
 
 == Important Information
 
-If you upgrade from an earlier version of xapian_db to 1.1, you MUST rebuild your entire index (XapianDb.rebuild_xapian_index)!
+Version 1.3 does not support YAML serialization for attributes anymore. If you don't apply a type information for an attribute (like attribute :birth, as: :date) in your blueprint,
+the attribute gets stored as a string. The new preferred type for complex attributes is JSON. Why the change? We were experiencing a unacceptable performance hit when we switched from Syck to Psych. The new philosophy for blueprints is therefore explicit type information. lib/type_codec.rb contains the most common codecs and of course, you can add your own (see examples/custom_serialization.rb).
+
+<b>Please note: You may want to fine tune your blueprints and you MUST rebuild your xapian index when switching to version 1.3</b>
 
 == What's in the box?
 
@@ -139,13 +142,14 @@ You may add a filter expression to exclude objects from the index. This is handy
       blueprint.ignore_if {active == false}
     end
 
-You can add a type information to an attribute. As of now the special types :string, :date, :date_time and :number are supported (and required for range queries):
+You can add a type information to an attribute (default format is string). As of now the special types :string, :date, :date_time and :number are supported (and required for range queries):
 
     XapianDb::DocumentBlueprint.setup(:Person) do |blueprint|
       blueprint.attribute :age,           :as => :number
       blueprint.attribute :date_of_birth, :as => :date
       blueprint.attribute :name,          :as => :string
       blueprint.attribute :updated_at,    :as => :date_time
+      blueprint.attribute :address,       :as => :json
     end
 
 If you don't need field searches for an attribute, turn off the prefixed option (makes your index smaller and more efficient):
@@ -165,7 +169,7 @@ one more class that is not stored in the database, but you want it to be indexed
 If you use associations in your blueprints, it might be a good idea to specify a base query to speed up rebuild_xapian_index calls (avoiding 1+n queries):
 
   XapianDb::DocumentBlueprint.setup(:Person) do |blueprint|
-    blueprint.index :addresses
+    blueprint.index :addresses, as: :json
     blueprint.base_query { |p| p.includes(:addresses) }
   end
 
@@ -374,7 +378,7 @@ you can use the auto_indexing_disabled method with a block and rebuild the whole
 
 == Add your own serializers for special objects
 
-XapianDb serializes objects to xapian documents using YAML by default. This way, type information is preserved und you get back what you put into a xapian document, not just a string.
+XapianDb serializes objects to xapian documents as strings by default.
 
 However, dates need special handling to support date range queries. To support date range queries and allow the addition of other custom data types in the future, XapianDb uses a simple, extensible mechanism to serialize / deserialize your objects. An example on how to extend this mechanism is provided in examples/custom_serialization.rb.
 

data/lib/generators/templates/beanstalk_worker

@@ -5,7 +5,7 @@ require "bundler/setup"
 require 'rails'
 require 'daemons'
 require "xapian_db"
-require 'yaml'
+require 'json'
 require 'rails/railtie' # execute xapian_db railtie
 
 # daemon script to manage the beanstalk worker daemon
@@ -33,7 +33,7 @@ module XapianDb
       loop do
         job = beanstalk.reserve
         begin
-          params = YAML::load job.body
+          params = JSON.parse(job.body).symbolize_keys!
           Rails.logger.info "XapianDb beanstalk worker: executing task #{params}"
           Rails.logger.flush
           task = params.delete :task
@@ -74,4 +74,4 @@ Daemons.run_proc('beanstalk_worker', daemon_options) do
   Dir.chdir dir
   require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
   XapianDb::BeanstalkWorker.new.process_requests
-end
+end

data/lib/type_codec.rb

@@ -24,31 +24,27 @@ module XapianDb
       end
     end
 
-    class GenericCodec
+    class JsonCodec
 
-      # Encode an object to its yaml representation
+      # Encode an object to its json representation
       # @param [Object] object an object to encode
-      # @return [String] the yaml string
+      # @return [String] the json string
       def self.encode(object)
         begin
-          if object.respond_to?(:attributes)
-            object.attributes.to_yaml
-          else
-            object.to_yaml
-          end
+          object.to_json
         rescue NoMethodError
-          raise ArgumentError.new "#{object} does not support yaml serialization"
+          raise ArgumentError.new "#{object} does not support json serialization"
         end
       end
 
-      # Decode an object from a yaml string
-      # @param [String] yaml_string a yaml string representing the object
-      # @return [Object] the parsed object
-      def self.decode(yaml_string)
+      # Decode an object from a json string
+      # @param [String] json_string a json string representing the object
+      # @return [Hash] a ruby hash
+      def self.decode(json_string)
         begin
-          YAML::load yaml_string
+          JSON.parse json_string
         rescue TypeError
-          raise ArgumentError.new "'#{yaml_string}' cannot be loaded by YAML"
+          raise ArgumentError.new "'#{json_string}' cannot be parsed"
         end
       end
     end
@@ -66,7 +62,24 @@ module XapianDb
       # @param [String] string a string
       # @return [String] the string
       def self.decode(string)
-        string
+        string.force_encoding("UTF-8")
+      end
+    end
+
+    class BooleanCodec
+
+      # Encode a boolean value to a string
+      # @param [Object] value a value to encode
+      # @return [String] the string
+      def self.encode(value)
+        value.to_s
+      end
+
+      # Decode a string representing a boolean
+      # @param [String] string a string
+      # @return [Boolean] the boolean value
+      def self.decode(string)
+        string == "true"
       end
     end
 
@@ -127,11 +140,11 @@ module XapianDb
     class NumberCodec
 
       # Encode a number to a sortable string
-      # @param [Integer, BigDecimal, Float] number a number object to encode
+      # @param [Integer, BigDecimal, Bignum, Float] number a number object to encode
       # @return [String] the encoded number
       def self.encode(number)
         case number.class.name
-          when "Fixnum", "Float"
+          when "Fixnum", "Float", "Bignum"
             Xapian::sortable_serialise number
           when "BigDecimal"
             Xapian::sortable_serialise number.to_f
@@ -152,5 +165,33 @@ module XapianDb
       end
     end
 
+    class IntegerCodec
+
+      # Encode an integer to a sortable string
+      # @param [Integer] integer an integer to encode
+      # @return [String] the encoded integer
+      def self.encode(number)
+        return nil if number.nil?
+        case number.class.name
+          when "Fixnum"
+            Xapian::sortable_serialise number
+          else
+            raise ArgumentError.new "#{number} was expected to be an integer"
+        end
+      end
+
+      # Decode a string to an integer
+      # @param [String] integer_as_string a string representing an integer
+      # @return [Integer] the decoded integer
+      def self.decode(encoded_integer)
+        begin
+          return nil if encoded_integer.nil? || encoded_integer.to_s.strip == ""
+          Xapian::sortable_unserialise(encoded_integer).to_i
+        rescue TypeError
+          raise ArgumentError.new "#{encoded_integer} cannot be unserialized"
+        end
+      end
+    end
+
   end
 end

data/lib/xapian_db/database.rb

@@ -137,7 +137,7 @@ module XapianDb
       enquiry.collapse_key = value_number
       facets = {}
       enquiry.mset(0, size).matches.each do |match|
-        facet_value = YAML::load match.document.value(value_number)
+        facet_value = match.document.value(value_number)
         # We must add 1 to the collapse_count since collapse_count means
         # "how many other matches are there?"
         facets[facet_value] = match.collapse_count + 1

data/lib/xapian_db/document_blueprint.rb

@@ -52,6 +52,11 @@ module XapianDb
         @attributes = @blueprints.values.map { |blueprint| blueprint.attribute_names}.flatten.compact.uniq.sort || []
       end
 
+      # reset the blueprint setup
+      def reset
+        @blueprints = {}
+      end
+
       # is a blueprint configured for the given name?
       # @return [Boolean]
       def configured?(name)
@@ -311,7 +316,7 @@ module XapianDb
     def attribute(name, options={}, &block)
       raise ArgumentError.new("You cannot use #{name} as an attribute name since it is a reserved method name of Xapian::Document") if reserved_method_name?(name)
       do_not_index    = options.delete(:index) == false
-      @type_map[name] = (options.delete(:as) || :generic)
+      @type_map[name] = (options.delete(:as) || :string)
 
       if block_given?
         @attributes_hash[name] = {:block => block}.merge(options)
@@ -328,7 +333,7 @@ module XapianDb
       attributes.each do |attr|
         raise ArgumentError.new("You cannot use #{attr} as an attribute name since it is a reserved method name of Xapian::Document") if reserved_method_name?(attr)
         @attributes_hash[attr] = {}
-        @type_map[attr] = :generic
+        @type_map[attr] = :string
         self.index attr
       end
     end

data/lib/xapian_db/index_writers/beanstalk_writer.rb

@@ -7,6 +7,7 @@
 # @author Gernot Kogler
 
 require 'beanstalk-client'
+require 'json'
 
 module XapianDb
   module IndexWriters
@@ -18,19 +19,19 @@ module XapianDb
         # Update an object in the index
         # @param [Object] obj An instance of a class with a blueprint configuration
         def index(obj, commit=true)
-          beanstalk.put( {:task => "index_task", :class => obj.class.name, :id => obj.id }.to_yaml )
+          beanstalk.put( {:task => "index_task", :class => obj.class.name, :id => obj.id }.to_json )
         end
 
         # Remove an object from the index
         # @param [String] xapian_id The document id
         def delete_doc_with(xapian_id, commit=true)
-          beanstalk.put( { :task => "delete_doc_task", :xapian_id => xapian_id }.to_yaml )
+          beanstalk.put( { :task => "delete_doc_task", :xapian_id => xapian_id }.to_json )
         end
 
         # Reindex all objects of a given class
         # @param [Class] klass The class to reindex
         def reindex_class(klass, options={})
-          beanstalk.put( { :task => "reindex_class_task", :class => klass.name }.to_yaml )
+          beanstalk.put( { :task => "reindex_class_task", :class => klass.name }.to_json )
         end
 
         def beanstalk

data/lib/xapian_db.rb

@@ -7,7 +7,7 @@
 # @author Gernot Kogler
 
 require 'xapian'
-require 'yaml'
+require 'json'
 
 do_not_require = %w(update_stopwords railtie base_adapter generic_adapter active_record_adapter datamapper_adapter
                     beanstalk_writer resque_writer utilities install_generator datamapper)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: xapian_db
 version: !ruby/object:Gem::Version
-  version: 1.2.5.1
+  version: '1.3'
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-01-07 00:00:00.000000000 Z
+date: 2013-02-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: daemons
@@ -244,6 +244,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -2176648600874761364
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -252,7 +255,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: 1.3.6
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: Ruby library to use a Xapian db as a key/value store with high performance