mongoid-giza 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 150e88a193d198b7ee59e633753808457f01f9d0
+  data.tar.gz: dd2c1cc23492df7aa5e1c3e9c3fd6b13a578486a
+SHA512:
+  metadata.gz: 632f8929ab2a00803101594e1dc0e0bdd98164ef984797bb889eaaa4d90ebfef1250e7c94ad3f00f2eadab1313b4b36b49dd9d2969010b585df34a6b10f2bff1
+  data.tar.gz: 5eadf9661bc8697db39111932e5677ed1df5199269cef298fc2cf0788a95de779d1c8a63e3f783bb9877556ce9e3f9d553ecfd07f24ef6f8ef3602fcb5dc6d7e

data/.gitignore

@@ -0,0 +1,21 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+.ruby-gemset
+.ruby-version
+.rspec
+turbulence

data/.travis.yml

@@ -0,0 +1,6 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.0.0
+services:
+  - mongodb

data/Gemfile

@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+
+gem 'coveralls', require: false
+
+# Specify your gem's dependencies in mongoid-giza.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 YADEV Team
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,152 @@
+# Mongoid::Giza [![Build Status](https://travis-ci.org/yadevteam/mongoid-giza.png)](https://travis-ci.org/yadevteam/mongoid-giza) [![Code Climate](https://codeclimate.com/github/yadevteam/mongoid-giza.png)](https://codeclimate.com/github/yadevteam/mongoid-giza) [![Coverage Status](https://coveralls.io/repos/yadevteam/mongoid-giza/badge.png)](https://coveralls.io/r/yadevteam/mongoid-giza)
+
+Mongoid layer for the Sphinx fulltext search server that supports block fields and dynamic indexes
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem "mongoid-giza"
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install mongoid-giza
+
+## Usage
+
+:warning: **Before proceeding is extremely recommended to read the [Sphinx documentation](http://sphinxsearch.com/docs/current.html) if you are not yet familiar with it. Reading up to chapter 5 is enought to get you going.**
+
+### Setting up indexes on models
+
+Use a `sphinx_index` block to create a new index.
+
+The `sphinx_index` method may receive optional settings that will be set in this index's section or in its source section on the generated sphinx configuration file.
+These settings take precedence to the defaults defined in `giza.yml`.
+
+A model may have more than one index, but they need to have different names.
+If two or more indexes have the same name the last one to be defined is the one which will exist.
+
+An index name is the name of the class it's defined on unless overwritten by the `name` method inside the index definition block.
+
+Besides `name`, `field`, `attribute` and `criteria` are the methods avaible inside the index definition block.
+
+Both `field` and `attribute` take a name as first parameter that may match with a Mongoid field. In this case the value of the field will be used when indexing the objects.
+The `attribute` method may receive a second paramenter that defines the type of the attribute. If it is ommited, than the type of the Mongoid field will be used.
+
+At last, both methods may take an block with the object as parameter. The return of the block will be used as the value of the field or attribute when indexing.
+
+The `criteria` method receives a `Mongoid::Criteria` that will be used to select the objects that will be indexed.
+It's `Class.all` by default.
+
+**Example:** Creating a index on the person model
+
+```
+class Person
+  include Mongoid::Document
+  include Mongoid::Giza
+
+  field :name
+  field :age, type: Integer
+
+  sphinx_index(enable_star: 1) do
+    field :name
+    field :bio do |person|
+      "#{person.name.capitalize} was born #{person.age.years.days} ago"
+    end
+    attribute :age
+  end
+end
+```
+
+#### Dynamic Indexes
+
+Because of the schemaless nature of MongoDB, sometimes you may find problems mapping your mongo models to sphinx indexes.
+To circunvent this limitation Mongoid::Giza supports dynamic indexes.
+
+When you define a dynamic index, it will generate a regular index based on your definition for each object of the class.
+This allows the creation of different indexes for objects of the same model that have different dynamic fields.
+
+Although it's not necessary, dynamic indexes are better used together with a `criteria`,
+so it's possible to control which objects of the class will be indexed on each determined index.
+
+To create a dynamic index all that needs to be done is pass the object to the `sphin_index` block.
+
+**Example:** Creating a dynamic index on the person model.
+This dynamic index will generate one index for each job that is associated to a person.
+On each index only the people that have that job will be indexed.
+Finally each dynamic attribute of the job will be a field on its index.
+
+```
+class Job
+  include Mongoid::Document
+
+  field :name
+  # each job object has specific dynamic fields
+
+  has_many :people
+end
+
+class Person
+  include Mongoid::Document
+  include Mongoid::Giza
+
+  field :name
+  field :age, type: Integer
+
+  belongs_to :job
+
+  sphinx_index do |person|
+    name person.job.name
+    criteria Person.where(job: person.job)
+    person.job.attributes.except("name").each do |attr, val|
+      field attr.to_sym
+    end
+  end
+end
+```
+
+### Searching
+
+Use the `search` block on the class that have the indexes where the search should run.
+It returns a result array, where each position of the array is a [riddle result hash](http://rdoc.info/github/pat/riddle/Riddle/Client#query-instance_method), plus a key with the class name, that has the `Mongoid::Criteria` that selects the matching objects from the mongo database.
+
+Inside the `search` block use the `fulltext` method to perform a fulltext search.
+If multiple `fulltext` are called inside a `search` block, then each one will generate a separated query and will return a new position o the results array.
+
+To filter your search using the attributes defined on the index creation, use the `with` and `without` methods, that accept the name of the attribute and the value or range.
+
+To order the results, use the `order_by` method, that receives the *attribute* used for sorting and a Symbol, that can be either `:asc` or `:desc`.
+
+Every other [Riddle::Client](http://rdoc.info/github/pat/riddle/Riddle/Client) setter is avaible without the **=**, to maintain the DSL syntax consistent.
+
+**Example:** Searching on the person class
+
+```
+results = Person.search do
+  fulltext "john"
+  with :age 18..40
+  order_by :age :asc
+end
+
+results.first[:Person].each do |person|
+ puts "#{person.name} is #{person.age} years old"
+end
+```
+
+## TODO
+
+* Support delta indexing
+* Support RT indexes
+* Support distributed indexes
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler"
+Bundler.setup
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new("spec") do |spec|
+  spec.pattern = "spec/**/*_spec.rb"
+end
+
+task default: :spec

data/lib/mongoid/giza.rb

@@ -0,0 +1,160 @@
+require "docile"
+require "mongoid"
+require "riddle"
+require "mongoid/giza/configuration"
+require "mongoid/giza/dynamic_index"
+require "mongoid/giza/index"
+require "mongoid/giza/index/field"
+require "mongoid/giza/index/attribute"
+require "mongoid/giza/indexer"
+require "mongoid/giza/models/giza_id"
+require "mongoid/giza/railtie" if defined?(Rails)
+require "mongoid/giza/search"
+require "mongoid/giza/version"
+require "mongoid/giza/xml_pipe2"
+
+module Mongoid
+
+  # Module that should be included in a Mongoid::Document in order to
+  # index fields of documents of this class
+  #
+  # @example Creating a simple index with a full-text field (named fts) and an attribute (named attr)
+  #   class Person
+  #     include Mongoid::Document
+  #     include Mongoid::Giza
+  #
+  #     field :name
+  #     field :age, type: Integer
+  #
+  #     sphinx_index do
+  #       field :name
+  #       attribute :age
+  #     end
+  #   end
+  #
+  # @example Searching the previously defined index for people named John between 18 and 59 years old
+  #   results = Person.search do
+  #     fulltext "john"
+  #     with age: 18..59
+  #   end
+  #
+  #   results.first[:Person].first # => First object that matched
+  module Giza
+    extend ActiveSupport::Concern
+
+    included do
+      Mongoid::Giza::GizaID.create(id: name.to_sym)
+      @giza_configuration = Configuration.instance
+      @static_sphinx_indexes = {}
+      @generated_sphinx_indexes = {}
+      @dynamic_sphinx_indexes = []
+    end
+
+    # Retrives the sphinx compatible id of the object.
+    # If the id does not exists yet, it will be generated
+    #
+    # @return [Integer] the object's integer id generated by Giza
+    def giza_id
+      set(:giza_id, GizaID.next_id(self.class.name.to_sym)) if self[:giza_id].nil?
+      self[:giza_id]
+    end
+
+    module ClassMethods
+      attr_reader :static_sphinx_indexes, :generated_sphinx_indexes, :dynamic_sphinx_indexes
+
+      # Class method that defines a index relative to the current class' objects.
+      # If an argument is given in the block then a dynamic index will be created.
+      # Otherwise it will create a static index.
+      #
+      # @param settings [Hash] optional settings for the index and it's source
+      # @param block [Proc] a block that will be evaluated on an {Mongoid::Giza::Index}
+      def sphinx_index(settings = {}, &block)
+        if block_given?
+          if block.arity > 0
+            add_dynamic_sphinx_index(settings, block)
+          else
+            add_static_sphinx_index(settings, block)
+          end
+        end
+      end
+
+      # Adds an dynamic index to the class.
+      # Will also automatically generate the dynamic index for each object of the class
+      #
+      # @param settings [Hash] settings for the index and it's source
+      # @param block [Proc] a block that will be evaluated on an {Mongoid::Giza::Index}.
+      #   The block receives one argument that is the current object of the class for which the index will be generated
+      def add_dynamic_sphinx_index(settings, block)
+        dynamic_index = DynamicIndex.new(self, settings, block)
+        dynamic_sphinx_indexes << dynamic_index
+        process_dynamic_sphinx_index(dynamic_index)
+      end
+
+      # Adds an static index to the class
+      #
+      # @param settings [Hash] settings for the index and it's source
+      # @param block [Proc] a block that will be evaluated on an {Mongoid::Giza::Index}.
+      def add_static_sphinx_index(settings, block)
+        index = Index.new(self, settings)
+        Docile.dsl_eval(index, &block)
+        static_sphinx_indexes[index.name] = index
+        @giza_configuration.add_index(index)
+      end
+
+      # Generates the indexes from the dynamic index and
+      # registers them on the class and on the configuration
+      #
+      # @param dynamic_index [Mongoid::Giza::DynamicIndex] the dynamic index which will generate the static indexes from
+      def process_dynamic_sphinx_index(dynamic_index)
+        generated = dynamic_index.generate!
+        generated_sphinx_indexes.merge!(generated)
+        generated.each { |name, index| @giza_configuration.add_index(index, true) }
+      end
+
+      # Class method that implements a search DSL using a {Mongoid::Giza::Search} object.
+      # The search will run on indexes defined on the class unless it's overwritten using {Mongoid::Giza::Search#indexes=}
+      #
+      # @param block [Proc] a block that will be evaluated on a {Mongoid::Giza::Search}
+      #
+      # @return [Array] an Array with Riddle result hashes containing an additional key with the name of the class.
+      #   The value of this aditional key is a Mongoid::Criteria that return the actual objects of the match
+      def search(&block)
+        search = Mongoid::Giza::Search.new(@giza_configuration.searchd.address, @giza_configuration.searchd.port, *sphinx_indexes_names)
+        Docile.dsl_eval(search, &block)
+        results = search.run
+        results.each { |result| result[name.to_sym] = self.in(giza_id: result[:matches].map { |match| match[:doc] }) }
+      end
+
+      # Regenerates all dynamic indexes of the class
+      def regenerate_dynamic_sphinx_indexes
+        generated_sphinx_indexes.clear
+        dynamic_sphinx_indexes.each { |dynamic_index| process_dynamic_sphinx_index(dynamic_index) }
+      end
+
+      # Execute the indexing routines of the indexes defined on the class.
+      # This means (re)create the sphinx configuration file and then execute the indexer program on it.
+      # If no index names are supplied than all indexes defined on the class will be indexed.
+      # If none of the index names supplied are on this class then nothing is indexed
+      #
+      # @param names [Array] a list of index names of this class that will be indexed
+      def sphinx_indexer!(*names)
+        indexes_names = names.length > 0 ? sphinx_indexes_names.select { |name| names.include? name } : sphinx_indexes_names
+        Mongoid::Giza::Indexer.instance.index!(*indexes_names) if indexes_names.length > 0
+      end
+
+      # Retrieves all the sphinx indexes defined on this class, static and dynamic
+      #
+      # @return [Array] an Array of indexes from the current class {Mongoid::Giza::Index}
+      def sphinx_indexes
+        static_sphinx_indexes.merge(generated_sphinx_indexes)
+      end
+
+      # Retrieves all the names of sphinx indexes defined on this class, static and dynamic
+      #
+      # @return [Array] an Array of names of indexes from the current class {Mongoid::Giza::Index}
+      def sphinx_indexes_names
+        static_sphinx_indexes.merge(generated_sphinx_indexes).keys
+      end
+    end
+  end
+end

data/lib/mongoid/giza/configuration.rb

@@ -0,0 +1,136 @@
+require "erb"
+require "ostruct"
+require "yaml"
+
+module Mongoid
+  module Giza
+
+    # Holds the configuration of the module
+    class Configuration < Riddle::Configuration
+      include Singleton
+
+      attr_reader :source, :index, :file
+
+      # Creates the configuration instance
+      def initialize
+        super
+        @source = Riddle::Configuration::XMLSource.new(:source, :xmlpipe2)
+        @index = Riddle::Configuration::Index.new(:index, @source)
+        @file = OpenStruct.new(output_path: "./sphinx.conf")
+        @static_indexes = {}
+        @generated_indexes = {}
+      end
+
+      # Loads a YAML file with settings defined.
+      # Settings that are not recognized are ignored
+      #
+      # @param path [String] path to the YAML file which contains the settings defined
+      # @param env [String] environment whoose settings will be loaded
+      def load(path, env)
+        YAML.load(File.open(path).read)[env].each do |section_name, settings|
+          section = instance_variable_get("@#{section_name}")
+          if !section.nil?
+            settings.each do |setting, value|
+              method = "#{setting}="
+              section.send(method, value) if section.respond_to?(method)
+            end
+          end
+        end
+      end
+
+      # Adds an index to the sphinx configuration,
+      # so this index can be rendered on the configuration file
+      #
+      # @param index [Mongoid::Giza::Index] the index that will be added to the configuration
+      # @param generated [TrueClass, FalseClass] determines if this index was generated from a {Mongoid::Giza::DynamicIndex}
+      def add_index(index, generated = false)
+        riddle_index = create_index(index)
+        if generated
+          position = register_index(riddle_index, @generated_indexes)
+        else
+          position = register_index(riddle_index, @static_indexes)
+        end
+        indices[position] = riddle_index
+      end
+
+      # Creates a new Riddle::Index based on the given {Mongoid::Giza::Index}
+      #
+      # @param index [Mongoid::Giza::Index] the index to generate the configuration from
+      #
+      # @return [Riddle::Configuration::Index] the created riddle index
+      def create_index(index)
+        source = Riddle::Configuration::XMLSource.new(index.name, :xmlpipe2)
+        riddle_index = Riddle::Configuration::Index.new(index.name, source)
+        apply_default_settings(@index, riddle_index, index)
+        apply_default_settings(@source, source, index)
+        apply_user_settings(index, riddle_index)
+        apply_user_settings(index, source)
+        riddle_index.path = File.join(riddle_index.path, index.name.to_s)
+        riddle_index.charset_type = "utf-8"
+        riddle_index
+      end
+
+      # Adds the riddle index to it's respective collection
+      #
+      # @param index [Riddle::Configuration::Index] the index that will be registrated
+      # @param indexes [Hash] the collection which will hold this index
+      #
+      # @return [Integer] the position where this index should be inserted on the configuration indices array
+      def register_index(index, indexes)
+        position = indexes.has_key?(index.name) ?  indices.index(indexes[index.name]) : indices.length
+        indexes[index.name] = index
+        position
+      end
+
+      # Applies the settings defined on an object loaded from the configuration to a Riddle::Configuration::Index or Riddle::Configuration::XMLSource instance.
+      # Used internally by {#add_index} so you should never need to call it directly
+      #
+      # @param default [Riddle::Configuration::Index, Riddle::Configuration::XMLSource] the object that holds the global settings values
+      # @param section [Riddle::Configuration::Index, Riddle::Configuration::XMLSource] the object that settings are being set
+      def apply_default_settings(default, section, index)
+        default.class.settings.each do |setting|
+          method = "#{setting}="
+          value = interpolate_string(default.send("#{setting}"), index)
+          section.send(method, value) if !value.nil? and section.respond_to?(method)
+        end
+      end
+
+      # Applies the settings defined on a {Mongoid::Giza::Index} to the Riddle::Configuration::Index or Riddle::Configuration::XMLSource.
+      # Used internally by {#add_index} so you should never need to call it directly
+      #
+      # @param index [Mongoid::Giza::Index] the index where the settings were defined
+      # @param section [Riddle::Configuration::Index, Riddle::Configuration::XMLSource] where the settings will be applied
+      def apply_user_settings(index, section)
+        index.settings.each do |setting, value|
+          method = "#{setting}="
+          section.send(method, interpolate_string(value, index)) if section.respond_to?(method)
+        end
+      end
+
+      # Interpolates a value if it's a String using ERB.
+      # Useful for defining dynamic settings.
+      # The ERB template may reference to the current {Mongoid::Giza::Index} and it's methods
+      #
+      # @param value [String] the ERB template that will be interpolated
+      # @param index [Mongoid::Giza::Index] the index that will be accessible from the template
+      #
+      # @return [Object] if value was a String and contains ERB syntax than it will beinterpolated and returned.
+      #   Otherwise it will return the original value
+      def interpolate_string(value, index)
+        namespace = OpenStruct.new(index: index)
+        value.is_a?(String) ? ERB.new(value).result(namespace.instance_eval { binding }) : value
+      end
+
+      # Renders the configuration to the output_path
+      def render
+        File.open(@file.output_path, "w") { |file| file.write(super) }
+      end
+
+      # Removes all Riddle::Index from the indices Array that where created from a generated {Mongoid::Giza::Index}
+      def clear_generated_indexes
+        @generated_indexes.each { |name, index| indices.delete(index) }
+        @generated_indexes =  {}
+      end
+    end
+  end
+end