dm-sphinx-adapter 0.4 → 0.5

data/History.txt

@@ -1,3 +1,15 @@
+=== 0.5 / 2008-12-01
+
+  * Moved sphinx extended query string generator into a class of its own.
+  * Improved generated extended query syntax and added tests.
+  * Support for sphinx "" phrase search operator (dm conditions as array).
+  * Support for sphinx | OR operator (dm conditions using {:field.in => %w{}}).
+
+=== 0.4 / 2008-11-21
+
+  * Fixed broken dm-is-searchable support.
+  * Bumped version because the read_one/read_many result structure had to change to support dm-is-searchable.
+
 === 0.3 / 2008-11-18
 
   * Removed calls to indexer on create/update. See README.txt

data/Manifest.txt

@@ -5,18 +5,29 @@ README.txt
 Rakefile
 dm-sphinx-adapter.gemspec
 lib/dm-sphinx-adapter.rb
-lib/dm-sphinx-adapter/sphinx_adapter.rb
-lib/dm-sphinx-adapter/sphinx_attribute.rb
-lib/dm-sphinx-adapter/sphinx_client.rb
-lib/dm-sphinx-adapter/sphinx_config.rb
-lib/dm-sphinx-adapter/sphinx_index.rb
-lib/dm-sphinx-adapter/sphinx_resource.rb
-test/data/sphinx.conf
-test/fixtures/item.rb
-test/fixtures/item.sql
-test/fixtures/item_resource_explicit.rb
-test/fixtures/item_resource_only.rb
-test/helper.rb
+lib/dm-sphinx-adapter/adapter.rb
+lib/dm-sphinx-adapter/attribute.rb
+lib/dm-sphinx-adapter/client.rb
+lib/dm-sphinx-adapter/config.rb
+lib/dm-sphinx-adapter/config_parser.rb
+lib/dm-sphinx-adapter/index.rb
+lib/dm-sphinx-adapter/query.rb
+lib/dm-sphinx-adapter/resource.rb
+test/files/dm_sphinx_adapter_test.sql
+test/files/resource_explicit.rb
+test/files/resource_resource.rb
+test/files/resource_searchable.rb
+test/files/resource_storage_name.rb
+test/files/resource_vanilla.rb
+test/files/sphinx.conf
+test/test_adapter.rb
+test/test_adapter_explicit.rb
+test/test_adapter_resource.rb
+test/test_adapter_searchable.rb
+test/test_adapter_vanilla.rb
 test/test_client.rb
 test/test_config.rb
-test/test_search.rb
+test/test_config_parser.rb
+test/test_query.rb
+test/test_type_attribute.rb
+test/test_type_index.rb

data/README.txt

@@ -1,23 +1,35 @@
 = DataMapper Sphinx Adapter
 
-A Sphinx DataMapper adapter.
+* http://dm-sphinx.rubyforge.org
+* http://rubyforge.org/projects/dm-sphinx
+* http://github.com/shanna/dm-sphinx-adapter/tree/master
+
+== Description
+
+A DataMapper Sphinx adapter.
+
+== Dependencies
+
+* dm-core           ~> 0.9.7
+* riddle            ~> 0.9
+* daemon_controller ~> 0.2   (optional)
+* dm-is-searchable  ~> 0.9.7 (optional)
+
+I'd recommend using the dm-more plugin dm-is-searchable instead of fetching the document id's yourself.
+
+== Install
+
+* Via git: git clone git://github.com/shanna/iso-country-codes.git
+* Via gem: gem install shanna-dm-sphinx-adapter -s http://gems.github.com
 
 == Synopsis
 
 DataMapper uses URIs or a connection has to connect to your data-stores. In this case the sphinx search daemon
 <tt>searchd</tt>.
 
-On its own this adapter will only return an array of document IDs when queried. The dm-more source (not the gem)
+On its own this adapter will only return an array of document hashes when queried. The DataMapper library dm-more
 however provides dm-is-searchable, a common interface to search one adapter and load documents from another. My
-suggestion is to use this adapter in tandem with dm-is-searchable.
-
-The dm-is-searchable plugin is part of dm-more though unfortunately isn't built and bundled with dm-more gem.
-You'll need to checkout the dm-more source with Git from git://github.com/sam/dm-more.git and build/install the
-gem yourself.
-
-  git clone git://github.com/sam/dm-more.git
-  cd dm-more/dm-is-searchable
-  sudo rake install_gem
+preference is to use this adapter in tandem with dm-is-searchable.
 
 Like all DataMapper adapters you can connect with a Hash or URI.
 
@@ -59,7 +71,19 @@ Alternatively supply a Hash:
   ids   = docs.map{|doc| doc[:id]}
   items = Item.all(:id => ids) # Search :default for all the document id's returned by sphinx.
 
-=== DataMapper and Is Searchable
+=== DataMapper and IsSearchable
+
+IsSearchable is a DataMapper plugin that provides a common search interface when searching from one adapter and reading
+documents from another.
+
+IsSearchable will read resources from your +:default+ repository on behalf of a search adapter such as
++dm-sphinx-adapter+ and +dm-ferret-adapter+. This saves some of the grunt work (as shown in the previous example) by
+mapping the resulting document id's from a search with your +:search+ adapter into a suitable #first or #all query for
+your +:default+ repository.
+
+IsSearchable adds a single class method to your resource. The first argument is a +Hash+ of DataMapper::Query
+conditions to pass to your search adapter (in this case +dm-sphinx-adapter+). An optional second Hash of
+DataMapper::Query conditions can also be passed and will be appended to the query on your +:default+ database.
 
   require 'rubygems'
   require 'dm-core'
@@ -82,7 +106,7 @@ Alternatively supply a Hash:
   # Fire up your sphinx search daemon and start searching.
   items = Item.search(:name => 'barney') # Search 'items' index for '@name barney'
 
-=== Merb, DataMapper and Is Searchable
+=== Merb, DataMapper and IsSearchable
 
   # config/init.rb
   dependency 'dm-is-searchable'
@@ -110,7 +134,7 @@ Alternatively supply a Hash:
   # Fire up your sphinx search daemon and start searching.
   Item.search(:name => 'barney') # Search 'items' index for '@name barney'
 
-=== DataMapper::SphinxResource
+=== DataMapper, IsSearchable and DataMapper::SphinxResource
 
 For finer grained control you can include DataMapper::SphinxResource. For instance you can search one or more indexes
 and sort, include or exclude by attributes defined in your sphinx configuration:
@@ -135,7 +159,7 @@ and sort, include or exclude by attributes defined in your sphinx configuration:
   # Search 'items, items_delta' index for '@name barney' updated in the last 30 minutes.
   Item.search(:name => 'barney', :updated => (Time.now - 1800 .. Time.now))
 
-== Sphinx Configuration.
+== Sphinx Configuration
 
 Though you don't have to supply the sphinx configuration file to dm-sphinx-adapter I'd recommend doing it anyway.
 It's more DRY since all searchd/indexer options can be read straight from the configuration.
@@ -155,7 +179,7 @@ or whatever in production. Use the Hash form of DataMapper#setup and supply the
 daemon_controller will start searchd on demand.
 
 It is already strongly encouraged but you will need to specify the path to your sphinx configuration file in order for
-searchd to run. See Sphinx Configuration, DataMapper::SphinxManagedClient.
+searchd to run. See Sphinx Configuration, DataMapper::Adapters::Sphinx::ManagedClient.
 
 The daemon_controller library can be found only on github, not rubyforge.
 See http://github.com/FooBarWidget/daemon_controller/tree/master
@@ -173,20 +197,12 @@ an endless loop, with a few seconds of sleep in between to allow searchd some he
 
 == Todo
 
-* Tests. Clearly I only test what I see as important or broken which drives TDD people crazy sometimes :)
 * Loads of documentation. Most of it is unchecked YARD at the moment.
-* Add DataMapper::SphinxClient#attribute_set to allow attribute modification on one or more indexes. It's the only
-  thing missing if you understand the pitfalls and still want to add thinking-sphinx like delta indexing to your
-  resource.
-
-== Dependencies
-
-dm-core and riddle are technically the only requirements though I'd recommend using the dm-more plugin dm-is-searchable
-instead of fetching the document id's yourself.
-
-Unfortunately dm-is-searchable isn't installed even when you build the dm-more gem from github master. You'll need to
-build and install the gem yourself from source.
+* Add DataMapper::Adapters::Sphinx::Client#attribute_set to allow attribute modification on one or more indexes. It's
+  the only thing missing if you understand the pitfalls and still want to add thinking-sphinx like delta indexing to
+  your resource.
 
 == Contributing
 
 Go nuts. Just send me a pull request (github or otherwise) when you are happy with your code.
+

data/Rakefile

@@ -3,7 +3,7 @@
 require 'rubygems'
 require 'hoe'
 
-Hoe.new('dm-sphinx-adapter', '0.4') do |p|
+Hoe.new('dm-sphinx-adapter', '0.5') do |p|
   p.developer('Shane Hanna', 'shane.hanna@gmail.com')
   p.extra_deps = [
     ['dm-core', '~> 0.9.7'],

data/dm-sphinx-adapter.gemspec

@@ -2,23 +2,23 @@
 
 Gem::Specification.new do |s|
   s.name = %q{dm-sphinx-adapter}
-  s.version = "0.4"
+  s.version = "0.5"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Shane Hanna"]
-  s.date = %q{2008-11-20}
-  s.description = %q{}
+  s.date = %q{2008-12-01}
+  s.description = %q{A DataMapper Sphinx adapter.}
   s.email = ["shane.hanna@gmail.com"]
   s.extra_rdoc_files = ["History.txt", "LICENCE.txt", "Manifest.txt", "README.txt"]
-  s.files = ["History.txt", "LICENCE.txt", "Manifest.txt", "README.txt", "Rakefile", "dm-sphinx-adapter.gemspec", "lib/dm-sphinx-adapter.rb", "lib/dm-sphinx-adapter/sphinx_adapter.rb", "lib/dm-sphinx-adapter/sphinx_attribute.rb", "lib/dm-sphinx-adapter/sphinx_client.rb", "lib/dm-sphinx-adapter/sphinx_config.rb", "lib/dm-sphinx-adapter/sphinx_index.rb", "lib/dm-sphinx-adapter/sphinx_resource.rb", "test/data/sphinx.conf", "test/fixtures/item.rb", "test/fixtures/item.sql", "test/fixtures/item_resource_explicit.rb", "test/fixtures/item_resource_only.rb", "test/helper.rb", "test/test_client.rb", "test/test_config.rb", "test/test_search.rb"]
+  s.files = ["History.txt", "LICENCE.txt", "Manifest.txt", "README.txt", "Rakefile", "dm-sphinx-adapter.gemspec", "lib/dm-sphinx-adapter.rb", "lib/dm-sphinx-adapter/adapter.rb", "lib/dm-sphinx-adapter/attribute.rb", "lib/dm-sphinx-adapter/client.rb", "lib/dm-sphinx-adapter/config.rb", "lib/dm-sphinx-adapter/config_parser.rb", "lib/dm-sphinx-adapter/index.rb", "lib/dm-sphinx-adapter/query.rb", "lib/dm-sphinx-adapter/resource.rb", "test/files/dm_sphinx_adapter_test.sql", "test/files/resource_explicit.rb", "test/files/resource_resource.rb", "test/files/resource_searchable.rb", "test/files/resource_storage_name.rb", "test/files/resource_vanilla.rb", "test/files/sphinx.conf", "test/test_adapter.rb", "test/test_adapter_explicit.rb", "test/test_adapter_resource.rb", "test/test_adapter_searchable.rb", "test/test_adapter_vanilla.rb", "test/test_client.rb", "test/test_config.rb", "test/test_config_parser.rb", "test/test_query.rb", "test/test_type_attribute.rb", "test/test_type_index.rb"]
   s.has_rdoc = true
-  s.homepage = %q{A Sphinx DataMapper adapter.}
+  s.homepage = %q{http://rubyforge.org/projects/dm-sphinx/}
   s.rdoc_options = ["--main", "README.txt"]
   s.require_paths = ["lib"]
   s.rubyforge_project = %q{dm-sphinx-adapter}
   s.rubygems_version = %q{1.3.0}
-  s.summary = %q{}
-  s.test_files = ["test/test_client.rb", "test/test_config.rb", "test/test_search.rb"]
+  s.summary = %q{A DataMapper Sphinx adapter.}
+  s.test_files = ["test/test_adapter.rb", "test/test_adapter_explicit.rb", "test/test_adapter_resource.rb", "test/test_adapter_searchable.rb", "test/test_adapter_vanilla.rb", "test/test_client.rb", "test/test_config.rb", "test/test_config_parser.rb", "test/test_query.rb", "test/test_type_attribute.rb", "test/test_type_index.rb"]
 
   if s.respond_to? :specification_version then
     current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION

data/lib/dm-sphinx-adapter.rb

@@ -8,10 +8,12 @@ require 'dm-core'
 # TODO: I think I might move everything to DataMapper::Sphinx::* and ignore the default naming convention.
 require 'pathname'
 dir = Pathname(__FILE__).dirname.expand_path / 'dm-sphinx-adapter'
-require dir / 'sphinx_config'
-require dir / 'sphinx_client'
-require dir / 'sphinx_adapter'
-require dir / 'sphinx_index'
-require dir / 'sphinx_attribute'
-require dir / 'sphinx_resource'
+require dir / 'config'
+require dir / 'config_parser'
+require dir / 'client'
+require dir / 'query'
+require dir / 'adapter'
+require dir / 'index'
+require dir / 'attribute'
+require dir / 'resource'
 

data/lib/dm-sphinx-adapter/adapter.rb

@@ -0,0 +1,218 @@
+require 'benchmark'
+
+module DataMapper
+  module Adapters
+    module Sphinx
+      # == Synopsis
+      #
+      # DataMapper uses URIs or a connection has to connect to your data-stores. In this case the sphinx search daemon
+      # <tt>searchd</tt>.
+      #
+      # On its own this adapter will only return an array of document hashes when queried. The DataMapper library dm-more
+      # however provides dm-is-searchable, a common interface to search one adapter and load documents from another. My
+      # preference is to use this adapter in tandem with dm-is-searchable.
+      #
+      # Like all DataMapper adapters you can connect with a Hash or URI.
+      #
+      # A URI:
+      #   DataMapper.setup(:search, 'sphinx://localhost')
+      #
+      # The breakdown is:
+      #   "#{adapter}://#{host}:#{port}/#{config}"
+      #   - adapter Must be :sphinx
+      #   - host    Hostname (default: localhost)
+      #   - port    Optional port number (default: 3312)
+      #   - config  Optional but recommended path to sphinx config file.
+      #
+      # Alternatively supply a Hash:
+      #   DataMapper.setup(:search, {
+      #     :adapter  => 'sphinx',       # required
+      #     :config   => './sphinx.conf' # optional. Recommended though.
+      #     :host     => 'localhost',    # optional. Default: localhost
+      #     :port     => 3312            # optional. Default: 3312
+      #     :managed  => true            # optional. Self managed searchd server using daemon_controller.
+      #   })
+      class Adapter < AbstractAdapter
+
+        # ==== See
+        # * DataMapper::Adapters::Sphinx::Config
+        # * DataMapper::Adapters::Sphinx::Client
+        #
+        # ==== Parameters
+        # uri_or_options<URI, DataObject::URI, Addressable::URI, String, Hash, Pathname>::
+        #   DataMapper uri or options hash.
+        def initialize(name, uri_or_options)
+          super
+
+          managed = !!(uri_or_options.kind_of?(Hash) && uri_or_options[:managed])
+          @client  = managed ? ManagedClient.new(uri_or_options) : Client.new(uri_or_options)
+        end
+
+        # Interaction with searchd and indexer.
+        #
+        # ==== See
+        # * DataMapper::Adapters::Sphinx::Client
+        # * DataMapper::Adapters::Sphinx::ManagedClient
+        #
+        # ==== Returns
+        # DataMapper::Adapters::Sphinx::Client:: The client.
+        attr_reader :client
+
+        def create(resources) #:nodoc:
+          true
+        end
+
+        def delete(query) #:nodoc:
+          true
+        end
+
+        # Query your Sphinx repository and return all matching documents.
+        #
+        # ==== Notes
+        #
+        # These methods are public but normally called indirectly through DataMapper::Resource#get,
+        # DataMapper::Resource#first or DataMapper::Resource#all.
+        #
+        # ==== Parameters
+        # query<DataMapper::Query>:: The query object.
+        #
+        # ==== Returns
+        # Array<Hash>:: An array of document hashes. <tt>[{:id => 1}, {:id => 2}]</tt>
+        # Array<>::     An empty array if no documents match.
+        def read_many(query)
+          read(query)
+        end
+
+        # Query your Sphinx repository and return the first document matched.
+        #
+        # ==== Notes
+        #
+        # These methods are public but normally called indirectly through DataMapper::Resource#get,
+        # DataMapper::Resource#first or DataMapper::Resource#all.
+        #
+        # ==== Parameters
+        # query<DataMapper::Query>:: The query object.
+        #
+        # ==== Returns
+        # Hash:: An document hash of the first document matched. <tt>{:id => 1}</tt>
+        # Nil::  If no documents match.
+        def read_one(query)
+          read(query).first
+        end
+
+        protected
+          # List sphinx indexes to search.
+          #
+          # If no indexes are explicitly declared using DataMapper::Adapters::Sphinx::Resource then the default storage
+          # name is used.
+          #
+          # ==== See
+          # * DataMapper::Adapters::Sphinx::Resource::ClassMethods#sphinx_indexes
+          #
+          # ==== Parameters
+          # model<DataMapper::Model>:: The DataMapper::Model.
+          #
+          # ==== Returns
+          # Array<DataMapper::Adapters::Sphinx::Index>:: Index objects from the model.
+          def indexes(model)
+            indexes = model.sphinx_indexes(repository(self.name).name) if model.respond_to?(:sphinx_indexes)
+            if indexes.nil? or indexes.empty?
+              indexes = [Index.new(model, model.storage_name)]
+            end
+            indexes
+          end
+
+          # List sphinx delta indexes to search.
+          #
+          # ==== See
+          # * DataMapper::Adapters::Sphinx::Resource::ClassMethods#sphinx_indexes
+          #
+          # ==== Parameters
+          # model<DataMapper::Model>:: The DataMapper::Model.
+          #
+          # ==== Returns
+          # Array<DataMapper::Adapters::Sphinx::Index>:: Index objects from the model.
+          def delta_indexes(model)
+            indexes(model).find_all{|i| i.delta?}
+          end
+
+          # Query sphinx for a list of document IDs.
+          #
+          # ==== Parameters
+          # query<DataMapper::Query>:: The query object.
+          #
+          # ==== Returns
+          # Array<Hash>:: An array of document hashes. <tt>[{:id => 1}, {:id => 2}]</tt>
+          # Array<>::     An empty array if no documents match.
+          def read(query)
+            from    = indexes(query.model).map{|index| index.name}.join(', ')
+            search  = Sphinx::Query.new(query).to_s
+            options = {
+              :match_mode => :extended,
+              :filters    => search_filters(query) # By attribute.
+            }
+            options[:limit]  = query.limit.to_i  if query.limit
+            options[:offset] = query.offset.to_i if query.offset
+
+            if order = search_order(query)
+              options.update(
+                :sort_mode => :extended,
+                :sort_by   => order
+              )
+            end
+
+            res = @client.search(search, from, options)
+            raise res[:error] unless res[:error].nil?
+
+            DataMapper.logger.info(
+              %q{Sphinx (%.3f): search '%s' in '%s' found %d documents} % [res[:time], search, from, res[:total]]
+            )
+            res[:matches].map{|doc| {:id => doc[:doc]}}
+          end
+
+
+          # Riddle search filters for attributes.
+          def search_filters(query) #:nodoc:
+            filters = []
+            query.conditions.each do |operator, attribute, value|
+              next unless attribute.kind_of? Sphinx::Attribute
+              filters << case operator
+                when :eql, :like then Riddle::Client::Filter.new(attribute.name.to_s, filter_value(value))
+                when :not        then Riddle::Client::Filter.new(attribute.name.to_s, filter_value(value), true)
+                else raise NotImplementedError.new("Sphinx: Query attributes do not support the #{operator} operator")
+              end
+            end
+            filters
+          end
+
+          # TODO: How do you tell the difference between the default query order and someone explicitly asking for
+          # sorting by the primary key? I don't think you can at the moment.
+          def search_order(query) #:nodoc:
+            by = []
+            query.order.each do |order|
+              next unless order.property.kind_of? Sphinx::Attribute
+              by << [order.property.field, order.direction].join(' ')
+            end
+            by.empty? ? nil : by.join(', ')
+          end
+
+          # TODO: Move this to Attribute#dump.
+          # This is ninja'd straight from TS just to get things going.
+          def filter_value(value) #:nodoc:
+            case value
+              when Range
+                value.first.is_a?(Time) ? value.first.to_i..value.last.to_i : value
+              when Array
+                value.collect { |val| val.is_a?(Time) ? val.to_i : val }
+              else
+                Array(value)
+            end
+          end
+      end # Adapter
+    end # Sphinx
+
+    # Keep magic in DataMapper#setup happy.
+    SphinxAdapter = Sphinx::Adapter
+  end # Adapters
+end # DataMapper
+