bdimcheff-dm-sphinx-adapter 0.8.0

data/README.rdoc

@@ -0,0 +1,200 @@
+= DataMapper Sphinx Adapter
+
+* http://dm-sphinx.rubyforge.org
+* http://rubyforge.org/projects/dm-sphinx
+* http://github.com/shanna/dm-sphinx-adapter/tree/master
+
+== Description
+
+A DataMapper Sphinx adapter.
+
+== Dependencies
+
+Ruby::
+* dm-core           ~> 0.9.7
+* dm-is-searchable  ~> 0.9.7 (optional)
+
+I'd recommend using the dm-more plugin dm-is-searchable instead of fetching the document id's yourself.
+
+Sphinx::
+* 0.9.8-r871
+* 0.9.8-r909
+* 0.9.8-r985
+* 0.9.8-r1065
+* 0.9.8-r1112
+* 0.9.8-rc1 (gem version: 0.9.8.1198)
+* 0.9.8-rc2 (gem version: 0.9.8.1231)
+* 0.9.8 (gem version: 0.9.8.1371)
+
+Internally the Riddle client library is used.
+
+== Install
+
+* Via git: git clone git://github.com/shanna/dm-sphinx-adapter.git
+* Via gem: gem install shanna-dm-sphinx-adapter -s http://gems.github.com
+
+== Synopsis
+
+DataMapper uses URIs or a connection has to connect to your data-stores. In this case the sphinx search daemon
+<tt>searchd</tt>.
+
+On its own this adapter will only return an array of document hashes when queried. The DataMapper library
+<tt>dm-is-searchable</tt> however provides a common interface to search one adapter and load documents from another. My
+preference is to use this adapter in tandem with <tt>dm-is-searchable</tt>. See further examples in the synopsis for
+usage with <tt>dm-is-searchable</tt>.
+
+Like all DataMapper adapters you can connect with a Hash or URI.
+
+A URI:
+  DataMapper.setup(:search, 'sphinx://localhost')
+
+The breakdown is:
+  "#{adapter}://#{host}:#{port}/#{config}"
+  - adapter Must be :sphinx
+  - host    Hostname (default: localhost)
+  - port    Optional port number (default: 3312)
+
+Alternatively supply a Hash:
+  DataMapper.setup(:search, {
+    :adapter  => 'sphinx',       # required
+    :config   => './sphinx.conf' # optional. Recommended though.
+    :host     => 'localhost',    # optional. Default: localhost
+    :port     => 3312            # optional. Default: 3312
+  }
+
+=== DataMapper
+
+  require 'rubygems'
+  require 'dm-sphinx-adapter'
+
+  DataMapper.setup(:default, 'sqlite3::memory:')
+  DataMapper.setup(:search, 'sphinx://localhost:3312')
+
+  class Item
+    include DataMapper::Resource
+    property :id, Serial
+    property :name, String
+  end
+
+  # Fire up your sphinx search daemon and start searching.
+  docs  = repository(:search){ Item.all(:name => 'barney') } # Search 'items' index for '@name barney'
+  ids   = docs.map{|doc| doc[:id]}
+  items = Item.all(:id => ids) # Search :default for all the document id's returned by sphinx.
+
+=== DataMapper and IsSearchable
+
+IsSearchable is a DataMapper plugin that provides a common search interface when searching from one adapter and reading
+documents from another.
+
+IsSearchable will read resources from your <tt>:default</tt> repository on behalf of a search adapter such as
+<tt>dm-sphinx-adapter</tt> and <tt>dm-ferret-adapter</tt>. This saves some of the grunt work (as shown in the previous
+example) by mapping the resulting document id's from a search with your <tt>:search</tt> adapter into a suitable
+<tt>#first</tt> or <tt>#all</tt> query for your <tt>:default</tt> repository.
+
+IsSearchable adds a single class method to your resource. The first argument is a <tt>Hash</tt> of
+<tt>DataMapper::Query</tt> conditions to pass to your search adapter (in this case <tt>dm-sphinx-adapter</tt>). An
+optional second <tt>Hash</tt> of <tt>DataMapper::Query</tt> conditions can also be passed and will be appended to the
+query on your <tt>:default</tt> database. This can be handy if you need to add extra exclusions that aren't possible
+using <tt>dm-sphinx-adapter</tt> such as <tt>#gt</tt> or <tt>#lt</tt> conditions.
+
+  require 'rubygems'
+  require 'dm-core'
+  require 'dm-is-searchable'
+  require 'dm-sphinx-adapter'
+
+  # Connections.
+  DataMapper.setup(:default, 'sqlite3::memory:')
+  DataMapper.setup(:search, 'sphinx://localhost:3312')
+
+  class Item
+    include DataMapper::Resource
+    property :id, Serial
+    property :name, String
+
+    is :searchable # defaults to :search repository though you can be explicit:
+    # is :searchable, :repository => :sphinx
+  end
+
+  # Fire up your sphinx search daemon and start searching.
+  items = Item.search(:name => 'barney') # Search 'items' index for '@name barney'
+
+=== Merb, DataMapper and IsSearchable
+
+  # config/init.rb
+  dependency 'dm-is-searchable'
+  dependency 'dm-sphinx-adapter'
+
+  # config/database.yml
+  ---
+  development: &defaults
+    repositories:
+      search:
+        adapter:  sphinx
+        host:     localhost
+        port:     3312
+
+  # app/models/item.rb
+  class Item
+    include DataMapper::Resource
+    property :id, Serial
+    property :name, String
+
+    is :searchable # defaults to :search repository though you can be explicit:
+    # is :searchable, :repository => :sphinx
+  end # Item
+
+  # Fire up your sphinx search daemon and start searching.
+  Item.search(:name => 'barney') # Search 'items' index for '@name barney'
+
+=== DataMapper, IsSearchable and DataMapper::SphinxResource
+
+For finer grained control you can include DataMapper::SphinxResource. For instance you can search one or more indexes
+and sort, include or exclude by attributes defined in your sphinx configuration:
+
+  class Item
+    include DataMapper::SphinxResource
+    property :id, Serial
+    property :name, String
+
+    is :searchable
+    repository(:search) do
+      index :items
+      index :items_delta, :delta => true
+
+      # Sphinx attributes to sort include/exclude by.
+      attribute :updated_on, DateTime
+    end
+
+  end # Item
+
+  # Search 'items, items_delta' index for '@name barney' updated in the last 30 minutes.
+  Item.search(:name => 'barney', :updated => (Time.now - 1800 .. Time.now))
+
+== Sphinx Configuration
+
+No limitations, restrictions or requirement are imposed on your sphinx configuration. The adapter will not generate nor
+overwrite your finely crafted config file.
+
+== Searchd
+
+To keep things simple, this adapter does not manage your sphinx server. Try one of these fine offerings:
+
+* god[http://god.rubyforge.org]
+* daemon_controller[http://github.com/FooBarWidget/daemon_controller/tree/master]
+* monit[http://www.tildeslash.com/monit]
+
+== Indexer and Live(ish) updates.
+
+As of 0.3 the indexer will no longer be fired on create/update even if you have delta indexes defined. Sphinx indexing
+is blazing fast but unless your resource sees very little activity you will run the risk of lock errors on
+the temporary delta index files (.tmpl.sp1) and your delta index won't be updated. Given this functionality is
+unreliable at best I've chosen to remove it.
+
+For reliable live(ish) updates in a main + delta scheme it's probably best you schedule them outside of your ORM.
+Andrew (Shodan) Aksyonoff of Sphinx suggests a cronjob or alternatively if you need even less lag to "run indexer in
+an endless loop, with a few seconds of sleep in between to allow searchd some headroom to pick up the changes".
+
+== Contributing
+
+Go nuts. Just send me a pull request (github or otherwise) when you are happy with your code.
+

data/Rakefile

@@ -0,0 +1,49 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name        = "dm-sphinx-adapter"
+    gem.summary     = %q{A DataMapper Sphinx adapter.}
+    gem.email       = "shane.hanna@gmail.com"
+    gem.homepage    = "http://github.com/shanna/dm-sphinx-adapter"
+    gem.authors     = ["Shane Hanna"]
+    gem.add_dependency 'dm-core', ['~> 0.9']
+    gem.files.reject!{|f| f=~ %r{test/files/tmp/.*}}
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = 'dm-sphinx-adapter'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/test_*.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :default => :test

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:major: 0
+:minor: 8
+:patch: 0

data/lib/dm-sphinx-adapter.rb

@@ -0,0 +1,23 @@
+require 'rubygems'
+
+# TODO: Hide the shitload of dm-core warnings or at least try to?
+old_verbose, $VERBOSE = $VERBOSE, nil
+  gem 'dm-core', '~> 0.9.8'
+  require 'dm-core'
+$VERBOSE = old_verbose
+
+require 'pathname'
+lib = Pathname(__FILE__).dirname.expand_path
+dir = lib / 'dm-sphinx-adapter'
+
+# Bundled Riddle since the gem is very old and we don't need any of the config generation stuff.
+$:.unshift lib
+require 'riddle'
+
+# TODO: Require farms suck. Do something about it.
+require dir / 'adapter'
+require dir / 'attribute'
+require dir / 'collection'
+require dir / 'index'
+require dir / 'query'
+require dir / 'resource'

data/lib/dm-sphinx-adapter/adapter.rb

@@ -0,0 +1,200 @@
+module DataMapper
+  module Adapters
+    module Sphinx
+      # == Synopsis
+      #
+      # DataMapper uses URIs or a connection has to connect to your data-stores. In this case the sphinx search daemon
+      # <tt>searchd</tt>.
+      #
+      # On its own this adapter will only return an array of document hashes when queried. The DataMapper library dm-more
+      # however provides dm-is-searchable, a common interface to search one adapter and load documents from another. My
+      # preference is to use this adapter in tandem with dm-is-searchable.
+      #
+      # Like all DataMapper adapters you can connect with a Hash or URI.
+      #
+      # A URI:
+      #   DataMapper.setup(:search, 'sphinx://localhost')
+      #
+      # The breakdown is:
+      #   "#{adapter}://#{host}:#{port}/#{config}"
+      #   - adapter Must be :sphinx
+      #   - host    Hostname (default: localhost)
+      #   - port    Optional port number (default: 3312)
+      #
+      # Alternatively supply a Hash:
+      #   DataMapper.setup(:search, {
+      #     :adapter  => 'sphinx',       # required
+      #     :host     => 'localhost',    # optional. Default: localhost
+      #     :port     => 3312            # optional. Default: 3312
+      #   })
+      class Adapter < AbstractAdapter
+
+        # ==== See
+        # * DataMapper::Adapters::AbstractAdapter
+        #
+        # ==== Parameters
+        # uri_or_options<URI, DataObject::URI, Addressable::URI, String, Hash, Pathname>::
+        #   DataMapper uri or options hash.
+        def initialize(name, uri_or_options)
+          super # Set up defaults.
+          @options = normalize_options(uri_or_options)
+        end
+
+        def create(resources) #:nodoc:
+          0
+        end
+
+        def delete(query) #:nodoc:
+          0
+        end
+
+        # Query your Sphinx repository and return all matching documents.
+        #
+        # ==== Notes
+        #
+        # These methods are public but normally called indirectly through DataMapper::Resource#get,
+        # DataMapper::Resource#first or DataMapper::Resource#all.
+        #
+        # The document hashes returned are those from Riddle::Client.
+        #
+        # ==== Parameters
+        # query<DataMapper::Query>:: The query object.
+        #
+        # ==== Returns
+        # Array<Hash>:: An array of document hashes. <tt>[{:id => 1, ...}, {:id => 2, ...}]</tt>
+        # Array<>::     An empty array if no documents match.
+        def read_many(query)
+          read(query)
+        end
+
+        # Query your Sphinx repository and return the first document matched.
+        #
+        # ==== Notes
+        #
+        # These methods are public but normally called indirectly through DataMapper::Resource#get,
+        # DataMapper::Resource#first or DataMapper::Resource#all.
+        #
+        # ==== Parameters
+        # query<DataMapper::Query>:: The query object.
+        #
+        # ==== Returns
+        # Hash:: An document hash of the first document matched. <tt>{:id => 1, ...}</tt>
+        # Nil::  If no documents match.
+        def read_one(query)
+          read(query).first
+        end
+
+        protected
+          # List sphinx indexes to search.
+          #
+          # If no indexes are explicitly declared using DataMapper::Adapters::Sphinx::Resource then the default storage
+          # name is used.
+          #
+          # ==== See
+          # * DataMapper::Adapters::Sphinx::Resource::ClassMethods#sphinx_indexes
+          #
+          # ==== Parameters
+          # model<DataMapper::Model>:: The DataMapper::Model.
+          #
+          # ==== Returns
+          # Array<DataMapper::Adapters::Sphinx::Index>:: Index objects from the model.
+          def indexes(query)
+            indexes = query.model.sphinx_indexes(name) if query.model.respond_to?(:sphinx_indexes)
+            if indexes.nil? or indexes.empty?
+              indexes = [Index.new(query.model, query.model.storage_name(name))]
+            end
+            indexes
+          end
+
+          # Query sphinx for a list of document IDs.
+          #
+          # ==== Parameters
+          # query<DataMapper::Query>:: The query object.
+          #
+          # ==== Returns
+          # Array<Hash>:: An array of document hashes. <tt>[{:id => 1, ...}, {:id => 2, ...}]</tt>
+          # Array<>::     An empty array if no documents match.
+          def read(query)
+            from   = indexes(query).map{|index| index.name}.join(', ')
+            search = Sphinx::Query.new(query).to_s
+            client = Riddle::Client.new(@options[:host], @options[:port])
+
+            # You can set some options that aren't set by the adapter.
+            @options.except(:host, :port, :match_mode, :limit, :offset, :sort_mode, :sort_by).each do |k, v|
+              client.method("#{k}=".to_sym).call(v) if client.respond_to?("#{k}=".to_sym)
+            end
+
+            client.match_mode = :extended
+            client.filters    = search_filters(query) # By attribute.
+            client.limit      = query.limit.to_i  if query.limit
+            client.offset     = query.offset.to_i if query.offset
+
+            if order = search_order(query)
+              client.sort_mode = :extended
+              client.sort_by   = order
+            end
+
+            result = client.query(search, from)
+            raise result[:error] unless result[:error].nil?
+
+            DataMapper.logger.info(
+              %q{Sphinx (%.3f): search '%s' in '%s' found %d documents} % [result[:time], search, from, result[:total]]
+            )
+            # TODO: Confusing, call it something other than collection?
+            Collection.new(result)
+            # result[:matches].map{|doc| doc[:id] = doc[:doc]; doc}
+          end
+
+
+          # Riddle search filters for attributes.
+          def search_filters(query) #:nodoc:
+            filters = []
+            query.conditions.each do |operator, attribute, value|
+              next unless attribute.kind_of? Sphinx::Attribute
+              filters << case operator
+                when :eql, :like then attribute.filter(value)
+                when :not        then attribute.filter(value, false)
+                else raise NotImplementedError.new("Sphinx: Query attributes do not support the #{operator} operator")
+              end
+            end
+            filters
+          end
+
+          # TODO: How do you tell the difference between the default query order and someone explicitly asking for
+          # sorting by the primary key? I don't think you can at the moment.
+          def search_order(query) #:nodoc:
+            by = []
+            query.order.each do |order|
+              next unless order.property.kind_of? Sphinx::Attribute
+              by << [order.property.field, order.direction].join(' ')
+            end
+            by.empty? ? nil : by.join(', ')
+          end
+
+          # Coerce +uri_or_options+ into a +Hash+ of options.
+          #
+          # ==== Parameters
+          # uri_or_options<URI, DataObject::URI, Addressable::URI, String, Hash, Pathname>::
+          #   DataMapper uri or options hash.
+          #
+          # ==== Returns
+          # Hash
+          def normalize_options(uri_or_options)
+            case uri_or_options
+              when String, Addressable::URI then DataObjects::URI.parse(uri_or_options).attributes
+              when DataObjects::URI         then uri_or_options.attributes
+              when Pathname                 then {:path => uri_or_options}
+              else
+                uri_or_options[:path] ||= uri_or_options.delete(:config) || uri_or_options.delete(:database)
+                uri_or_options
+            end
+          end
+
+      end # Adapter
+    end # Sphinx
+
+    # Keep magic in DataMapper#setup happy.
+    SphinxAdapter = Sphinx::Adapter
+  end # Adapters
+end # DataMapper
+