ferret 0.1.0

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2005 David Balmain
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,109 @@
+= Ferret
+
+Ferret is a Ruby port of the Java Lucene search engine.
+(http://jakarta.apache.org/lucene/) In the same way as Lucene, it is not a
+standalone application, but a library you can use to index documents and
+search for things in them later.
+
+== Requirements
+
+* Ruby 1.8
+* (C compiler to build the extension but not required to use Ferret)
+
+== Installation
+
+De-compress the archive and enter its top directory.
+
+  tar zxpvf ferret-0.1.tar.gz
+  cd ferret-0.1
+
+Run the setup config;
+
+  $ ruby setup.rb config
+
+Then to compile the C extension (optional) type:
+
+  $ ruby setup.rb setup
+
+If you don't have a C compiler, never mind. Just go straight to the next step.
+On *nix you'll need to run this with root privalages. Type;
+
+  # ruby setup.rb install
+  
+These simple steps install ferret in the default location of Ruby libraries.
+You can also install files into your favorite directory by supplying setup.rb
+some options. Try;
+
+ $ ruby setup.rb --help
+
+
+== Usage
+
+You can read the TUTORIAL which you'll find in the same directory as this
+README. You can also check the following modules for more specific
+documentation.
+
+* Ferret::Analysis: for more information on how the data is processed when it
+  is tokenized. There are a number of things you can do with your data such as
+  adding stop lists or perhaps a porter stemmer. There are also a number of
+  analyzers already available and it is almost trivial to create a new one
+  with a simple regular expression.
+
+* Ferret::Search: for more information on querying the index. There are a
+  number of already available queries and it's unlikely you'll need to create
+  your own. You may however want to take advantage of the sorting or filtering
+  abilities of Ferret to present your data the best way you see fit.
+
+* Ferret::Document: to find out how to create documents. This part of Ferret
+  is relatively straightforward. The main thing that we haven't gone into here
+  is the use of term vectors. These allow you to store and retrieve the
+  positions and offsets of the data which can be very useful in document
+  comparison amoung other things.  == More information
+
+* Ferret::QueryParser: if you want to find out more about what you can do with
+  Ferret's Query Parser, this is the place to look. The query parser is one
+  area that could use a bit of work so please send your suggestions.
+
+* Ferret::Index: for more advanced access to the index you'll probably want to
+  use the Ferret::Index::IndexWriter and Ferret::Index::IndexReader. This is
+  the place to look for more information on them.
+
+* Ferret::Store: This is the module used to access the actual index storage
+  and won't be of much interest to most people.
+
+=== Performance
+
+Currently Ferret is an order of magnitude slower than Java Lucene which can be
+quite a pain at times. I have written some basic C extensions which may or may
+not have installed when you installed Ferret. These double the speed but still
+leave it a lot slower than the Java version. I have, however, ported the
+indexing part of Java Lucene to C and it is an order of magnitude faster then
+the Java version. Once I'm pretty certain that the API of Ferret has settled
+and won't be changing much, I'll intergrate my C version. So expect to see
+Ferret running faster than Java Lucene some time in the future. If you'd like
+to try cferret and test my claims, let me know (if you haven't already found
+it in my subversion repository). It's not currently portable and will probably
+only run on linux.
+
+== Contact
+
+Bug reports, patches, queries, discussion etc should be addressed to
+the mailing list. More information on the list can be found at:
+
+http://ferret.davebalmain.com/
+
+Of course, since Ferret is almost a straight port of Java Lucene,
+everything said about Lucene at http://jakarta.apache.org/lucene/ should
+be true about Ferret. Apart from the bits about it being in Java.
+
+== Authors
+
+[<b>David Balmain</b>] Port to Ruby
+
+[<b>Doug Cutting and friends</b>] Original Java Lucene
+
+== License
+
+Ferret is available under an MIT-style license.
+
+:include: MIT-LICENSE

data/Rakefile

@@ -0,0 +1,275 @@
+$:. << 'lib'
+# Some parts of this Rakefile where taken from Jim Weirich's Rakefile for
+# Rake. Other parts where stolen from the David Heinemeier Hansson's Rails
+# Rakefile. Both are under MIT-LICENSE. Thanks to both for their excellent
+# projects.
+
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rake/clean'
+require 'rake_utils/code_statistics'
+require 'lib/ferret'
+
+begin
+  require 'rubygems'
+  require 'rake/gempackagetask'
+rescue Exception
+  nil
+end
+
+CURRENT_VERSION = Ferret::VERSION
+if ENV['REL']
+  PKG_VERSION = ENV['REL']
+else
+  PKG_VERSION = CURRENT_VERSION
+end
+
+def announce(msg='')
+  STDERR.puts msg
+end
+
+$VERBOSE = nil
+CLEAN.include(FileList['**/*.o', 'InstalledFiles', '.config'])
+CLOBBER.include(FileList['**/*.so'], 'ext/Makefile')
+
+task :default => :all_tests
+desc "Run all tests"
+task :all_tests => [ :test_units, :test_functional ]
+
+desc "Generate API documentation, and show coding stats"
+task :doc => [ :stats, :appdoc ]
+
+desc "run unit tests in test/unit"
+Rake::TestTask.new("test_units" => :parsers) do |t|
+  t.libs << "test/unit"
+  t.pattern = 'test/unit/t[cs]_*.rb'
+  t.verbose = true
+end
+
+desc "run unit tests in test/unit"
+Rake::TestTask.new("test_long") do |t|
+  t.libs << "test"
+  t.libs << "test/unit"
+  t.test_files = FileList["test/longrunning/tm_store.rb"]
+  t.pattern = 'test/unit/t[cs]_*.rb'
+  t.verbose = true
+end
+
+desc "run funtional tests in test/funtional"
+Rake::TestTask.new("test_functional") do |t|
+  t.libs << "test"
+  t.pattern = 'test/funtional/tc_*.rb'
+  t.verbose = true
+end
+
+desc "Report code statistics (KLOCS, etc) from application"
+task :stats do
+  CodeStatistics.new(
+                      ["Ferret", "lib/ferret"],
+                      ["Units", "test/unit"],
+                      ["Units-extended", "test/longrunning"]
+                    ).to_s
+end
+
+desc "Generate documentation for the application"
+rd = Rake::RDocTask.new("appdoc") do |rdoc|
+  rdoc.rdoc_dir = 'doc/api'
+  rdoc.title    = "Ferret Search Library Documentation"
+  rdoc.options << '--line-numbers --inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('TODO')
+  rdoc.rdoc_files.include('TUTORIAL')
+  rdoc.rdoc_files.include('MIT-LICENSE')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+EXT = "ferret_ext.so"
+
+desc "Build the extension"
+task :ext => "ext/#{EXT}"
+
+file "ext/#{EXT}" => "ext/Makefile" do
+  sh "cd ext; make"
+end
+
+file "ext/Makefile" do
+  sh "cd ext; ruby extconf.rb"
+end
+
+# Make Parsers ---------------------------------------------------------------
+
+RACC_SRC = FileList["**/*.y"]
+RACC_OUT = RACC_SRC.collect { |fn| fn.sub(/\.y$/, '.tab.rb') }
+
+task :parsers => RACC_OUT
+rule(/\.tab\.rb$/ => [proc {|tn| tn.sub(/\.tab\.rb$/, '.y')}]) do |t|
+  sh "racc #{t.source}" 
+end
+
+# Create Packages ------------------------------------------------------------
+
+PKG_FILES = FileList[
+  'setup.rb',
+  '[-A-Z]*',
+  'ext/**/*', 
+  'lib/**/*.rb', 
+  'test/**/*.rb',
+  'rake_utils/**/*.rb',
+  'Rakefile'
+]
+PKG_FILES.exclude('**/*.o')
+
+
+if ! defined?(Gem)
+  puts "Package Target requires RubyGEMs"
+else
+  spec = Gem::Specification.new do |s|
+    
+    #### Basic information.
+
+    s.name = 'ferret'
+    s.version = PKG_VERSION
+    s.summary = "Ruby indexing library."
+    s.description = <<-EOF
+      Ferret is a port of the Java Lucene project. It is a powerful
+      indexing and search library.
+    EOF
+
+    #### Dependencies and requirements.
+
+    #s.add_dependency('log4r', '> 1.0.4')
+    #s.requirements << ""
+
+    #### Which files are to be included in this gem?  Everything!  (Except CVS directories.)
+
+    s.files = PKG_FILES.to_a
+
+    #### C code extensions.
+
+    s.extensions << "ext/extconf.rb"
+
+    #### Load-time details: library and application (you will need one or both).
+
+    s.require_path = 'lib'                         # Use these for libraries.
+
+    #s.bindir = "bin"                               # Use these for applications.
+    #s.executables = ["rake"]
+    #s.default_executable = "rake"
+
+    #### Documentation and testing.
+
+    s.has_rdoc = true
+    s.extra_rdoc_files = rd.rdoc_files.reject { |fn| fn =~ /\.rb$/ }.to_a
+    s.rdoc_options <<
+      '--title' <<  'Ferret -- Ruby Indexer' <<
+      '--main' << 'README' << '--line-numbers' <<
+      'TUTORIAL' << 'TODO'
+
+    #### Author and project details.
+
+    s.author = "David Balmain"
+    s.email = "dbalmain@gmail.com"
+    s.homepage = "http://ferret.davebalmain.com"
+    s.rubyforge_project = "ferret"
+#     if ENV['CERT_DIR']
+#       s.signing_key = File.join(ENV['CERT_DIR'], 'gem-private_key.pem')
+#       s.cert_chain  = [File.join(ENV['CERT_DIR'], 'gem-public_cert.pem')]
+#     end
+  end
+
+  package_task = Rake::GemPackageTask.new(spec) do |pkg|
+    pkg.need_zip = true
+    pkg.need_tar = true
+  end
+end
+
+# Support Tasks ------------------------------------------------------
+
+desc "Look for TODO and FIXME tags in the code"
+task :todo do
+  FileList['**/*.rb'].egrep /#.*(FIXME|TODO|TBD)/
+end
+# --------------------------------------------------------------------
+# Creating a release
+
+desc "Make a new release"
+task :prerelease => [:clobber, :all_tests, :parsers]
+task :package => [:prerelease]
+task :tag => [:prerelease]
+task :update_version => [:prerelease]
+task :release => [:tag, :update_version, :package] do
+  announce 
+  announce "**************************************************************"
+  announce "* Release #{PKG_VERSION} Complete."
+  announce "* Packages ready to upload."
+  announce "**************************************************************"
+  announce 
+end
+
+# Validate that everything is ready to go for a release.
+task :prerelease do
+  announce 
+  announce "**************************************************************"
+  announce "* Making RubyGem Release #{PKG_VERSION}"
+  announce "* (current version #{CURRENT_VERSION})"
+  announce "**************************************************************"
+  announce  
+
+  # Is a release number supplied?
+  unless ENV['REL']
+    fail "Usage: rake release REL=x.y.z [REUSE=tag_suffix]"
+  end
+
+  # Is the release different than the current release.
+  # (or is REUSE set?)
+  if PKG_VERSION == CURRENT_VERSION && ! ENV['REUSE']
+    fail "Current version is #{PKG_VERSION}, must specify REUSE=tag_suffix to reuse version"
+  end
+
+  # Are all source files checked in?
+  data = `svn -q status`
+  unless data =~ /^$/
+    fail "'svn -q status' is not clean ... do you have unchecked-in files?"
+  end
+  
+  announce "No outstanding checkins found ... OK"
+end
+
+task :update_version => [:prerelease] do
+  if PKG_VERSION == CURRENT_VERSION
+    announce "No version change ... skipping version update"
+  else
+    announce "Updating Ferret version to #{PKG_VERSION}"
+    open("lib/ferret.rb") do |ferret_in|
+      open("lib/ferret.rb.new", "w") do |ferret_out|
+        ferret_in.each do |line|
+          if line =~ /^  VERSION\s*=\s*/
+            ferret_out.puts "  VERSION = '#{PKG_VERSION}'"
+          else
+            ferret_out.puts line
+          end
+        end
+      end
+    end
+    if ENV['RELTEST']
+      announce "Release Task Testing, skipping commiting of new version"
+    else
+      mv "lib/ferret.rb.new", "lib/ferret.rb"
+    end
+    sh %{svn ci -m "Updated to version #{PKG_VERSION}" lib/ferret.rb}
+  end
+end
+
+desc "Tag all the SVN files with the latest release number (REL=x.y.z)"
+task :tag => [:prerelease] do
+  reltag = "REL-#{PKG_VERSION}"
+  reltag << ENV['REUSE'] if ENV['REUSE']
+  announce "Tagging SVN with [#{reltag}]"
+  if ENV['RELTEST']
+    announce "Release Task Testing, skipping SVN tagging. Would do the following;"
+    announce %{svn copy -m "creating release #{reltag}" svn://www.davebalmain.com/ferret/trunk svn://www.davebalmain.com/ferret/tags/#{reltag}}
+  else
+    sh %{svn copy -m "creating release #{reltag}" svn://www.davebalmain.com/ferret/trunk svn://www.davebalmain.com/ferret/tags/#{reltag}}
+  end
+end

data/TODO

@@ -0,0 +1,9 @@
+= Ferret Project -- To Do List
+
+Send suggestions for this list to mailto:dbalmain@gmail.com
+
+=== To Do
+
+* Add the ability to persist an in memory index to Ferret::Index::Index
+* Add unicode support
+

data/TUTORIAL

@@ -0,0 +1,197 @@
+= Quick Introduction to Ferret
+
+The simplest way to use Ferret is through the Ferret::Index::Index class.
+Start by including the Ferret module.
+
+  require 'ferret'
+  include Ferret
+
+=== Creating an index
+
+To create an in memory index is very simple;
+  
+  index = Index::Index.new()
+
+To create a persistent index;
+
+  index = Index::Index.new(:path => '/path/to/index')
+
+Both of these methods create new Indexes with the StandardAnalyzer. An
+analyzer is what you use to divide the input data up into tokens which you can
+search for later. If you'd like to use a different analyzer you can specify it
+here, eg;
+
+  index = Index::Index.new(:path => '/path/to/index',
+                           :analyzer => WhiteSpaceAnalyzer.new)
+
+For more options when creating an Index refer to Ferret::Index::Index.
+
+=== Adding Documents
+
+To add a document you can simply add a string or an array of strings.
+
+  index << "This is a new document to be indexed"
+  index << ["And here", "is another", "new document", "to be indexed"]
+
+But these are pretty simple documents. If this is all you want to index you
+could probably just use SimpleSearch. So let's give our documents some fields;
+
+  index << {:title => "Programming Ruby", :content => "blah blah blah"}
+  index << {:title => "Programming Ruby", :content => "yada yada yada"}
+
+Or if you are indexing data stored in a database, you'll probably want to
+store the id;
+
+  index << {:id => row.id, :title => row.title, :date => row.date}
+
+The methods above while store all of the input data as well tokenizing and
+indexing it. Sometimes we won't want to tokenize (divide the string into
+tokens) the data. For example, we might want to leave the title as a complete
+string and only allow searchs for that complete string. Sometimes we won't
+want to store the data as it's already stored in the database so it'll be a
+waste to store it in the index. Or perhaps we are doing without a database and 
+using Ferret to store all of our data, in which case we might not want to
+index it. For example, if we are storing images in the index, we won't want to
+index them. All of this can be done using Ferret's Ferret::Document module.
+eg;
+
+  include Ferret::Document
+  doc = Document.new
+  doc << Field.new("id",    row.id,    Field::Store::NO,  Field::Index::UNTOKENIZED)
+  doc << Field.new("title", row.title, Field::Store::YES, Field::Index::UNTOKENIZED)
+  doc << Field.new("data",  row.data,  Field::Store::YES, Field::Index::TOKENIZED)
+  doc << Field.new("image", row.image, Field::Store::YES, Field::Index::NO)
+  index << doc
+
+You can also compress the data that you are storing or store term vectors with
+the data. Read more about this in Ferret::Document::Field.
+
+=== Searching
+
+Now that we have data in our index, how do we actually use this index to
+search the data? The Index offers two search methods, Index#search and
+Index#search_each. The first method returns a Ferret::Index::TopDocs object.
+The second we'll show here. Lets say we wanted to find all documents with the
+phrase "quick brown fox" in the content field. We'd write;
+
+  index.search('content:"quick brown fox"') do |doc, score|
+    puts "Document #{doc} found with a score of #{score}"
+  end
+
+But "fast" has a pretty similar meaning to "quick" and we don't mind if the
+fox is a little red. So we could expand our search like this;
+
+  index.search('content:"quick|fast brown|red fox"') do |doc, score|
+    puts "Document #{doc} found with a score of #{score}"
+  end
+
+What if we want to find all documents entered on or after 5th of September,
+2005 with the words "ruby" or "rails" in it. We could type something like;
+
+  index.search('date:( >= 20050905) content:(ruby OR rails)') do |doc, score|
+    puts "Document #{doc} found with a score of #{score}"
+  end
+
+Ferret has quite a complex query language. To find out more about Ferret's
+query language, see Ferret::QueryParser. You can also construct even more
+complex queries like Ferret::Search::Spans by hand. See Ferret::Search::Query
+for more information.
+
+=== Accessing Documents
+
+You may have noticed that when we run a search we only get the document number
+back. By itself this isn't much use to us. Getting the data from the index is
+very straightforward. For example if we want the title field form the 3rd
+document type;
+  
+  index[2]["title"]
+
+NOTE: documents are indexed from 0.
+
+Let's go back to the database example above. If we store all of our documents
+with an id then we can access that field using the id. As long as we called
+our id field "id" we can do this
+
+  id = "89721347"
+  index[id]["title"]
+
+If however we called our id field "key" we'll have to do this;
+
+  id = Index::Term.new("key", "89721347")
+  index[id]["title"]
+
+Pretty simple huh? You should note though that if there are more then one
+document with the same *id* or *key* then only the first one will be returned
+so it is probably better that you ensure the key is unique somehow. (Ferret
+cannot do that for you)
+
+
+=== Modifying and Deleting Documents
+
+What if we want to change the data in the index. Ferret doesn't actually let
+you change the data once it is in the index. But you can delete documents so
+the standard way to modify data is to delete it and re-add it again with the
+modifications made. It is important to note that when doing this the documents
+will get a new document number so you should be careful not to use a document
+number after the document has been deleted. Here is an examle of modifying a
+document;
+
+  index << {:title => "Programing Rbuy", :content => "blah blah blah"}
+  doc_num = nil
+  index.search('title:"Programing Rbuy"') {|doc, score| doc_num = doc}
+  return unless doc_num
+  doc = index[doc_num]
+  index.delete(doc_num)
+
+  # modify doc
+  doc["title"] = "Programming Ruby"
+
+  index << doc
+
+Again, we can use the the id field as above. This time though every document
+that matches the id will be deleted. Again, it is probably a good idea if you
+somehow ensure that your *ids* are kept unique.
+
+  id = "23453422"
+  index.delete(id)
+
+Or;
+
+  id = Index::Term.new("key", "23452345")
+  index.delete(id)
+  
+=== Onwards
+
+This is just a small sampling of what Ferret allows you to do.  Ferret, like
+Lucene, is designed to be extended, and allows you to construct your own query
+types, analyzers, and so on. Future versions of Ferret will contain more of
+these, as well as instructions for how to subclass the base modules to create
+your own. For now you can look in the following places for more documentation;
+
+* Ferret::Analysis: for more information on how the data is processed when it
+  is tokenized. There are a number of things you can do with your data such as
+  adding stop lists or perhaps a porter stemmer. There are also a number of
+  analyzers already available and it is almost trivial to create a new one
+  with a simple regular expression.
+
+* Ferret::Search: for more information on querying the index. There are a
+  number of already available queries and it's unlikely you'll need to create
+  your own. You may however want to take advantage of the sorting or filtering
+  abilities of Ferret to present your data the best way you see fit.
+
+* Ferret::Document: to find out how to create documents. This part of Ferret
+  is relatively straightforward. The main thing that we haven't gone into here
+  is the use of term vectors. These allow you to store and retrieve the
+  positions and offsets of the data which can be very useful in document
+  comparison amoung other things.  == More information
+
+* Ferret::QueryParser: if you want to find out more about what you can do with
+  Ferret's Query Parser, this is the place to look. The query parser is one
+  area that could use a bit of work so please send your suggestions.
+
+* Ferret::Index: for more advanced access to the index you'll probably want to
+  use the Ferret::Index::IndexWriter and Ferret::Index::IndexReader. This is
+  the place to look for more information on them.
+
+* Ferret::Store: This is the module used to access the actual index storage
+  and won't be of much interest to most people.