ferret 0.11.6 → 0.11.8.4

data/README

@@ -1,39 +1,27 @@
 = Ferret
 
-Ferret is a Ruby port of the Java Lucene search engine.
-(http://jakarta.apache.org/lucene/) In the same way as Lucene, it is not a
-standalone application, but a library you can use to index documents and
+Ferret is a Ruby search library inspired by the Apache Lucene search engine for
+Java (http://jakarta.apache.org/lucene/). In the same way as Lucene, it is not
+a standalone application, but a library you can use to index documents and
 search for things in them later.
 
 == Requirements
 
 * Ruby 1.8
-* C compiler to build the extension. Tested with gcc, VC6 and VC2005
+* C compiler to build the extension. Tested with gcc, VC6
+* make (or nmake on windows)
 
 == Installation
 
-If you have gems installed you can simply do;
+  $ sudo gem install ferret
 
-  gem install ferret
+If you don't have rubygems installed you can still install Ferret. Just
+download one of the zipped up versions of Ferret, unzip it and change into the
+unzipped directory. Then run the following set of commands;
 
-Otherwise, you will need Rake installed. De-compress the archive and enter its top directory.
-
-  tar zxpvf ferret-<version>.tar.gz
-  cd ferret-<version>
-
-Run the following;
-
-  $ rake ext
   $ ruby setup.rb config
   $ ruby setup.rb setup
-  # ruby setup.rb install
-  
-These simple steps install ferret in the default location of Ruby libraries.
-You can also install files into your favorite directory by supplying setup.rb
-some options. Try;
-
- $ ruby setup.rb --help
-
+  $ sudo ruby setup.rb install
 
 == Usage
 

data/RELEASE_CHANGES

@@ -0,0 +1,137 @@
+(in /home/dave/w/ferret/ruby)
+------------------------------------------------------------------------
+r830 | dbalmain | 2008-03-01 14:10:47 +1100 (Sat, 01 Mar 2008) | 1 line
+
+A few more updates to the build system
+------------------------------------------------------------------------
+r829 | dbalmain | 2008-03-01 13:55:16 +1100 (Sat, 01 Mar 2008) | 1 line
+
+A few more updates to the build system
+------------------------------------------------------------------------
+r828 | dbalmain | 2008-02-29 10:48:33 +1100 (Fri, 29 Feb 2008) | 1 line
+
+Moved largefile test to a new long_running test directory so that tests can be run more easily
+------------------------------------------------------------------------
+r826 | dbalmain | 2008-02-29 10:09:48 +1100 (Fri, 29 Feb 2008) | 1 line
+
+Updated references to ferret/version.rb. Updated Rakefile to allow release task testing
+------------------------------------------------------------------------
+r825 | dbalmain | 2008-02-29 10:02:27 +1100 (Fri, 29 Feb 2008) | 1 line
+
+Updated Rakefile to clean it up (added namespaces).
+------------------------------------------------------------------------
+r824 | dbalmain | 2008-02-29 10:01:08 +1100 (Fri, 29 Feb 2008) | 1 line
+
+Updated :filter_proc so that custom PostFilter extensions can be built and applied to searches. See ruby/examples/c_extensions/age_filter
+------------------------------------------------------------------------
+r823 | dbalmain | 2008-02-26 18:44:39 +1100 (Tue, 26 Feb 2008) | 1 line
+
+Fixed ticket #277. This will also help with the correct highlighting of matching urls in queries
+------------------------------------------------------------------------
+r822 | dbalmain | 2008-02-22 12:46:48 +1100 (Fri, 22 Feb 2008) | 1 line
+
+Updated to latest posh.h => http://poshlib.hookatooka.com/poshlib/
+------------------------------------------------------------------------
+r821 | dbalmain | 2008-02-22 09:05:25 +1100 (Fri, 22 Feb 2008) | 1 line
+
+Added ruby bindings to MultiMapper for testing in Ruby
+------------------------------------------------------------------------
+r820 | dbalmain | 2008-02-09 14:30:55 +1100 (Sat, 09 Feb 2008) | 1 line
+
+Updated svn:ignore properties to handle new stemmer files
+------------------------------------------------------------------------
+r819 | dbalmain | 2008-02-09 14:27:46 +1100 (Sat, 09 Feb 2008) | 9 lines
+
+Fixed Ticket #337. StemFilter.new now works with 'English', :english or
+'EnGlIsH' and 'UTF_8' or 'utf-8' etc.
+
+Also, good news for Norwegians, Romanians, Turks and Finns. We now have 3 new
+stemmers for Norwegian, Romanian and Turkish and 2 new stop-word lists for
+Finnish and Hungarian. Please try them out and let me know if there are any
+problems.
+
+
+------------------------------------------------------------------------
+r818 | dbalmain | 2008-02-09 13:20:37 +1100 (Sat, 09 Feb 2008) | 1 line
+
+Removed old version of stemmer. Making way for new version. This will break build temporarily
+------------------------------------------------------------------------
+r817 | dbalmain | 2008-02-09 11:55:02 +1100 (Sat, 09 Feb 2008) | 1 line
+
+Added a very useful group_by example.
+------------------------------------------------------------------------
+r816 | dbalmain | 2008-02-09 09:26:20 +1100 (Sat, 09 Feb 2008) | 1 line
+
+Updated documentation for :filter_proc to indicate that you can return a Float to be used to modify the score.
+------------------------------------------------------------------------
+r815 | dbalmain | 2008-02-09 00:27:58 +1100 (Sat, 09 Feb 2008) | 1 line
+
+Made TypedRangeQuery the default range query when used from Ferret::Index::Index
+------------------------------------------------------------------------
+r814 | dbalmain | 2008-02-08 23:16:55 +1100 (Fri, 08 Feb 2008) | 1 line
+
+Made the TypedRangeQuery optional in the query parser
+------------------------------------------------------------------------
+r813 | dbalmain | 2008-02-08 23:12:13 +1100 (Fri, 08 Feb 2008) | 1 line
+
+Added TypedRangeQuery and TypedRangeFilter to the ruby bindings
+------------------------------------------------------------------------
+r812 | dbalmain | 2008-02-08 22:19:31 +1100 (Fri, 08 Feb 2008) | 1 line
+
+Added TypedRangeQuery so that you can do range queries with unpadded numbers
+------------------------------------------------------------------------
+r811 | dbalmain | 2008-02-08 16:22:06 +1100 (Fri, 08 Feb 2008) | 3 lines
+
+Whoops, quick fix. Had unnecessarily nested locks.
+
+
+------------------------------------------------------------------------
+r810 | dbalmain | 2008-02-08 16:17:33 +1100 (Fri, 08 Feb 2008) | 1 line
+
+Added patch for Ticket #340 which adds batch updating and deleting. Made significant modifications from the patch.
+------------------------------------------------------------------------
+r809 | dbalmain | 2008-02-08 13:49:07 +1100 (Fri, 08 Feb 2008) | 18 lines
+
+Added score filter. This enables you to filter the results and modify the score
+to change the sort order.
+
+For example to modify the scoring so that a document with todays date gets
+twice the score factor as a document 50 days ago and four times the score
+factor of a document 100 days ago (ie a half life of 50 days) you would do
+this;
+
+    fifty_day_half_life_filter = lambda do |doc, score, searcher|
+      days = (Date.today() - Date.parse(searcher[doc][:date])).to_i
+      1.0 / (2.0 ** (days.to_f / 50.0))
+    end
+
+    top_docs = @searcher.search(q, :filter_proc => fifty_day_half_life_filter)
+
+
+
+
+------------------------------------------------------------------------
+r808 | dbalmain | 2008-01-11 07:14:01 +1100 (Fri, 11 Jan 2008) | 1 line
+
+Changed unsigned long longs to f_u64 type to fix ticket #336
+------------------------------------------------------------------------
+r807 | dbalmain | 2008-01-11 07:12:40 +1100 (Fri, 11 Jan 2008) | 1 line
+
+Changed unsigned long longs to f_u64 type to fix ticket #336
+------------------------------------------------------------------------
+r806 | dbalmain | 2008-01-11 07:01:00 +1100 (Fri, 11 Jan 2008) | 1 line
+
+Minor comment correction
+------------------------------------------------------------------------
+r805 | dbalmain | 2007-12-12 10:28:23 +1100 (Wed, 12 Dec 2007) | 1 line
+
+Fixed Ticket #332. Added spaces so that code parses correctly.
+------------------------------------------------------------------------
+r804 | dbalmain | 2007-12-03 11:20:34 +1100 (Mon, 03 Dec 2007) | 1 line
+
+Added test for ticket #324
+------------------------------------------------------------------------
+r803 | dbalmain | 2007-12-03 11:12:55 +1100 (Mon, 03 Dec 2007) | 1 line
+
+Fixed Ferret::Index::Index#query_update for ticket #324. Was only updating a maximum of 10 records.
+------------------------------------------------------------------------

data/RELEASE_NOTES

@@ -0,0 +1,60 @@
+The most significant update in this release is that you can now alter the
+scoring, thereby altering the ordering of search results. A great application of
+this is that you can now change the weight of a document based on its age. In
+this example we have 365 day half life, ie a year-old document has half the
+weight of a new document and twice the weight of a 2-year-old document;
+
+    require 'ferret'
+
+    age_weight = lambda do |doc, score, searcher|
+      age = (Date.today - Date.parse(searcher[doc][:date])).to_i
+      1 / 2 ** (age.to_f/365)
+    end
+
+    index = Ferret::I.new
+
+    sales = [
+      { :artist => 'Giovanni Bellini',
+        :date => '2006-10-23',
+        :work => 'Transfiguration'
+      },
+      { :artist => 'Giovanni Bellini',
+        :date => '2008-01-05',
+        :work => 'Pesaro Altarpiece'
+      },
+      { :artist => 'Gentile Bellini',
+        :date => '2008-02-10',
+        :work => 'St. Dominic'
+      },
+    ].each {|doc| index << doc}
+
+    puts index.search('artist:(Giovanni Bellini)').to_s(:work)
+    # =>
+    #   TopDocs: total_hits = 3, max_score = 0.767351 [
+    #           0 "Transfiguration": 0.767351
+    #           1 "Pesaro Altarpiece": 0.767351
+    #           2 "St. Dominic": 0.129147
+    #   ]
+
+    puts index.search('artist:(Giovanni Bellini)',
+                      :filter_proc => age_weight).to_s(:work)
+    # =>
+    #   TopDocs: total_hits = 3, max_score = 0.718006 [
+    #           1 "Pesaro Altarpiece": 0.718006
+    #           0 "Transfiguration": 0.311937
+    #           2 "St. Dominic": 0.129147
+    #   ]
+
+You can also now write your own C extensions to filter the search results. You
+can see an example of this by downloading the source and looking at;
+
+    ferret_unzipped/ruby/examples/c_extensions/age_filter/
+
+Alternatively, you can view the code at;
+
+    http://ferret.davebalmain.com/trac/browser/trunk/ruby/examples/c_extensions?rev=828
+
+Also, good news for Norwegians, Romanians, Turks and Finns. We now have 3 new
+stemmers for Norwegian, Romanian and Turkish and 2 new stop-word lists for
+Finnish and Hungarian. Please try them out and let me know if there are any
+problems.

data/Rakefile

@@ -1,338 +1,443 @@
-$:. << 'lib'
-# Some parts of this Rakefile where taken from Jim Weirich's Rakefile for
-# Rake. Other parts where taken from the David Heinemeier Hansson's Rails
-# Rakefile. Both are under MIT-LICENSE. Thanks to both for their excellent
-# projects.
-
 require 'rake'
-require 'rake/testtask'
-require 'rake/rdoctask'
 require 'rake/clean'
-require 'ferret_version'
+require 'rake/gempackagetask'
+require 'rake/rdoctask'
+require 'rake/testtask'
 
-begin
-  require 'rubygems'
-  require 'rake/gempackagetask'
-rescue Exception
-  nil
-end
+$:. << 'lib'
+require 'ferret/version'
 
-CURRENT_VERSION = Ferret::VERSION
-if ENV['REL']
-  PKG_VERSION = ENV['REL']
-else
-  PKG_VERSION = CURRENT_VERSION
-end
 
-def announce(msg='')
+def say(msg='')
   STDERR.puts msg
 end
 
-EXT = "ferret_ext.so"
-EXT_SRC = FileList["../c/src/*.[c]", "../c/include/*.h",
-                   "../c/lib/libstemmer_c/src_c/*.[ch]",
-                   "../c/lib/libstemmer_c/runtime/*.[ch]",
-                   "../c/lib/libstemmer_c/libstemmer/*.[ch]",
-                   "../c/lib/libstemmer_c/include/libstemmer.h"]
-EXT_SRC.exclude('../**/ind.[ch]')
-
-EXT_SRC_DEST = EXT_SRC.map {|fn| File.join("ext", File.basename(fn))}
-SRC = (FileList["ext/*.[ch]"] + EXT_SRC_DEST).uniq
-
-CLEAN.include(FileList['**/*.o', '**/*.obj', 'InstalledFiles',
-                       '.config', 'ext/cferret.c'])
-CLOBBER.include(FileList['**/*.so'], 'ext/Makefile', EXT_SRC_DEST)
-POLISH = Rake::FileList.new.include(FileList['**/*.so'], 'ext/Makefile')
-
-desc "Clean specifically for the release."
-task :polish => [:clean] do
-  POLISH.each { |fn| rm_r fn rescue nil }
+def prompt(msg)
+  STDERR.print "#{msg} [Yna]: "
+  while true
+    case STDIN.gets.chomp!
+    when /^(y(es)?)?$/i then return true
+    when /^no?$/i then return false
+    when /^a(bort)?$/i then fail('aborted')
+    else
+      STDERR.print "Sorry, I don't understand. Please type y, n or a: "
+    end
+  end
 end
 
-desc "Run tests with Valgrind"
-task :valgrind do
-  sh "valgrind --gen-suppressions=yes --suppressions=ferret_valgrind.supp " +
-     "--leak-check=yes --show-reachable=yes -v ruby test/test_all.rb"
-  #sh "valgrind --suppressions=ferret_valgrind.supp " +
-  #   "--leak-check=yes --show-reachable=yes -v ruby test/unit/index/tc_index_reader.rb"
-  #valgrind --gen-suppressions=yes --suppressions=ferret_valgrind.supp --leak-check=yes --show-reachable=yes -v ruby test/test_all.rb
-end
+windows = (RUBY_PLATFORM =~ /win32|cygwin/) rescue nil
+SUDO = windows ? "" : "sudo "
 
-task :default => :test_all
-#task :default => :ext do
+
+task :default => 'test:unit'
+#task :default => :build do
 #  sh "ruby test/unit/index/tc_index.rb"
 #end
 
-desc "Run all tests"
-task :test_all => [ :test_units ]
-
-desc "Generate API documentation"
-task :doc => [ :appdoc ]
-
-desc "run unit tests in test/unit"
-Rake::TestTask.new("test_units" => :ext) do |t|
-  t.libs << "test/unit"
-  t.pattern = 'test/unit/t[cs]_*.rb'
-  #t.pattern = 'test/unit/search/tc_index_searcher.rb'
-  t.verbose = true
-end
-
-desc "Generate documentation for the application"
-rd = Rake::RDocTask.new("appdoc") do |rdoc|
-  rdoc.rdoc_dir = 'doc/api'
-  rdoc.title    = "Ferret Search Library Documentation"
-  rdoc.options << '--line-numbers'
-  rdoc.options << '--inline-source'
-  rdoc.options << '--charset=utf-8'
-  rdoc.rdoc_files.include('README')
-  rdoc.rdoc_files.include('TODO')
-  rdoc.rdoc_files.include('TUTORIAL')
-  rdoc.rdoc_files.include('MIT-LICENSE')
-  rdoc.rdoc_files.include('lib/**/*.rb')
-  rdoc.rdoc_files.include('ext/r_*.c')
-  rdoc.rdoc_files.include('ext/ferret.c')
-end
+  BZLIB_SRC = FileList["../c/lib/bzlib/*.h"] +
+              FileList["../c/lib/bzlib/*.c"].map do |fn|
+                fn.gsub(%r{/([^/]*.c)}, '/BZ_\1')
+              end
+##############################################################################
+# Building
+##############################################################################
+
+task :build => 'build:compile'
+namespace :build do
+  EXT = "ferret_ext.so"
+  # Note: libstemmer.[h] is necessary so that the file isn't included when it
+  # doesn't exist. It needs to have one regular expression element.
+  EXT_SRC = FileList["../c/src/*.[ch]", "../c/include/*.h",
+                     "../c/lib/bzlib/*.[ch]",
+                     "../c/lib/libstemmer_c/src_c/*.[ch]",
+                     "../c/lib/libstemmer_c/runtime/*.[ch]",
+                     "../c/lib/libstemmer_c/libstemmer/*.[ch]",
+                     "../c/lib/libstemmer_c/include/libstemmer.[h]"]
+  EXT_SRC.exclude('../c/**/ind.[ch]',
+                  '../c/**/symbol.[ch]',
+                  '../c/include/threading.h',
+                  '../c/include/scanner.h',
+                  '../c/include/internal.h',
+                  '../c/src/lang.c',
+                  '../c/include/lang.h')
+
+  EXT_SRC_MAP = {}
+  EXT_SRC_DEST = EXT_SRC.map do |fn|
+    ext_fn = File.join("ext", File.basename(fn))
+    if fn =~ /.c$/ and fn =~ /(bzlib|stemmer)/
+      prefix = $1.upcase
+      ext_fn.gsub!(/ext\//, "ext/#{prefix}_")
+    end
+    EXT_SRC_MAP[fn] = ext_fn
+  end
+  SRC = FileList["ext/*.[ch]", EXT_SRC_DEST, 'ext/internal.h'].uniq
+
+  CLEAN.include   ['**/*.o', '**/*.obj', '.config', 'ext/cferret.c']
+  CLOBBER.include ['doc/api', 'ext/*.so', 'ext/Makefile',
+                   'ext/internal.h', EXT_SRC_DEST]
+
+  # The following block creates file tasks for all of the c files. They
+  # belong in the ../c directory in source the working copy and they need
+  # to be linked to in the ext directory
+  EXT_SRC.each do |fn|
+    dest_fn = EXT_SRC_MAP[fn]
+    # prepend lib files to avoid conflicts
+    file dest_fn => fn do |t|
+      ln_sf File.expand_path(fn), File.expand_path(dest_fn)
+
+      if fn =~ /stemmer/
+        # flatten the directory structure for lib_stemmer
+        open(dest_fn) do |in_f|
+          open(dest_fn + ".out", "w") do |out_f|
+            in_f.each do |line|
+              out_f.write(line.sub(/(#include ["<])[.a-z_\/]*\//, '\1'))
+            end
+          end
+        end
+        mv dest_fn + ".out", dest_fn
+      end
+    end
+  end if File.exists?("../c")
 
-EXT_SRC.each do |fn|
-  dest_fn = File.join("ext", File.basename(fn))
-  file dest_fn => fn do |t|
-    begin
-      raise "copy for release" if ENV["REL"]
-      ln_s File.join("..", fn), dest_fn
-    rescue Exception => e
-      cp File.expand_path(fn), dest_fn
+  file 'ext/internal.h' => '../c/include/internal.h' do
+    File.open('ext/internal.h', 'w') do |f|
+      File.readlines('../c/include/internal.h').each do |l|
+        next if l =~ /ALLOC/ and l !~ /ZERO|MP_/
+        f.puts(l)
+      end
     end
+  end
 
-    if fn =~ /stemmer/
-      # flatten the directory structure for lib_stemmer
-      open(dest_fn) do |in_f|
-        open(dest_fn + ".out", "w") do |out_f|
-          in_f.each {|line| out_f.write(line.sub(/(#include ["<])[.a-z_\/]*\//) {"#{$1}"})}
+  desc "Build the extension (ferret_ext.so). You'll need a C compiler and Make."
+  task :compile => ["ext/#{EXT}"] + SRC
+
+  file "ext/#{EXT}" => "ext/Makefile" do
+    cd "ext"
+    if windows and ENV['make'].nil?
+      begin
+        sh "nmake"
+      rescue Exception => e
+        path = ':\Program Files\Microsoft Visual Studio\VC98\Bin\VCVARS32.BAT'
+        if File.exists? "f#{path}"
+          sh "f#{path}"
+        elsif File.exists? "c#{path}"
+          sh "c#{path}"
+        else
+          say
+          say "***************************************************************"
+          say "You need to have Visual C++ 6 to build Ferret on Windows."
+          say "If you have it installed, you may need to run;"
+          say ' C:\Program Files\Microsoft Visual Studio\VC98\Bin\VCVARS32.BAT'
+          say "***************************************************************"
+          say
+          raise e
         end
+        sh "nmake"
       end
-      mv dest_fn + ".out", dest_fn
+    else
+      sh "make"
     end
+    cd ".."
   end
-end if File.exists?("../c")
-
-desc "Build the extension"
-task :ext => ["ext/#{EXT}"] + SRC do
-  rm_f 'ext/mem_pool.*'
-  rm_f 'ext/defines.h'
-end
 
-file "ext/#{EXT}" => ["ext/Makefile"] do
-  cp "ext/inc/lang.h", "ext/lang.h"
-  cp "ext/inc/threading.h", "ext/threading.h"
-  cd "ext"
-  if (/mswin/ =~ RUBY_PLATFORM) and ENV['make'].nil?
-    begin
-      sh "nmake"
-    rescue Exception => e
-      puts
-      puts "**********************************************************************"
-      puts "You may need to call VCVARS32.BAT to set the environment variables."
-      puts '  "f:\Program Files\Microsoft Visual Studio\VC98\Bin\VCVARS32.BAT"'
-      puts "**********************************************************************"
-      puts
-      raise e
-    end
-  else
-    sh "make"
+  file "ext/Makefile" => SRC do
+    cd "ext"
+    ruby "extconf.rb"
+    cd ".."
   end
-  cd ".."
 end
 
-file "ext/lang.h" => ["ext/inc/lang.h"] do
-  rm_f "ext/lang.h"
-  cp "ext/inc/lang.h", "ext/lang.h"
-end
+##############################################################################
+# Testing
+##############################################################################
+
+task :test => 'test:units'
+namespace :test do
+  desc "Run tests with Valgrind"
+  task :valgrind do
+    sh "valgrind --suppressions=ferret_valgrind.supp " +
+       "--leak-check=yes --show-reachable=yes " +
+       "-v ruby test/unit/index/tc_index_reader.rb"
+  end
 
-file "ext/threading.h" => ["ext/inc/threading.h"] do
-  rm_f "ext/threading.h"
-  cp "ext/inc/threading.h", "ext/threading.h"
-end
+  desc "Run all tests"
+  task :all => [ :units ]
 
-file "ext/Makefile" => SRC do
-  cd "ext"
-  `ruby extconf.rb`
-  cd ".."
+  desc "run unit tests in test/unit"
+  Rake::TestTask.new("units" => :build) do |t|
+    t.libs << "test/unit"
+    t.pattern = 'test/unit/t[cs]_*.rb'
+    t.verbose = true
+  end
+  task :unit => :units
+
+  desc "run tests using locally installed gem"
+  Rake::TestTask.new("installed") do |t|
+    t.libs << "test/unit"
+    t.ruby_opts << '-rtest/test_installed'
+    t.pattern = 'test/unit/t[cs]_*.rb'
+    t.verbose = true
+  end
 end
 
-# Make Parsers ---------------------------------------------------------------
+##############################################################################
+# Documentation
+##############################################################################
 
-RACC_SRC = FileList["lib/**/*.y"]
-RACC_OUT = RACC_SRC.collect { |fn| fn.sub(/\.y$/, '.tab.rb') }
+desc "Generate API documentation"
+task :doc => 'doc:rdoc'
+namespace :doc do
+  if allison = Gem.cache.find_name('allison').last
+    allison_template = File.join(allison.full_gem_path, 'lib/allison.rb')
+  end
+  desc "Generate documentation for the application"
+  $rd = Rake::RDocTask.new do |rdoc|
+    rdoc.rdoc_dir = 'doc/api'
+    rdoc.title    = "Ferret Search Library Documentation"
+    rdoc.options << '--line-numbers'
+    rdoc.options << '--inline-source'
+    rdoc.options << '--charset=utf-8'
+    rdoc.template = allison_template if allison_template
+    rdoc.rdoc_files.include('README')
+    rdoc.rdoc_files.include('TODO')
+    rdoc.rdoc_files.include('TUTORIAL')
+    rdoc.rdoc_files.include('MIT-LICENSE')
+    rdoc.rdoc_files.include('lib/**/*.rb')
+    rdoc.rdoc_files.include('ext/r_*.c')
+    rdoc.rdoc_files.include('ext/ferret.c')
+  end
 
-task :parsers => RACC_OUT
-rule(/\.tab\.rb$/ => [proc {|tn| tn.sub(/\.tab\.rb$/, '.y')}]) do |t|
-  sh "racc #{t.source}" 
+  desc "Look for TODO and FIXME tags in the code"
+  task :todo do
+    FileList['**/*.rb', 'ext/*.[ch]'].egrep /[#*].*(FIXME|TODO|TBD)/i
+  end
 end
 
-# Create Packages ------------------------------------------------------------
+##############################################################################
+# Packaging and Installing
+##############################################################################
 
 PKG_FILES = FileList[
   'setup.rb',
   '[-A-Z]*',
-  'ext/**/*.[ch]', 
-  'lib/**/*.rb', 
-  'lib/**/*.rhtml', 
-  'lib/**/*.css', 
-  'lib/**/*.js', 
+  'lib/**/*.rb',
+  'lib/**/*.rhtml',
+  'lib/**/*.css',
+  'lib/**/*.js',
   'test/**/*.rb',
   'test/**/wordfile',
   'rake_utils/**/*.rb',
-  'Rakefile'
+  'Rakefile',
+  SRC
 ]
-PKG_FILES.exclude('**/*.o')
-PKG_FILES.exclude('**/Makefile')
-PKG_FILES.exclude('ext/ferret_ext.so')
-
-
-if ! defined?(Gem)
-  puts "Package Target requires RubyGEMs"
-else
-  spec = Gem::Specification.new do |s|
-    
-    #### Basic information.
-    s.name = 'ferret'
-    s.version = PKG_VERSION
-    s.summary = "Ruby indexing library."
-    s.description = <<-EOF
-      Ferret is a port of the Java Lucene project. It is a powerful
-      indexing and search library.
-    EOF
 
-    #### Dependencies and requirements.
-    s.add_dependency('rake')
-    s.files = PKG_FILES.to_a
-    s.extensions << "ext/extconf.rb"
-    s.require_path = 'lib'
-    s.autorequire = 'ferret'
-    s.bindir = 'bin'
-    s.executables = ['ferret-browser']
-    s.default_executable = 'ferret-browser'
-
-    #### Author and project details.
-    s.author = "David Balmain"
-    s.email = "dbalmain@gmail.com"
-    s.homepage = "http://ferret.davebalmain.com/trac"
-    s.rubyforge_project = "ferret"
-
-    s.has_rdoc = true
-    s.extra_rdoc_files = rd.rdoc_files.reject { |fn| fn =~ /\.rb$/ }.to_a
-    s.rdoc_options <<
-      '--title' <<  'Ferret -- Ruby Indexer' <<
-      '--main' << 'README' << '--line-numbers' <<
-      'TUTORIAL' << 'TODO'
-
-    if RUBY_PLATFORM =~ /mswin/
-      s.files = PKG_FILES.to_a + ["ext/#{EXT}"]
-      s.extensions.clear
-      s.platform = Gem::Platform::WIN32
-    end
+spec = Gem::Specification.new do |s|
+
+  #### Basic information.
+  s.name = 'ferret'
+  s.version = Ferret::VERSION
+  s.summary = "Ruby indexing library."
+  s.description = "Ferret is a super fast, highly configurable search library."
+
+  #### Dependencies and requirements.
+  s.add_dependency('rake')
+  s.files = PKG_FILES.to_a
+  s.extensions << "ext/extconf.rb"
+  s.require_path = 'lib'
+  s.bindir = 'bin'
+  s.executables = ['ferret-browser']
+  s.default_executable = 'ferret-browser'
+
+  #### Author and project details.
+  s.author = "David Balmain"
+  s.email = "dbalmain@gmail.com"
+  #s.homepage = "http://ferret.davebalmain.com/trac"
+  s.homepage = "http://github.com/jkraemer/ferret"
+  s.rubyforge_project = "ferret"
+
+  s.has_rdoc = true
+  s.extra_rdoc_files = $rd.rdoc_files.reject { |fn| fn =~ /\.rb$/ }.to_a
+  s.rdoc_options <<
+    '--title' <<  'Ferret -- Ruby Search Library' <<
+    '--main' << 'README' << '--line-numbers' <<
+    'TUTORIAL' << 'TODO'
+
+  key_file = File.expand_path('~/.gem/gem-private_key.pem')
+  key_file = nil unless File.exists?(key_file)
+  cert_file = File.expand_path('~/.gem/gem-public_cert.pem')
+  cert_file = nil unless File.exists?(cert_file)
+  if key_file and cert_file
+    s.signing_key = key_file
+    s.cert_chain  = cert_file
   end
 
-  package_task = Rake::GemPackageTask.new(spec) do |pkg|
-    unless RUBY_PLATFORM =~ /mswin/
-      pkg.need_zip = true
-      pkg.need_tar = true
-    end
+  if windows
+    s.files = PKG_FILES.to_a + ["ext/#{EXT}"]
+    s.extensions.clear
+    s.platform = Gem::Platform::WIN32
+  else
+    s.platform = Gem::Platform::RUBY
   end
 end
 
-# Support Tasks ------------------------------------------------------
+package_task = Rake::GemPackageTask.new(spec) do |pkg|
+  unless windows
+    pkg.need_zip = true
+    pkg.need_tar = true
+  end
+end
 
-desc "Look for TODO and FIXME tags in the code"
-task :todo do
-  FileList['**/*.rb'].egrep /#.*(FIXME|TODO|TBD)/
+desc "Run :gem and install the resulting gem"
+task :install => :gem do
+  sh "#{SUDO}gem install pkg/ferret-#{Ferret::VERSION}.gem --no-rdoc --no-ri -l"
 end
-# --------------------------------------------------------------------
-# Creating a release
-
-desc "Make a new release"
-task :release => [
-  :prerelease,
-  :polish,
-  :test_all,
-  :update_version,
-  :package,
-  :tag] do
-  announce 
-  announce "**************************************************************"
-  announce "* Release #{PKG_VERSION} Complete."
-  announce "* Packages ready to upload."
-  announce "**************************************************************"
-  announce 
+
+desc "Run :clobber and uninstall the .gem"
+task :uninstall => :clobber do
+  sh "#{SUDO}gem uninstall ferret"
 end
 
-# Validate that everything is ready to go for a release.
-task :prerelease do
-  announce 
-  announce "**************************************************************"
-  announce "* Making RubyGem Release #{PKG_VERSION}"
-  announce "* (current version #{CURRENT_VERSION})"
-  announce "**************************************************************"
-  announce  
-
-  # Is a release number supplied?
-  unless ENV['REL']
-    fail "Usage: rake release REL=x.y.z [REUSE=tag_suffix]"
-  end
+desc "Same as :install but you must be rootgem"
+task :root_install => :gem do
+  sh "gem install pkg/ferret-#{Ferret::VERSION}.gem --no-rdoc --no-ri -l"
+end
 
-  # Is the release different than the current release.
-  # (or is REUSE set?)
-  if PKG_VERSION == CURRENT_VERSION && ! ENV['REUSE']
-    fail "Current version is #{PKG_VERSION}, must specify REUSE=tag_suffix to reuse version"
-  end
+desc "Same as :uninstall but you must be root"
+task :root_uninstall => :clobber do
+  sh "gem uninstall ferret"
+end
 
-  # Are all source files checked in?
-  data = `svn -q --ignore-externals status`
-  unless data =~ /^$/
-    fail "'svn -q status' is not clean ... do you have unchecked-in files?"
-  end
-  
-  announce "No outstanding checkins found ... OK"
+def list_changes_since_last_release
+  tag_listing = `svn list svn://davebalmain.com/ferret/tags`
+  last_tag = tag_listing.split("\n").last
+  log = `svn log --stop-on-copy svn://davebalmain.com/ferret/tags/#{last_tag}`
+  first_log = log.split(/-------+/)[-2]
+  last_revision = /^r(\d+)\s+\|/.match(first_log)[1]
+  `svn log .. -rHEAD:#{last_revision}`
 end
 
-def reversion(fn)
-  open(fn) do |ferret_in|
-    open(fn + ".new", "w") do |ferret_out|
-      ferret_in.each do |line|
-        if line =~ /^  VERSION\s*=\s*/
-          ferret_out.puts "  VERSION = '#{PKG_VERSION}'"
-        else
-          ferret_out.puts line
+desc "List changes since last release"
+task :changes do
+  puts list_changes_since_last_release
+end
+
+if ENV['FERRET_DEV']
+  ##############################################################################
+  # Releasing
+  ##############################################################################
+
+  desc "Generate and upload a new release"
+  task :release => 'release:release'
+  namespace :release do
+    task :release => [:status_check, 'test:all', :package, :tag] do
+      say
+      say "**************************************************************"
+      say "* Release #{Ferret::VERSION} Complete."
+      say "* Packages ready to upload."
+      say "**************************************************************"
+      say
+      reversion("lib/ferret/version.rb")
+    end
+
+    # Validate that everything is ready to go for a release.
+    task :status_check do
+      # Are all source files checked in?
+      unless `svn -q --ignore-externals status` =~ /^$/
+        fail "'svn -q status' is not clean ... do you have unchecked-in files?"
+      end
+
+      say "No outstanding checkins found ... OK"
+    end
+
+    def reversion(fn)
+      new_version = nil
+      begin
+        print "Ferret is currently at #{Ferret::VERSION}. What version now? "
+        new_version = STDIN.gets.chomp!
+      end until prompt("Change to version #{new_version}?")
+
+      if ENV['RELTEST']
+        say "Would change the version in lib/ferret/version.rb from"
+        say "    #{Ferret::VERSION} => #{new_version}"
+        say "and then commit the changes with the command"
+        say "   svn ci -m \"Updated to version #{new_version}\" " +
+            "lib/ferret/version.rb"
+      else
+        open(fn) do |ferret_in|
+          open(fn + ".new", "w") do |ferret_out|
+            ferret_in.each do |line|
+              if line =~ /^  VERSION\s*=\s*/
+                ferret_out.puts "  VERSION = '#{new_version}'"
+              else
+                ferret_out.puts line
+              end
+            end
+          end
         end
+        mv fn + ".new", fn
+        sh %{svn ci -m "Updated to version #{new_version}" lib/ferret/version.rb}
       end
     end
-  end
-  mv fn + ".new", fn
-end
 
-task :update_version => [:prerelease] do
-  if PKG_VERSION == CURRENT_VERSION
-    announce "No version change ... skipping version update"
-  else
-    announce "Updating Ferret version to #{PKG_VERSION}"
-    reversion("lib/ferret_version.rb")
-    if ENV['RELTEST']
-      announce "Release Task Testing, skipping commiting of new version"
-    else
-      sh %{svn ci -m "Updated to version #{PKG_VERSION}" lib/ferret_version.rb}
+    # Tag all the SVN files with the latest release number
+    task :tag => :status_check do
+      reltag = "REL-#{Ferret::VERSION}"
+      say "Tagging SVN with [#{reltag}]"
+      if ENV['RELTEST']
+        say "Release Task Testing, skipping SVN tagging. Would do;"
+        say %{svn copy -m "creating release #{reltag}" svn://www.davebalmain.com/ferret/trunk svn://www.davebalmain.com/ferret/tags/#{reltag}}
+      else
+        sh %{svn copy -m "creating release #{reltag}" svn://www.davebalmain.com/ferret/trunk svn://www.davebalmain.com/ferret/tags/#{reltag}}
+      end
     end
+
   end
-end
 
-desc "Tag all the SVN files with the latest release number (REL=x.y.z)"
-task :tag => [:prerelease] do
-  reltag = "REL-#{PKG_VERSION}"
-  reltag << ENV['REUSE'] if ENV['REUSE']
-  announce "Tagging SVN with [#{reltag}]"
-  if ENV['RELTEST']
-    announce "Release Task Testing, skipping SVN tagging. Would do the following;"
-    announce %{svn copy -m "creating release #{reltag}" svn://www.davebalmain.com/ferret/trunk svn://www.davebalmain.com/ferret/tags/#{reltag}}
-  else
-    sh %{svn copy -m "creating release #{reltag}" svn://www.davebalmain.com/ferret/trunk svn://www.davebalmain.com/ferret/tags/#{reltag}}
+  ##############################################################################
+  # Publishing
+  ##############################################################################
+
+  namespace :publish do
+    PUBLISH_PROMPT = <<-EOF
+      Make sure you updated RELEASE_NOTES and RELEASE_CHANGES and that the
+      package exists. Are you sure you want to continue?
+    EOF
+    desc "Publish gem on rubyforge for download. Will only do the linux version"
+    task :release do
+      exit unless prompt(PUBLISH_PROMPT)
+      require 'rubyforge'
+      require 'rake/contrib/rubyforgepublisher'
+      version = Ferret::VERSION
+
+      packages = %w(gem tgz zip).map {|ext| "pkg/ferret-#{version}.#{ext}"}
+
+      rubyforge = RubyForge.new
+      rubyforge.login
+      rubyforge.add_release('ferret', 'ferret',
+                            "ferret-#{version}", *packages)
+    end
+
+    desc "Publish the documentation"
+    task :docs => 'doc:rdoc' do
+      sh %{rsync -rzv --delete  -e 'ssh -p 8900' doc/api/ davebalmain.com:/var/www/ferret/api}
+    end
+
+    desc "Publish the documentation and release"
+    task :all => [:doc, :release]
   end
 end
+
+
+
+
+#
+# In case I ever need to add another racc parser, here's how
+#
+# # Make Parsers ---------------------------------------------------------------
+#
+# RACC_SRC = FileList["lib/**/*.y"]
+#
+# task :parsers => RACC_OUT
+# rule(/\.tab\.rb$/ => [proc {|tn| tn.sub(/\.tab\.rb$/, '.y')}]) do |t|
+#   sh "racc #{t.source}"
+# end