words-wordnet 0.4.6

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Roja Buck
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,169 @@
+# Words - A fast, easy to use interface to WordNet® with cross ruby distribution compatibility. #
+
+## About ##
+
+Words implements a fast interface to [Wordnet®](http://wordnet.princeton.edu) which provides both a pure ruby and an FFI powered backend over the same easy-to-use API. The FFI backend makes use of [Tokyo Cabinet](http://1978th.net/tokyocabinet/) and the FFI interface, [rufus-tokyo](http://github.com/jmettraux/rufus-tokyo), to provide cross ruby distribution compatibility and blistering speed. The pure ruby interface operates on a special ruby optimised index along with the basic dictionary files provided by WordNet®. I have attempted to provide ease of use in the form of a simple yet powerful api and installation is a sintch!
+
+* Version 0.2 Introduced Pure Ruby Backend
+* Version 0.3 Introduced Evocation Support (see examples & below) as developed by the [Wordnet® Evocation Project](http://wordnet.cs.princeton.edu/downloads/evocation/release-0.4/README.TXT) 
+* Version 0.4 Substantial performance increase in pure mode (now faster at some things than the tokyo backend) and simplification of use! Full refactoring. Move to RSpec for testing. API CHANGES: Words::Words -> Words::Wordnet, close -> close!, connected -> connected? and evocations_enabled? -> evocations?
+
+Documentation: [Yardoc Live](http://yardoc.org/docs/roja-words)
+
+## Pre-Installation ##
+
+First ensure you have a copy of the wordnet data files. This is generally available from your Linux/OSX package manager:
+
+    #Ubuntu
+    sudo apt-get install wordnet-base
+    
+    #Fedora/RHL
+    sudo yum update wordnet
+
+    #OSX HomeBrew
+    sudo brew install wordnet
+    
+    #MacPorts
+    sudo port install wordnet
+    
+or you can simply download and install (Unix/OSX):
+
+	wget http://wordnetcode.princeton.edu/3.0/WNdb-3.0.tar.gz
+	sudo mkdir /usr/local/share/wordnet
+	sudo tar -C /usr/local/share/wordnet/ -xzf WNdb-3.0.tar.gz
+	
+or (Windows)
+
+	Download http://wordnetcode.princeton.edu/3.0/WNdb-3.0.tar.gz
+	Unzip
+
+        # due to the way windows tends not to have a folder for things you will
+        # have to specify the location of the wordnet files when using the gem
+
+## For Tokyo Backend Only ##
+
+Unless you want to use the tokyo backend you are now ready to install Words (and build the data if you want extras), otherwise if you want to use the tokyo backend (FAST!) you will also need [Tokyo Cabinet](http://1978th.net/tokyocabinet/) installed. It should be nice and easy... something like:
+
+    # osx users should, if ports is installed, simply do
+    sudo ports install tokyocabinet
+
+    # otherwise the best route is from source
+    wget http://1978th.net/tokyocabinet/tokyocabinet-1.4.41.tar.gz
+    tar -xzf tokyocabinet-1.4.41.tar.gz
+    cd tokyocabinet-1.4.41/
+    ./configure
+    make
+    sudo make install
+    
+## GEM Installation ##
+
+After this it should be just a gem to install. For those of you with old rubygems versions first:
+
+    gem install gemcutter  # These two steps are only necessary if you haven't
+    gem tumble             # yet installed the gemcutter tools
+    
+Otherwise and after it's simply:
+
+	gem install words
+	
+Then your ready to rock and roll. :)
+
+## Build Data ##
+
+If all you want to do is use wordnet in it's standard form you don't have to do any databuilding and can skip this section. If however you either
+want to take advantage of evocations ([Wordnet® Evocation Project](http://wordnet.cs.princeton.edu/downloads/evocation/release-0.4/README.TXT)) or want to use the tokyo backend, read on!
+
+To build the wordnet dataset file yourself, from the original wordnet files, you can use the bundled "build_wordnet" command
+
+	build_wordnet -h # this will give you the usage information & additional options/features
+	
+	# this would attempt to build the tokyo backend data locating the original wordnet
+        # files through a search...
+	sudo build_wordnet --build-tokyo
+	
+	# this would attempt to build the tokyo backend locating the original wordnet files
+        # through a search with the addition of evocation support...
+	sudo build_wordnet --build-tokyo-with-evocations
+	
+	# this would attempt to build evocation support for the pure backend
+        # (remember no dataset needs to be built to use wordnet with the pure backend)
+	sudo build_wordnet --build-pure-evocations
+
+## Usage ##
+
+Heres a few little examples of using words within your programs.
+
+    require 'rubygems'
+    require 'words'
+    
+    data = Words::Wordnet.new # or: data = Words::Wordnet.new(:tokyo) for the tokyo backend
+
+    # to specify a wordnet path Words::Words.new(:pure, '/path/to/wordnet')
+    # to specify the tokyo dataset Words::Words.new(:pure, :search, '/path/to/data.tct')
+
+    # play with connections
+    data.connected? # => true
+    data.close!
+    data.connected? # => false
+    data.open!
+    data.connected? # => true
+    data.connection_type # => :pure or :tokyo depending...
+
+    # locate a word
+    lemma = data.find("bat")
+    
+    lemma.to_s # => bat, noun/verb
+    lemma.available_pos.inspect # => [:noun, :verb]
+    
+    lemma.synsets(:noun) # => array of synsets which represent nouns of the lemma bat
+    # or
+    lemma.nouns # => array of synsets which represent nouns of the lemma bat
+    lemma.noun_ids # => array of synsets ids which represent nouns of the lemma bat
+    lemma.verbs? #=> true
+    
+    # specify a sense
+    sense = lemma.nouns.last
+    sense2 = lemma.nouns[2]
+    
+    sense.gloss # => a club used for hitting a ball in various games
+    sense2.words # => ["cricket bat", "bat"]
+    sense2.lexical_description # => a description of the lexical meaning of the synset
+    sense.relations.first # => "Semantic hypernym relation between n02806379 and n03053474"
+
+    sense.relations(:hyponym) # => Array of hyponyms associated with the sense
+    # or
+    sense.hyponyms # => Array of hyponyms associated with the sense
+    sense.hyponyms? # => true
+    
+    sense.relations.first.is_semantic? # => true
+    sense.relations.first.source_word # => nil
+    sense.relations.first.destination # => the synset of n03053474
+    
+    sense.derivationally_related_forms.first.is_semantic? # => false
+    sense.derivationally_related_forms.first.source_word # => "bat"
+    sense.derivationally_related_forms.first.destination_word # => "bat"
+    sense.derivationally_related_forms.first.destination # => the synset of v01413191
+
+    if data.evocations? # check for evocation support
+       data.find("broadcast").senses.first.evocations # => sense relevant evocations
+       data.find("broadcast").senses.first.evocations[1] # => the evocation at index 1
+       data.find("broadcast").senses.first.evocations[1][:destination].words # => synset
+    end
+
+    data.close!
+    
+These and more examples are available from within the examples.rb file!        
+
+## Note on Patches/Pull Requests ##
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+## Copyright ##
+
+Copyright (c) 2010 Roja Buck. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,54 @@
+# coding: utf-8
+require 'rubygems'
+require 'rake'
+
+begin
+    require 'jeweler'
+    Jeweler::Tasks.new do |gem|
+	gem.name = "words-wordnet"
+	gem.summary = %Q{A Fast & Easy to use interface to WordNet with cross ruby distribution compatibility.}
+	gem.description = %Q{Words, with both pure ruby & tokyo-cabinate backends, implements a fast interface to Wordnet over the same easy-to-use API. The FFI backend makes use of Tokyo Cabinet and the FFI interface, rufus-tokyo, to provide cross ruby distribution compatability and blistering speed. The pure ruby interface operates on a special ruby optimised index along with the basic dictionary files provided by WordNet. I have attempted to provide ease of use in the form of a simple yet powerful api and installation is a sintch!}
+	gem.email = "roja@arbia.co.uk"
+	gem.homepage = "http://github.com/roja/words"
+	gem.authors = ["Roja Buck"]
+	gem.executables = [ "build_wordnet" ]
+	gem.default_executable = "build_wordnet"
+	gem.rubyforge_project = 'words'
+	gem.add_development_dependency "rspec", ">= 2.11.0"
+    end
+    Jeweler::GemcutterTasks.new
+rescue LoadError
+    puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+    spec.pattern = 'spec/**/*_spec.rb'
+end
+
+begin
+    require 'rcov'
+    RSpec::Core::RakeTask.new(:rcov) do |spec|
+	spec.libs << 'lib' << 'spec'
+	spec.pattern = 'spec/**/*_spec.rb'
+	spec.rcov = true
+    end
+rescue LoadError
+    task :rcov do
+	abort "RCov is not available. In order to run rcov, you must: sudo gem install rcov"
+    end
+end
+
+task :spec => :check_dependencies
+
+task :default => :spec
+
+require 'rdoc/task'
+Rake::RDocTask.new do |rdoc|
+    version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+    rdoc.rdoc_dir = 'rdoc'
+    rdoc.title = "test #{version}"
+    rdoc.rdoc_files.include('README*')
+    rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.4.6

data/bin/build_wordnet

@@ -0,0 +1,177 @@
+#!/usr/bin/env ruby
+
+# std includes
+require 'pathname'
+
+# gem includes
+require 'rubygems'
+
+# standard library includes
+#require 'trollop'
+require 'zlib'
+require 'net/http'
+require 'optparse'
+require 'pp'
+
+# local includes
+require File.join(File.dirname(__FILE__), '..', 'lib', 'words.rb')
+
+POS_FILE_TYPES = %w{ adj adv noun verb }
+POS_FILE_TYPE_TO_SHORT = { 'adj' => 'a', 'adv' => 'r', 'noun' => 'n', 'verb' => 'v' }
+
+puts "Words Dataset Constructor 2010 (c) Roja Buck"
+
+opts = { :quiet => false, :build_tokyo => false, :build_tokyo_with_evocations => false, :build_pure_evocations => false, :wordnet => 'Search...' }
+
+optparse = OptionParser.new do|option|
+
+    option.on( '-q', '--quiet', "Don't output verbose program detail. (Default: false)" ) do
+	opts[:quiet] = true
+    end
+
+    option.on( '-w', '--wordnet FILE', "Location of the wordnet dictionary directory. (Default: Search)" ) do|f|
+	opts[:wordnet] = f
+    end
+
+    option.on( '-t', '--build-tokyo', "Build the tokyo wordnet dataset? (Default: false)" ) do
+	opts[:build_tokyo] = true
+    end
+
+    option.on( '-x', '--build-tokyo-with-evocations', "Build the tokyo dataset with the similarity dataset based on the wordnet evocation project? (Default: false) NOTE: requires internet connection." ) do
+	opts[:build_tokyo_with_evocations] = true
+    end
+
+    option.on( '-e', '--build-pure-evocations', "Build the similarity dataset based on the wordnet evocation project for use with the pure words mode. (Default: false) NOTE: requires internet connection." ) do
+	opts[:build_pure_evocations] = true
+    end
+
+    option.on( '-h', '--help', 'Display this screen' ) do
+	puts option
+	exit
+    end
+
+end
+
+optparse.parse!
+
+if !opts[:build_tokyo] && !opts[:build_tokyo_with_evocations] && !opts[:build_pure_evocations]
+    puts "ERROR: You need to specify at least one dataset you want to build."
+    exit
+end
+puts "Verbose mode enabled" if (VERBOSE = !opts[:quiet])
+
+require 'rufus-tokyo' if opts[:build_tokyo] || opts[:build_tokyo_with_evocations]
+
+gem_path = Pathname.new "#{File.dirname(__FILE__)}/.."
+abort "Ensure you run the command using sudo or as a Superuser / Administrator" unless gem_path.writable?
+data_path = gem_path + "data/"
+data_path.mkpath
+
+wordnet_dir = nil
+if opts[:wordnet] == "Search..."
+    wordnet_dir = Words::Wordnet.locate_wordnet :search
+    abort( "Unable to locate wordnet dictionary. To specify check --help." ) if wordnet_dir.nil?
+else
+    wordnet_dir = Words::Wordnet.locate_wordnet opts[:wordnet]
+    abort( "Unable to locate wordnet dictionary in directory #{opts[:wordnet]}. Please check and try again." ) if wordnet_dir.nil?
+end
+
+# At this point we know we should have a wordnet directory within wordnet_dir
+puts "Found wordnet files in #{wordnet_dir}..." if VERBOSE
+
+index_files = POS_FILE_TYPES.map { |pos| wordnet_dir + "index.#{pos}" }
+data_files = POS_FILE_TYPES.map { |pos| wordnet_dir + "data.#{pos}" }
+
+(index_files + data_files).each do |required_file|
+    abort( "Unable to locate #{required_file} within the wordnet dictionary. Please check your wordnet copy is valid and try again." ) unless required_file.exist?
+    abort( "Cannot get readable permissions to #{required_file} within the wordnet dictionary. Please check the file permissions and try again." ) unless required_file.readable?
+end
+
+# At this point we know we have the correct files, though we don't know there validity
+puts "Validated existance of wordnet files in #{wordnet_dir}..." if VERBOSE
+
+# Build data
+
+index_hash = Hash.new
+data_hash = Hash.new
+POS_FILE_TYPES.each do |file_pos|
+  
+    puts "Building #{file_pos} indexes..." if VERBOSE
+  
+    # add indexes
+    (wordnet_dir + "index.#{file_pos}").each_line do |index_line|
+	next if index_line[0, 2] == "  "
+	index_parts = index_line.split(" ")
+    
+	lemma, pos, synset_count, pointer_count = index_parts.shift, index_parts.shift, index_parts.shift.to_i, index_parts.shift.to_i
+	pointer_symbols = Array.new(pointer_count).map { POS_FILE_TYPE_TO_SHORT[file_pos] + index_parts.shift }
+	sense_count = index_parts.shift
+	tagsense_count = pos + index_parts.shift
+	synset_ids = Array.new(synset_count).map { POS_FILE_TYPE_TO_SHORT[file_pos] + index_parts.shift }
+    
+	index_hash[lemma] = { "synset_ids" => [], "tagsense_counts" => [] } if index_hash[lemma].nil?
+	index_hash[lemma] = { "lemma" => lemma, "synset_ids" => index_hash[lemma]["synset_ids"] + synset_ids, "tagsense_counts" => index_hash[lemma]["tagsense_counts"] + [tagsense_count] }
+    
+    end
+  
+    if opts[:build_tokyo] || opts[:build_tokyo_with_evocations]
+	puts "Building #{file_pos} data..." if VERBOSE
+    
+	# add data
+	(wordnet_dir + "data.#{file_pos}").each_line do |data_line|
+	    next if data_line[0, 2] == "  "
+	    data_line, gloss = data_line.split(" | ")
+	    data_parts = data_line.split(" ")
+      
+	    synset_id, lexical_filenum, synset_type, word_count = POS_FILE_TYPE_TO_SHORT[file_pos] + data_parts.shift, data_parts.shift, data_parts.shift, data_parts.shift.to_i(16)
+	    words = Array.new(word_count).map { "#{data_parts.shift}.#{data_parts.shift}" }
+	    relations = Array.new(data_parts.shift.to_i).map { "#{data_parts.shift}.#{data_parts.shift}.#{data_parts.shift}.#{data_parts.shift}" }
+      
+	    data_hash[synset_id] = { "synset_id" => synset_id, "lexical_filenum" => lexical_filenum, "synset_type" => synset_type,
+		"words" => words.join('|'), "relations" => relations.join('|'), "gloss" => gloss.strip }
+	end
+    end
+  
+end
+
+score_hash = Hash.new
+if opts[:build_tokyo_with_evocations] || opts[:build_pure_evocations]
+    puts "Downloading score data..." if VERBOSE
+    scores_file = data_path + "scores.txt.gz"
+    scores_file.delete if scores_file.exist?
+    File.open(scores_file,'w') do |file|
+	file.write Net::HTTP.get(URI.parse('http://cloud.github.com/downloads/roja/words/scores.txt.gz'))
+    end
+    abort( "Unable to gather similarities information from http://cloud.github.com/downloads/roja/words/scores.txt.gz... Try again later." ) unless scores_file.exist?
+  
+    puts "Compiling score data..." if VERBOSE
+    Zlib::GzipReader.open(scores_file) do |gz|
+	gz.each_line do |line|
+	    mean, median, sense1, sense2 = line.split(',')
+	    senses = [sense1, sense2].map! { |sense| sense.strip.split('.') }.map! { |sense| index_hash[sense[0]]["synset_ids"].select { |synset_id| synset_id[0,1] == sense[1].gsub("s", "a") }[sense[2].to_i-1] }
+	    senses.each do |sense|
+		relation = (senses - [sense]).first.nil? ? sense : (senses - [sense]).first
+		score_name = sense + "s"
+		score_hash[score_name] = { "relations" => [], "means" => [], "medians" => [] } if score_hash[score_name].nil?
+		score_hash[score_name] = { "relations" => score_hash[score_name]["relations"] << relation, "means" => score_hash[score_name]["means"] << mean, "medians" => score_hash[score_name]["medians"] << median }
+	    end unless senses.include? nil
+	end
+    end
+end
+
+if opts[:build_tokyo] || opts[:build_tokyo_with_evocations]
+    tokyo_hash = Rufus::Tokyo::Table.new((data_path + "wordnet.tct").to_s)
+    index_hash.each { |k,v| tokyo_hash[k] = { "lemma" => v["lemma"], "synset_ids" => v["synset_ids"].join('|'), "tagsense_counts" => v["tagsense_counts"].join('|') } }
+    data_hash.each { |k,v| tokyo_hash[k] = v }
+    score_hash.each { |k,v| tokyo_hash[k] = { "relations" => v["relations"].join('|'), "means" => v["means"].join('|'), "medians" => v["medians"].join('|') } } if opts[:build_tokyo_with_evocations]
+    tokyo_hash.close
+end
+
+if opts[:build_pure_evocations]
+    score = Hash.new
+    score_hash.each { |k,v| score[k] = [v["relations"].join('|'), v["means"].join('|'), v["medians"].join('|')] }
+    File.open(data_path + "evocations.dmp",'w') do |file|
+	file.write Marshal.dump(score)
+    end
+end
+

data/examples.rb

@@ -0,0 +1,55 @@
+#!/usr/bin/env ruby
+# coding: utf-8
+
+#require 'rubygems'
+#require 'words'
+require 'lib/words.rb'
+
+if __FILE__ == $0
+  
+    wordnet = Words::Wordnet.new #:tokyo
+
+    puts wordnet.connected? 
+    wordnet.close!
+    puts wordnet.connected?
+    wordnet.open!
+    puts wordnet.connected?
+  
+    puts wordnet
+
+    puts wordnet.find('squash racquet')
+
+    puts wordnet.find('bat')
+    puts wordnet.find('bat').available_pos.inspect
+    puts wordnet.find('bat').lemma
+    puts wordnet.find('bat').nouns?
+    puts wordnet.find('bat').synsets('noun')
+    puts wordnet.find('bat').noun_ids
+    puts wordnet.find('bat').synsets(:noun)[2].words.inspect
+    puts wordnet.find('bat').nouns.last.relations
+    wordnet.find('bat').synsets('noun').last.relations.each { |relation| puts relation.inspect }
+    puts wordnet.find('bat').synsets('noun').last.hyponyms?
+    puts wordnet.find('bat').synsets('noun').last.participle_of_verbs?
+  
+    puts wordnet.find('bat').synsets('noun').last.relations(:hyponym)
+    puts wordnet.find('bat').synsets('noun').last.hyponyms?
+    puts wordnet.find('bat').synsets('noun').last.relations("~")
+    puts wordnet.find('bat').synsets('verb').last.inspect
+    puts wordnet.find('bat').synsets('verb').last.words.inspect
+    puts wordnet.find('bat').synsets('verb').last.words_with_lexical_ids.inspect
+  
+    puts wordnet.find('bat').synsets('verb').first.lexical.inspect
+    puts wordnet.find('bat').synsets('verb').first.lexical_description
+  
+    puts wordnet.find('jkashdfajkshfksjdhf')
+
+    if wordnet.evocations?
+	puts wordnet.find("broadcast").senses.first.evocations
+	puts wordnet.find("broadcast").senses.first.evocations.means
+	puts wordnet.find("broadcast").senses.first.evocations[1].inspect
+	puts wordnet.find("broadcast").senses.first.evocations[20][:destination].words
+    end
+
+    wordnet.close!
+  
+end