anystyle-parser 0.0.10 → 0.1.0

data/HISTORY.md

@@ -1,3 +1,7 @@
+0.1.0 / 2012-03-03
+==================
+* Added redis as data store option
+
 0.0.10 / 2012-03-01
 ===================
 * Added new output format: tags (to generate training data)

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright 2011 Sylvester Keil. All rights reserved.
+Copyright 2011-2012 Sylvester Keil. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:

data/README.md

@@ -3,12 +3,14 @@ Anystyle-Parser
 
 Anystyle-Parser is a very fast and smart parser for academic references. It
 is inspired by [ParsCit](http://aye.comp.nus.edu.sg/parsCit/) and
-[FreeCite](http://freecite.library.brown.edu/); Anystyle-Parser is designed
-for raw speed (it uses [wapiti](https://github.com/inukshuk/wapiti-ruby) based
+[FreeCite](http://freecite.library.brown.edu/); Anystyle-Parser uses machine
+learning algorithms and is designed
+for raw speed (it [wapiti](https://github.com/inukshuk/wapiti-ruby) based
 conditional random fields and [Kyoto Cabinet](http://fallabs.com/kyotocabinet/)
-as a key-value store), flexibility (it is easy to train the model with
-data that is relevant to your parsing needs), and compatibility (Anystyle-Parser
-exports to Ruby Hashes, BibTeX, or the CiteProc JSON format).
+or [Redis](http://redis.io) as a key-value store), flexibility (it is easy to
+train the model with data that is relevant to your parsing needs), and
+compatibility (Anystyle-Parser exports to Ruby Hashes, BibTeX, or the
+CSL/CiteProc JSON format).
 
 Installation
 ------------
@@ -31,7 +33,36 @@ note that you will need write permissions in the directory where the file
 is to be created. You can change the Dictionary's default path in the
 Dictionary's options:
 
-    Anystyle::Parser::Dictionary.instance.options[:path]
+    Anystyle::Parser::Dictionary.instance.options[:cabinet]
+
+Starting with version 0.1.0, Anystyle-Parser also supports
+[Redis](http://redis.io); to use Redis as the data store you need to install
+the `redis` gem (and, optionally, the `hiredis` gem).
+
+    $ [sudo] gem install hiredis 
+    $ [sudo] gem install redis 
+
+To see which data store modes are available in you current environment,
+check the output of `Dictionary.modes`:
+
+    > Anystyle::Parser::Dictionary.modes
+    => [:kyoto, :redis, :hash]
+
+To select one of the available modes, use the dictionary instance options:
+
+    > Anystyle::Parser::Dictionary.instance.options[:mode]
+    => :kyoto
+    
+To use [Redis](http://redis.io) you also need to set the host or unix socket
+where your redis server is available. For example:
+
+    Anystyle::Parser::Dictionary.instance.options[:mode] = :redis
+    Anystyle::Parser::Dictionary.instance.options[:host] = 'localhost'
+
+When the data store is opened using redis-mode and the data store is empty,
+the feature dictionary will be imported automatically. If you want to import
+the data explicitly you can use `Dictionary#create` after setting the
+required options.
 
 
 Usage
@@ -152,7 +183,7 @@ and open a pull request on GitHub.
 License
 -------
 
-Copyright 2011 Sylvester Keil. All rights reserved.
+Copyright 2011-2012 Sylvester Keil. All rights reserved.
 
 Some of the code in Anystyle-Parser's post processing (normalizing) routines
 was originally based on the source code of FreeCite and

data/anystyle-parser.gemspec

@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
   s.description = 'A sophisticated parser for academic references based on conditional random fields.'
   s.license     = 'FreeBSD'
   
-  s.add_runtime_dependency('bibtex-ruby', '~>1.3')
+  s.add_runtime_dependency('bibtex-ruby', '~>2.0')
   s.add_runtime_dependency('wapiti', '~>0.0')
   
   s.add_development_dependency('rake', ['~>0.9'])

data/lib/anystyle/parser/dictionary.rb

@@ -1,165 +1,229 @@
 module Anystyle
-	module Parser
-		
-		# Dictionary is a Singleton object that provides a key-value store of
-		# the Anystyle Parser dictionary required for feature elicitation.
-		# This dictionary acts essentially like a Ruby Hash object, but because
-		# of the dictionary's size it is not efficient to keep the entire
-		# dictionary in memory at all times. For that reason, Dictionary
-		# creates a persistent data store on disk using Kyoto Cabinet; if
-		# Kyoto Cabinet is not installed a Ruby Hash is used as a fall-back.
-		# 
-		# The database will be automatically created from the dictionary file
-		# using the best available DBM the first time it is accessed. Once
-		# database file exists, the database will be restored from file.
-		# Therefore, if you make changes to the dictionary file, you will have
-		# to delete the old database file for a new one to be created.
-		#
-		# Database creation requires write permissions. By default, the database
-		# will be created in the support directory of the Parser; if you have
-		# installed the gem version of the Parser, you may not have write
-		# permissions, but you can change the path in the Dictionary's options.
-		#
-		#     Dictionary.instance.options[:path] # => the database file
-		#     Dictionary.instance.options[:source] # => the (zipped) dictionary file
-		#
-		class Dictionary
+  module Parser
+    
+    # Dictionary is a Singleton object that provides a key-value store of
+    # the Anystyle Parser dictionary required for feature elicitation.
+    # This dictionary acts essentially like a Ruby Hash object, but because
+    # of the dictionary's size it is not efficient to keep the entire
+    # dictionary in memory at all times. For that reason, Dictionary
+    # creates a persistent data store on disk using Kyoto Cabinet; if
+    # Kyoto Cabinet is not installed a Ruby Hash is used as a fall-back.
+    #
+    # Starting with version 0.1.0 Redis support was added. If you would
+    # like to use Redis as the dictionary data store you can do so by
+    # 
+    # 
+    # The database will be automatically created from the dictionary file
+    # using the best available DBM the first time it is accessed. Once
+    # database file exists, the database will be restored from file.
+    # Therefore, if you make changes to the dictionary file, you will have
+    # to delete the old database file for a new one to be created.
+    #
+    # Database creation in Kyoto-Cabinet mode requires write permissions.
+    # By default, the database
+    # will be created in the support directory of the Parser; if you have
+    # installed the gem version of the Parser, you may not have write
+    # permissions, but you can change the path in the Dictionary's options.
+    #
+    # ## Configuration
+    #
+    # To set the database mode:
+    #
+    #     Dictionary.instance.options[:mode] # => the database mode
+    #
+    # For a list of database modes available in your environment consult:
+    #
+    #     Dictionary.modes # => [:kyoto, :redis, :hash]
+    #
+    # Further options include:
+    #
+    #     Dictionary.instance.options[:source] # => the zipped dictionary file
+    #     Dictionary.instance.options[:cabinet] # => the database file (kyoto)
+    #     Dictionary.instance.options[:path] # => the database socket (redis)
+    #     Dictionary.instance.options[:host] # => dictionary host (redis)
+    #     Dictionary.instance.options[:part] # => dictionary port (redis)
+    #
+    class Dictionary
 
-			include Singleton
-			
-			@defaults = {
-				:source => File.expand_path('../support/dict.txt.gz', __FILE__),
-				:path => File.expand_path('../support/dict.kch', __FILE__)
-			}.freeze
-			
-			@keys = [:male, :female, :surname, :month, :place, :publisher, :journal].freeze
+      include Singleton
+      
+      @keys = [:male, :female, :surname, :month, :place, :publisher, :journal].freeze
 
-			@code = Hash[*@keys.zip(0.upto(@keys.length-1).map { |i| 2**i }).flatten]
-			@code.default = 0
-			@code.freeze
+      @code = Hash[*@keys.zip(0.upto(@keys.length-1).map { |i| 2**i }).flatten]
+      @code.default = 0
+      @code.freeze
 
-			@mode = begin
-				require 'kyotocabinet'
-				:kyoto
-			rescue LoadError
-				:hash
-			end
-			
-			class << self
-				
-				attr_reader :keys, :code, :defaults, :mode
-				
-			end
+      @modes = [:hash]
 
-			attr_reader :options
-			
-			def initialize
-				@options = Dictionary.defaults.dup
-			end
-			
-			def [](key)
-				db[key.to_s].to_i
-			end
-			
-			def []=(key, value)
-				db[key.to_s] = value
-			end
-			
-			def create
-				case Dictionary.mode
-				when :kyoto
-					truncate
-					@db = KyotoCabinet::DB.new
-					unless @db.open(path, KyotoCabinet::DB::OWRITER | KyotoCabinet::DB::OCREATE)
-						raise DatabaseError, "failed to create cabinet file #{path}: #{@db.error}"
-					end
-					populate
-					close
-				else
-					# nothing
-				end
-			end
-			
-			def truncate
-				close
-				File.unlink(path) if File.exists?(path)				
-			end
-			
-			def open
-				create unless File.exists?(path)
+      begin
+        require 'redis/connection/hiredis'
+      rescue LoadError
+        # ignore
+      end
 
-				case Dictionary.mode
-				when :kyoto
-					at_exit { ::Anystyle::Parser::Dictionary.instance.close }
-	
-					@db = KyotoCabinet::DB.new
-					unless @db.open(path, KyotoCabinet::DB::OREADER)
-						raise DictionaryError, "failed to open cabinet file #{path}: #{@db.error}"
-					end
-				else
-					@db = Hash.new(0)
-					populate
-				end
-				
-				@db
-			end
-			
-			def open?; !!@db; end
-			
-			def close
-				@db.close if @db.respond_to?(:close)
-				@db = nil
-			end
-			
-			def path
-				options[:path]
-			end
-			
-			private
-						
-			def db
-				@db || open
-			end
-						
-			def populate
-				require 'zlib'
+      begin
+        require 'redis'
+        @modes.unshift :redis
+      rescue LoadError
+        # info 'no redis support detected'
+      end
+      
+      begin
+        require 'kyotocabinet'
+        @modes.unshift :kyoto
+      rescue LoadError
+        # info 'no kyoto-cabinet support detected'
+      end
+      
+      @defaults = {
+        :mode => @modes[0],
+        :source => File.expand_path('../support/dict.txt.gz', __FILE__),
+        :cabinet => File.expand_path('../support/dict.kch', __FILE__),
+        :port => 6379
+      }.freeze
+      
+      
+      class << self
+        
+        attr_reader :keys, :code, :defaults, :modes
+        
+      end
 
-				File.open(options[:source], 'r:UTF-8') do |f|
-					mode = 0
+      attr_reader :options
+      
+      def initialize
+        @options = Dictionary.defaults.dup
+      end
+      
+      def [](key)
+        db[key.to_s].to_i
+      end
+      
+      def []=(key, value)
+        db[key.to_s] = value
+      end
+      
+      def create
+        case Dictionary.mode
+        when :kyoto
+          truncate
+          @db = KyotoCabinet::DB.new
+          unless @db.open(path, KyotoCabinet::DB::OWRITER | KyotoCabinet::DB::OCREATE)
+            raise DatabaseError, "failed to create cabinet file #{path}: #{@db.error}"
+          end
+          populate
+          close
+        
+        when :redis
+          @db ||= Redis.new(options)
+          populate
+          close
+          
+        else
+          # nothing
+        end
+      end
+      
+      def truncate
+        close
+        File.unlink(path) if File.exists?(path)       
+      end
+      
+      def open
+        case options[:mode]
+        when :kyoto
+          at_exit { ::Anystyle::Parser::Dictionary.instance.close }
 
-					Zlib::GzipReader.new(f).each do |line|
-						line.strip!
-						
-						if line.start_with?('#')
-							case line
-							when /^## male/i
-								mode = Dictionary.code[:male]
-			        when /^## female/i
-			          mode = Dictionary.code[:female]
-			        when /^## (?:surname|last|chinese)/i
-			          mode = Dictionary.code[:surname]
-			        when /^## months/i
-			          mode = Dictionary.code[:month]
-			        when /^## place/i
-			          mode = Dictionary.code[:place]
-			        when /^## publisher/i
-			          mode = Dictionary.code[:publisher]
-			        when /^## journal/i
-			          mode = Dictionary.code[:journal]
-							else
-								# skip comments
-							end
-						else
-							key, probability = line.split(/\s+(\d+\.\d+)\s*$/)
-							value = self[key]
-							self[key] = value + mode if value < mode
-						end
-					end
-				end
+          create unless File.exists?(path)
+  
+          @db = KyotoCabinet::DB.new
+          unless @db.open(path, KyotoCabinet::DB::OREADER)
+            raise DictionaryError, "failed to open cabinet file #{path}: #{@db.error}"
+          end
+        
+        when :redis
+          at_exit { ::Anystyle::Parser::Dictionary.instance.close }
+          @db = Redis.new(options)
+          
+          populate if @db.dbsize.zero?
+           
+        else
+          @db = Hash.new(0)
+          populate
+        end
+        
+        @db
+      end
+      
+      def open?; !!@db; end
+      
+      def close
+        case
+        when @db.respond_to?(:close)
+          @db.close 
+        when @db.respond_to?(:quit)
+          @db.quit
+        end
+        
+        @db = nil
+      end
+      
+      def path
+        case options[:mode]
+        when :kyoto
+          options[:cabinet] || options[:path]
+        when :redis
+          options[:path] || options.values_at(:host, :port).join(':')
+        else
+          'hash'
+        end
+      end
+            
+      private
+  
+      def db
+        @db || open
+      end
+            
+      def populate
+        require 'zlib'
 
-			end
-						
-		end
-	
-	end
+        File.open(options[:source], 'r:UTF-8') do |f|
+          mode = 0
+
+          Zlib::GzipReader.new(f).each do |line|
+            line.strip!
+            
+            if line.start_with?('#')
+              case line
+              when /^## male/i
+                mode = Dictionary.code[:male]
+              when /^## female/i
+                mode = Dictionary.code[:female]
+              when /^## (?:surname|last|chinese)/i
+                mode = Dictionary.code[:surname]
+              when /^## months/i
+                mode = Dictionary.code[:month]
+              when /^## place/i
+                mode = Dictionary.code[:place]
+              when /^## publisher/i
+                mode = Dictionary.code[:publisher]
+              when /^## journal/i
+                mode = Dictionary.code[:journal]
+              else
+                # skip comments
+              end
+            else
+              key, probability = line.split(/\s+(\d+\.\d+)\s*$/)
+              value = self[key]
+              self[key] = value + mode if value < mode
+            end
+          end
+        end
+
+      end
+            
+    end
+  
+  end
 end

data/lib/anystyle/parser/version.rb

@@ -1,5 +1,5 @@
 module Anystyle
   module Parser
-    VERSION = '0.0.10'.freeze
+    VERSION = '0.1.0'.freeze
   end
 end

data/spec/anystyle/parser/dictionary_spec.rb

@@ -10,6 +10,15 @@ module Anystyle
 			it { Dictionary.should_not respond_to(:new) }		
 			it { dict.should_not be nil }
 
+      describe '.modes' do
+        it 'returns an array' do
+          Dictionary.modes.should be_a(Array)
+        end
+        
+        it 'contains at least :hash' do
+          Dictionary.modes.should include(:hash)
+        end
+      end
 
 			describe "the dictionary" do
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: anystyle-parser
 version: !ruby/object:Gem::Version
-  version: 0.0.10
+  version: 0.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,22 +9,22 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-03-01 00:00:00.000000000 Z
+date: 2012-03-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bibtex-ruby
-  requirement: &70338916773120 !ruby/object:Gem::Requirement
+  requirement: &70166358058660 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '1.3'
+        version: '2.0'
   type: :runtime
   prerelease: false
-  version_requirements: *70338916773120
+  version_requirements: *70166358058660
 - !ruby/object:Gem::Dependency
   name: wapiti
-  requirement: &70338916772120 !ruby/object:Gem::Requirement
+  requirement: &70166358047080 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,10 +32,10 @@ dependencies:
         version: '0.0'
   type: :runtime
   prerelease: false
-  version_requirements: *70338916772120
+  version_requirements: *70166358047080
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70338916770100 !ruby/object:Gem::Requirement
+  requirement: &70166358046040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: '0.9'
   type: :development
   prerelease: false
-  version_requirements: *70338916770100
+  version_requirements: *70166358046040
 - !ruby/object:Gem::Dependency
   name: racc
-  requirement: &70338916769520 !ruby/object:Gem::Requirement
+  requirement: &70166358045460 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -54,10 +54,10 @@ dependencies:
         version: '1.4'
   type: :development
   prerelease: false
-  version_requirements: *70338916769520
+  version_requirements: *70166358045460
 - !ruby/object:Gem::Dependency
   name: cucumber
-  requirement: &70338916768880 !ruby/object:Gem::Requirement
+  requirement: &70166358044240 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -65,10 +65,10 @@ dependencies:
         version: '1.0'
   type: :development
   prerelease: false
-  version_requirements: *70338916768880
+  version_requirements: *70166358044240
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70338916768220 !ruby/object:Gem::Requirement
+  requirement: &70166358043560 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -76,10 +76,10 @@ dependencies:
         version: '2.6'
   type: :development
   prerelease: false
-  version_requirements: *70338916768220
+  version_requirements: *70166358043560
 - !ruby/object:Gem::Dependency
   name: ZenTest
-  requirement: &70338916767680 !ruby/object:Gem::Requirement
+  requirement: &70166358042940 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -87,7 +87,7 @@ dependencies:
         version: '4.6'
   type: :development
   prerelease: false
-  version_requirements: *70338916767680
+  version_requirements: *70166358042940
 description: A sophisticated parser for academic references based on conditional random
   fields.
 email:
@@ -148,7 +148,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -640366899922045737
+      hash: 1324194107450037111
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -157,7 +157,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -640366899922045737
+      hash: 1324194107450037111
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.10