lingo 1.8.2 → 1.8.3

data/ChangeLog

@@ -1,5 +1,38 @@
 = Revision history for Lingo
 
+== 1.8.3 [2012-09-20]
+
+* Fixed regression introduced in 1.8.2 where reading input from STDIN was no
+  longer possible.
+* Fixed regression introduced in 1.8.2 where Lingo would no longer run on Ruby
+  1.9.2.
+* Fixed length limit handling for multibyte characters in SDBM store.
+* Fixed encoding issue in SDBM store.
+* Fixed issue with BOM in config files.
+* Modified character handling to accept *any* Unicode letter (_Alphabetical_)
+  and digit (<i>Decimal Number</i>).
+* Modified Attendee::Tokenizer to use only hard-coded tokenization rules.
+* Modified Attendee::VectorFilter's +lexicals+ option to be case-sensitive.
+* Improved overall performance and memory usage; Attendee::Sequencer changed
+  the order sequences are inserted into the stream.
+* Eliminated performance penalty caused by Attendee::Abbreviator.
+* Added Russian language support (Yulia Dorokhova, Thomas Müller).
+* Added +fields+ option to Attendee::TextReader to cut off field labels;
+  defaults to +true+ in record (LIR) mode.
+* Added +skip+ option to Attendee::TextReader to skip lines matching the given
+  pattern.
+* Added +src+ option to Attendee::VectorFilter to print "source" part of
+  compounds.
+* Added +lingosrv+ and +lingoweb+ executables. The former provides a simple
+  HTTP endpoint with JSON output; the latter serves a demo web interface.
+* Refactored internal caching.
+* Made dependency on Ruby version >= 1.9.2 explicit.
+* Removed reporting facility (options '--perfmon' and '--status').
+* Learned '--profile' option to collect profiling information while running.
+* Deprecated Language::Grammar option +compositum+ (now +compound+), Config
+  option +textreader+ (now +text_reader+), and Attendee::TextReader option
+  +lir-record-pattern+ (now +records+); they will be removed in Lingo 1.9.
+
 == 1.8.2 [2012-04-19]
 
 * Performance improvements regarding Attendee::VectorFilter's (as well as

data/README

@@ -25,7 +25,7 @@
 
 == VERSION
 
-This documentation refers to Lingo version 1.8.2
+This documentation refers to Lingo version 1.8.3
 
 
 == DESCRIPTION
@@ -173,8 +173,8 @@ to run that command with administrator privileges, depending on your
 environment). Then you can call the +lingo+ executable to process your
 data. See <tt>lingo --help</tt> for starters.
 
-Please note that Lingo requires Ruby version 1.9 to run
-(1.9.3[http://ruby-lang.org/en/downloads/] is the currently recommended
+Please note that Lingo requires Ruby version 1.9 to run (1.9.2 or greater;
+1.9.3[http://ruby-lang.org/en/downloads/] is the currently recommended
 version). If you want to use Lingo on Ruby 1.8, please refer to the legacy
 version (see below).
 
@@ -225,7 +225,7 @@ the development efforts will be directed towards Lingo 1.8+.
 To install the legacy version, download and extract the ZIP archive from
 RubyForge[http://rubyforge.org/frs/?group_id=5663]. No additional dependencies
 are required. This version of Lingo works with both Ruby 1.8 (1.8.5 or greater)
-and 1.9.
+and 1.9 (1.9.2 or greater).
 
 The executable is named +lingo.rb+. It's located at the root of the installation
 directory and may only be run from there. See <tt>ruby lingo.rb -h</tt> for
@@ -320,7 +320,8 @@ Lingo is based on a collective development by Klaus Lepsky and John Vorhauer.
 
 * Klaus Lepsky <mailto:klaus@lepsky.de>
 * Jan-Helge Jacobs <mailto:plancton@web.de>
-* Thomas Müller <mailto:thomas.mueller@fh-koeln.de>
+* Thomas Müller <mailto:thomas.mueller@gesis.org>
+* Yulia Dorokhova <mailto:jdorokhova@hse.ru>
 
 
 == LICENSE AND COPYRIGHT

data/Rakefile

@@ -39,11 +39,13 @@ The main functions of Lingo are:
   of word classes
       EOT
       extra_files:  FileList[
-        'lingo.rb', 'lingo{,-call}.cfg', 'lir.cfg', '{de,en}.lang',
-        '{de,en}/{lingo-*,user-dic,test_*}.txt', 'txt/{artikel{,-en},lir}.txt'
+        'lingo.rb', 'lingo{,-call}.cfg', 'lir.cfg',
+        '{de,en,ru}.lang', '{de,en,ru}/{lingo-*,user-dic,test_*}.txt',
+        'txt/{artikel{,-en,-ru},lir}.txt', 'lib/lingo/{srv,web}/**/*'
       ].to_a,
-      required_ruby_version:    '>= 1.9',
-      dependencies:             [['ruby-nuggets', '>= 0.8.5'], 'unicode', 'highline'],
+      required_ruby_version:    '>= 1.9.2',
+      dependencies:             [['ruby-nuggets', '>= 0.8.5'],
+                                 'unicode', 'highline', 'sinatra'],
       development_dependencies: [['diff-lcs', '>= 1.1.3'], 'open4']
     }
   }}

data/{lib/lingo/cachable.rb → bin/lingosrv}

@@ -1,58 +1,30 @@
-# encoding: utf-8
-
-#--
-###############################################################################
-#                                                                             #
-# Lingo -- A full-featured automatic indexing system                          #
-#                                                                             #
-# Copyright (C) 2005-2007 John Vorhauer                                       #
-# Copyright (C) 2007-2012 John Vorhauer, Jens Wille                           #
-#                                                                             #
-# Lingo is free software; you can redistribute it and/or modify it under the  #
-# terms of the GNU Affero General Public License as published by the Free     #
-# Software Foundation; either version 3 of the License, or (at your option)   #
-# any later version.                                                          #
-#                                                                             #
-# Lingo is distributed in the hope that it will be useful, but WITHOUT ANY    #
-# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS   #
-# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for     #
-# more details.                                                               #
-#                                                                             #
-# You should have received a copy of the GNU Affero General Public License    #
-# along with Lingo. If not, see <http://www.gnu.org/licenses/>.               #
-#                                                                             #
-###############################################################################
-#++
-
-class Lingo
-
-  # Provides a simple caching mechanism.
-
-  module Cachable
-
-    def init_cachable
-      @cachable_hash = Hash.new(false)
-    end
-
-    def hit?(key)
-      @cachable_hash.has_key?(key)
-    end
-
-    def store(key, val)
-      @cachable_hash[key] = cache_value(val)
-      val
-    end
-
-    def retrieve(key)
-      cache_value(@cachable_hash[key])
-    end
-
-    private
-
-    def cache_value(val)
-      val.dup unless val.nil?
-    end
-
-  end
-
-end
+#! /usr/bin/env ruby
+
+#--
+###############################################################################
+#                                                                             #
+# Lingo -- A full-featured automatic indexing system                          #
+#                                                                             #
+# Copyright (C) 2005-2007 John Vorhauer                                       #
+# Copyright (C) 2007-2012 John Vorhauer, Jens Wille                           #
+#                                                                             #
+# Lingo is free software; you can redistribute it and/or modify it under the  #
+# terms of the GNU Affero General Public License as published by the Free     #
+# Software Foundation; either version 3 of the License, or (at your option)   #
+# any later version.                                                          #
+#                                                                             #
+# Lingo is distributed in the hope that it will be useful, but WITHOUT ANY    #
+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS   #
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for     #
+# more details.                                                               #
+#                                                                             #
+# You should have received a copy of the GNU Affero General Public License    #
+# along with Lingo. If not, see <http://www.gnu.org/licenses/>.               #
+#                                                                             #
+###############################################################################
+#++
+
+require 'lingo'
+require 'lingo/srv'
+
+Lingo::Srv.run!

data/bin/lingoweb

@@ -0,0 +1,30 @@
+#! /usr/bin/env ruby
+
+#--
+###############################################################################
+#                                                                             #
+# Lingo -- A full-featured automatic indexing system                          #
+#                                                                             #
+# Copyright (C) 2005-2007 John Vorhauer                                       #
+# Copyright (C) 2007-2012 John Vorhauer, Jens Wille                           #
+#                                                                             #
+# Lingo is free software; you can redistribute it and/or modify it under the  #
+# terms of the GNU Affero General Public License as published by the Free     #
+# Software Foundation; either version 3 of the License, or (at your option)   #
+# any later version.                                                          #
+#                                                                             #
+# Lingo is distributed in the hope that it will be useful, but WITHOUT ANY    #
+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS   #
+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for     #
+# more details.                                                               #
+#                                                                             #
+# You should have received a copy of the GNU Affero General Public License    #
+# along with Lingo. If not, see <http://www.gnu.org/licenses/>.               #
+#                                                                             #
+###############################################################################
+#++
+
+require 'lingo'
+require 'lingo/web'
+
+Lingo::Web.run!

data/de.lang

@@ -72,7 +72,7 @@ language:
       min-avg-part-size: "4"
       append-wordclass: "+"
       skip-sequences: [ xx ]
-      
+
     suffix:
       #    Suffixliste, Stand: 30-06-2005
       #    Suffixklasse:   s = Substantiv, a = Adjektiv, v = Verb, e = Eigenwort, f = Fugung
@@ -84,17 +84,6 @@ language:
       - [f, "s n e en es er ch/che /en"]
 
   attendees:
-    tokenizer:
-      regulars:
-        - _char_: '_baslat_|_lat1sp_|_latexa_|_latexb_|_ipaext_'
-        - NUMS:  '[+-]?(\d{4,}|\d{1,3}(\.\d{3,3})*)(\.|(,\d+)?%?)'
-        - URLS:  '((mailto:|(news|http|https|ftp|ftps)://)\S+|^(www(\.\S+)+)|[^\s.]+([\._]\S+)+@\S+(\.\S+)+)'
-        - ABRV:  '(((_char_)+\.)+)(_char_)+'
-        - WORD:  '(_char_|_digit_|\-)+'
-        - PUNC:  '([!,\.:;?]|\xc2\xa1|\xc2\xbf)'
-        - OTHR:  '([\"#$%&\x27()*\+\-/<=>@\[\\\]^_{|}~]|\xc2\xa2|\xc2\xa3|\xc2\xa4|\xc2\xa5|\xc2\xa6|\xc2\xa7|\xc2\xa8|\xc2\xa9|\xc2\xaa|\xc2\xab|\xc2\xac|\xc2\xae|\xc2\xaf|\xc2\xb0|\xc2\xb1|\xc2\xb2|\xc2\xb3|\xc2\xb4|\xc2\xb5|\xc2\xb6|\xc2\xb7|\xc2\xb8|\xc2\xb9|\xc2\xba|\xc2\xbb|\xc2\xbc|\xc2\xbd|\xc2\xbe|\xc3\x97|\xc3\xb7)'
-        - HELP:  '[^ ]*'
-
     variator:
       variations:
         - [ ieh, sch ]
@@ -116,6 +105,6 @@ language:
         - [ fl,  st  ]
         - [ li,  h   ]
         - [ i,   s   ]
-        
+
     sequencer:
       sequences: [ [AS, "2, 1"], [AK, "2, 1"], [AAK, "3, 1 2"], [AAS, "3, 1 2"] ]

data/en/lingo-irr.txt

@@ -0,0 +1,266 @@
+#
+#  Wörterbuch der unregelmäßigen Verben
+#  
+#  Stand: 26.05.08 / Thomas Müller
+
+ate=eat #v
+awoke=awake #v
+awoken=awake #v
+beat=beat #v
+became=become #v
+become=become #v
+began=begin #v
+begun=begin #v
+bent=bend #v
+beset=beset #v
+bet=bet #v
+bit=bite #v
+bitten=bite #v
+bled=bleed #v
+blew=blow #v
+blown=blow #v
+bore=bear #v
+born=bear #v
+bought=buy #v
+bound=bind #v
+bred=breed #v
+broadcast=broadcast #v
+broke=break #v
+broken=break #v
+brought=bring #v
+built=build #v
+burst=burst #v
+came=come #v
+cast=cast #v
+caught=catch #v
+chose=choose #v
+chosen=choose #v
+clung=cling #v
+come=come #v
+cost=cost #v
+crept=creep #v
+cut=cut #v
+dealt=deal #v
+did=do #v
+done=do #v
+drank=drink #v
+drawn=draw #v
+drew=draw #v
+driven=drive #v
+drove=drive #v
+drunk=drink #v
+dug=dig #v
+eaten=eat #v
+fallen=fall #v
+fed=feed #v
+fell=fall #v
+felt=feel #v
+fit=fit #v
+fled=flee #v
+flew=fly #v
+flown=fly #v
+flung=fling #v
+forbade=forbid #v
+forbidden=forbid #v
+forgave=forgive #v
+forgiven=forgive #v
+forgot=forget #v
+forgotten=forget #v
+forsaken=forsake #v
+forsook=forsake #v
+fought=fight #v
+found=find #v
+froze=freeze #v
+frozen=freeze #v
+gave=give #v
+given=give #v
+gone=go #v
+got=get #v
+gotten=get #v
+grew=grow #v
+ground=grind #v
+grown=grow #v
+heard=hear #v
+held=hold #v
+hidden=hide #v
+hid=hide #v
+hit=hit #v
+hung=hang #v
+hurt=hurt #v
+kept=keep #v
+knelt=kneel #v
+knew=know #v
+knit=knit #v
+know=know #v
+laid=lay #v
+lain=lie #v
+lay=lie #v
+led=lead #v
+left=leave #v
+lent=lend #v
+let=let #v
+lost=lose #v
+made=make #v
+meant=mean #v
+met=meet #v
+mistaken=mistake #v
+mistook=mistake #v
+overcame=overcome #v
+overcome=overcome #v
+overdid=overdo #v
+overdone=overdo #v
+overtaken=overtake #v
+overthrew=overthrow #v
+overthrown=overthrow #v
+overtook=overtake #v
+paid=pay #v
+pled=plead #v
+put=put #v
+quit=quit #v
+rang=ring #v
+ran=run #v
+read=read #v
+ridden=ride #v
+rid=rid #v
+risen=rise #v
+rode=ride #v
+rose=rise #v
+rung=ring #v
+run=run #v
+said=say #v
+sang=sing #v
+sank=sink #v
+sat=sit #v
+saw=see #v
+seen=see #v
+sent=send #v
+set=set #v
+shaken=shake #v
+shed=shed #v
+shone=shine #v
+shook=shake #v
+shore=shear #v
+shorn=shear #v
+shot=shoot #v
+shrank=shrink #v
+shrunk=shrink #v
+shut=shut #v
+slain=slay #v
+slept=sleep #v
+slew=slay #v
+slid=slide #v
+slit=slit #v
+slung=sling #v
+smitten=smite #v
+smote=smite #v
+sold=sell #v
+sought=seek #v
+sped=speed #v
+spent=spend #v
+split=split #v
+spoken=speak #v
+spoke=speak #v
+spread=spread #v
+spun=spin #v
+stank=stink #v
+stolen=steal #v
+stole=steal #v
+stood=stand #v
+stridden=stride #v
+striven=strive #v
+strode=stride #v
+strove=strive #v
+struck=strike #v
+strung=string #v
+stuck=stick #v
+stung=sting #v
+stunk=stink #v
+sung=sing #v
+sunk=sink #v
+swam=swim #v
+swept=sweep #v
+swore=swear #v
+sworn=swear #v
+swum=swim #v
+swung=swing #v
+taken=take #v
+taught=teach #v
+thought=think #v
+threw=throw #v
+thrown=throw #v
+thrust=thrust #v
+told=tell #v
+took=take #v
+tore=tear #v
+torn=tear #v
+trodden=tread #v
+trod=tread #v
+understood=understand #v
+upheld=uphold #v
+upset=upset #v
+wed=wed #v
+went=go #v
+wept=weep #v
+withheld=withhold #v
+withstood=withstand #v
+woken=wake #v
+woke=wake #v
+won=win #v
+wore=wear #v
+worn=wear #v
+wound=wind #v
+written=write #v
+wrote=write #v
+wrung=wring #v
+bade=bid #v
+been=be #v
+bid=bid #v
+bidden=bid #v
+burned=burn #v
+burnt=burn #v
+dived=dive #v
+dove=dive #v
+dreamed=dream #v
+dreamt=dream #v
+foregone=forego #v
+forewent=forego #v
+leaped=leap #v
+learned=learn #v
+learnt=learn #v
+lept=leap #v
+lighted=light #v
+lit=light #v
+misspelled=misspell #v
+misspelt=misspell #v
+mowed=mow #v
+mown=mow #v
+proved=prove #v
+proven=prove #v
+sawed=saw #v
+sawn=saw #v
+sewed=sew #v
+sewn=sew #v
+shaved=shave #v
+shaven=shave #v
+shod=shoe #v
+shoed=shoe #v
+showed=show #v
+shown=show #v
+sowed=sow #v
+sown=sow #v
+spat=spit #v
+spilled=spill #v
+spilt=spill #v
+spit=spit #v
+sprang=spring #v
+sprung=spring #v
+swelled=swell #v
+swollen=swell #v
+thrived=thrive #v
+thriven=thrive #v
+throve=thrive #v
+was=be #v
+weaved=weave #v
+were=be #v
+woven=weave #v
+wove=weave #v