lingo 1.8.1 → 1.8.2

data/ChangeLog

@@ -1,5 +1,23 @@
 = Revision history for Lingo
 
+== 1.8.2 [2012-04-19]
+
+* Performance improvements regarding Attendee::VectorFilter's (as well as
+  Attendee::NonewordFilter's) memory usage; set <tt>sort: false</tt> in the config.
+* Added Attendee::Stemmer (implementing Porter's algorithm for suffix stripping).
+* Added progress reporting to Attendee::TextReader; set <tt>progress: true</tt>
+  in the config.
+* Added directory and glob processing to Attendee::TextReader (new options
+  +glob+ and +recursive+).
+* Renamed Attendee::TextReader's option +lir-record-pattern+ to +records+.
+* Fixed Attendee::Debugger to forward all objects so it can be inserted
+  between any two attendees in the config.
+* Fixed regression introduced in 1.8.0 where Lingo would not use existing
+  compiled dictionary when source file is not present.
+* Fixed "invalid byte sequence in UTF-8" on Windows for SDBM store.
+* Enabled pluggable (compiled) dictionaries and storage backends.
+* Extensive internal refactoring and cleanup. (Finished for now.)
+
 == 1.8.1 [2012-02-19]
 
 * Introduced alternative storage backends, mainly to circumvent SDBM's record
@@ -62,13 +80,13 @@
   (requires diff-lcs[http://raa.ruby-lang.org/project/diff-lcs/]).
 * Provide alternatives to standard zip command on windows platforms.
 * Use +UNK+ itself if it doesn't have any lexicals.
-* Use compo form instead of word form when lexicalizing compositum entry for
+* Use compo form instead of word form when lexicalizing compound entry for
   multiword dictionaries.
 * LexicalHash#[] must use target (not source) form.
 * Optionally, try to find matches for all lexicals a word has.
 * Make '-' a PRINTABLE_CHAR.
 * Allow synonyms to be considered for multiword matching.
-* Don't use compositum parts.
+* Don't use compound parts.
 * Introduced some (more or less arbitrary) line length limit. We can only
   store values of a certain length anyway (with SDBM). Entries exceeding this
   limit will be rejected and logged in the .rev file.
@@ -252,12 +270,12 @@
 * <b>Decomposer mit zusätzlicher Validitätsprüfung und Kennzeichnung</b>
 
   Der Decomposer kann bei Bedarf Komposita einer zusätzlichen Prüfung unterziehen.
-  Ist der Schlüssel <tt>de.lang:language/dictionary/compositum/skip-sequences</tt>
+  Ist der Schlüssel <tt>de.lang:language/dictionary/compound/skip-sequences</tt>
   angegeben, der z.B. in der Form <tt>skip-sequences: [ VS ]</tt> definiert wird,
   wird zusätzlich geprüft, ob das Kompositum mit seinen Teilen diesen Wortklassen
   entspricht. Hiernach werden Komposita verworfen, die aus Verb-Substantiv-Kombination
   bestehen. Die Angabe des Parameters ist optional.
-  Zusätzlich werden bei Angabe des Schlüssels <tt>de.lang:language/dictionary/compositum/append-wordclass</tt>,
+  Zusätzlich werden bei Angabe des Schlüssels <tt>de.lang:language/dictionary/compound/append-wordclass</tt>,
   der i.d.R einen ein Zeichen langen String enthält, die durch Zerlegung erkannten
   Wortstämme markiert, in dem ihre Wortklasse das über diesen Schlüssel definierte
   Zeichen angehangen bekommt.
@@ -476,7 +494,7 @@
 * <b>Kompositum-Zerlegung mit weiterer Einschränkung</b>
 
   Ein weiterer Parameter ist für die Kompositumzerlegung hinzugekommen. Als
-  Attribute des Tags <tt>XML:dictionary/compositum</tt> können jetzt angegeben werden:
+  Attribute des Tags <tt>XML:dictionary/compound</tt> können jetzt angegeben werden:
 
     Attribut          Default    Funktion
     ============================================================================

data/README

@@ -25,7 +25,7 @@
 
 == VERSION
 
-This documentation refers to Lingo version 1.8.1
+This documentation refers to Lingo version 1.8.2
 
 
 == DESCRIPTION

data/Rakefile

@@ -39,10 +39,8 @@ The main functions of Lingo are:
   of word classes
       EOT
       extra_files:  FileList[
-        'lingo.rb', 'lingo{,-all,-call}.cfg', 'lingo.opt', 'doc/**/*',
-        '{de,en}.lang', '{de,en}/{lingo-*,user-dic}.txt', 'txt/artikel{,-en}.txt',
-        'info/gpl-hdr.txt', 'info/*.png', 'lir.cfg', 'txt/lir.txt', 'porter/*',
-        'test.cfg', '{de,en}/test_*.txt'
+        'lingo.rb', 'lingo{,-call}.cfg', 'lir.cfg', '{de,en}.lang',
+        '{de,en}/{lingo-*,user-dic,test_*}.txt', 'txt/{artikel{,-en},lir}.txt'
       ].to_a,
       required_ruby_version:    '>= 1.9',
       dependencies:             [['ruby-nuggets', '>= 0.8.5'], 'unicode', 'highline'],
@@ -54,7 +52,7 @@ rescue LoadError => err
 end
 
 CLEAN.include(
-  'txt/*.{log,mul,non,seq,syn,ve?,csv}',
+  'txt/*.{log,mul,non,seq,ste,syn,ve?}',
   'test/{test.*,text.non}',
   'store/*/*.rev',
   'bench/tmp.*'
@@ -78,7 +76,7 @@ end
 
 desc 'Test against reference file (TXT)'
 task 'test:txt' do
-  test_ref('artikel', 'test')
+  test_ref('artikel', 'lingo')
 end
 
 desc 'Test against reference file (LIR)'
@@ -116,7 +114,7 @@ def test_ref(name, cfg = name)
   }.success? or abort msg.join("\n\n")
 
   Dir["test/ref/#{name}.*"].each { |ref|
-    puts "#{'#' * 60} #{org = ref.sub(/test\/ref/, 'txt')}"
+    puts "## #{org = ref.sub(/test\/ref/, 'txt')}"
     continue += Diff::LCS::Ldiff.run(ARGV.clear << '-a' << org << ref)
   }
 

data/TODO

@@ -1,5 +1,7 @@
 = ToDo list for Lingo
 
+* Configuration parameter validation.
+* Replace regex-based tokenizer with a (Racc/Ragel/ANTLR-based?) lexer.
 * Update and translate old documentation.
 * Allow for handling of documents in various encodings, not just the one the
   dictionaries are encoded in.

data/bin/lingo

@@ -26,4 +26,8 @@
 
 require 'lingo'
 
-Lingo.talk
+begin
+  Lingo.talk
+rescue Lingo::LingoError => err
+  $VERBOSE ? raise : abort(err.to_s)
+end

data/de.lang

@@ -65,7 +65,7 @@ language:
       tst-cry: { name: de/test_cry.txt, txt-format: WordClass, crypt } #  TEST: Verschlüsselung
       tst-sgw: { name: de/test_singleword.txt, txt-format: SingleWord } #  TEST: SingleWord-Format
 
-    compositum:
+    compound:
       min-word-size: "7"
       min-part-size: "3"
       max-parts: "5"

data/en.lang

@@ -52,12 +52,13 @@ language:
 
       #    Systemwörterbücher
       sys-dic: { name: en/lingo-dic.txt, txt-format: WordClass, separator: '=' }
+      sys-syn: { name: en/lingo-syn.txt, txt-format: KeyValue, separator: '=', def-wc: y }
       sys-mul: { name: en/lingo-mul.txt, txt-format: SingleWord, use-lex: 'sys-dic', def-wc: m }
 
       #    Benutzerwörterbücher
       usr-dic: { name: en/user-dic.txt, txt-format: WordClass, separator: '=' }
 
-    compositum:
+    compound:
       min-word-size: "7"
       min-part-size: "3"
       max-parts: "5"

data/lib/lingo/attendee/abbreviator.rb

@@ -76,17 +76,15 @@ class Lingo
         set_dic
       end
 
-      def control(cmd, par)
-        @dic.report.each_pair { |key, value| set(key, value) } if cmd == STR_CMD_STATUS
-
-        # Jedes Control-Object ist auch Auslöser der Verarbeitung
+      def control(cmd, param)
+        report_on(cmd, @dic)
         process_buffer
       end
 
       private
 
       def process_buffer?
-        @buffer[-1].kind_of?(Token) && @buffer[-1].form == CHAR_PUNCT
+        form_at(-1, Token) == CHAR_PUNCT
       end
 
       def process_buffer
@@ -95,13 +93,14 @@ class Lingo
           return
         end
 
-        # Wort vor dem Punkt im Abkürzungswörterbuch suchen
-        if @buffer[-2].kind_of?(Token)
+        if form = form_at(-2, Token)
           inc('Anzahl gesuchter Abkürzungen')
-          abbr = @dic.find_word(@buffer[-2].form)
-          if abbr.identified?
+
+          if (abbr = find_word(form)).identified?
             inc('Anzahl gefundener Abkürzungen')
+
             abbr.form += CHAR_PUNCT
+
             @buffer[-2] = abbr
             @buffer.delete_at(-1)
           end

data/lib/lingo/attendee/debugger.rb

@@ -95,14 +95,15 @@ class Lingo
         @prompt   = get_key('prompt', 'lex:) ')
       end
 
-      def control(cmd, par)
-        if cmd != STR_CMD_STATUS
-          @lingo.warn "#{@prompt} #{AgendaItem.new(cmd, par).inspect}" if eval(@cmd_eval)
+      def control(cmd, param)
+        if cmd != STR_CMD_STATUS && eval(@cmd_eval)
+          warn "#{@prompt} #{AgendaItem.new(cmd, param).inspect}"
         end
       end
 
       def process(obj)
-        @lingo.warn "#{@prompt} #{obj.inspect}" if eval(@obj_eval)
+        warn "#{@prompt} #{obj.inspect}" if eval(@obj_eval)
+        forward(obj)
       end
 
     end

data/lib/lingo/attendee/decomposer.rb

@@ -79,12 +79,17 @@ class Lingo
         set_gra
       end
 
-      def control(cmd, par)
-        @gra.report.each { |key, val| set(key, val) } if cmd == STR_CMD_STATUS
+      def control(cmd, param)
+        report_on(cmd, @gra)
       end
 
       def process(obj)
-        forward(obj.is_a?(Word) && obj.unknown? ? @gra.find_compositum(obj.form) : obj)
+        if obj.is_a?(Word) && obj.unknown?
+          com = @gra.find_compound(obj.form)
+          obj = com unless com.unknown?
+        end
+
+        forward(obj)
       end
 
     end

data/lib/lingo/attendee/dehyphenizer.rb

@@ -41,9 +41,6 @@ class Lingo
     # <b>out</b>:: siehe allgemeine Beschreibung des Attendee
     # <b>source</b>:: siehe allgemeine Beschreibung des Dictionary
     # <b><i>mode</i></b>:: (Standard: all) siehe allgemeine Beschreibung des Dictionary
-    # <b><i>stopper</i></b>:: (Standard: TA_PUNCTUATION, TA_OTHER) Gibt die Begrenzungen an, zwischen
-    #                         denen der Multiworder suchen soll, i.d.R. Satzzeichen und Sonderzeichen,
-    #                         weil sie kaum in einer Mehrwortgruppen vorkommen.
     #
     # === Beispiele
     # Bei der Verarbeitung einer normalen Textdatei mit der Ablaufkonfiguration <tt>t1.cfg</tt>
@@ -74,87 +71,46 @@ class Lingo
       protected
 
       def init
-        @stopper = get_array('stopper', TA_PUNCTUATION+','+TA_OTHER).map(&:upcase)
-
         set_dic
         set_gra
 
-        @skip = get_array('skip', '').map(&:downcase)
-
-        @number_of_expected_tokens_in_buffer = 2
-        @eof_handling = false
-      end
+        @skip = get_array('skip', '', :downcase)
 
-      def control(cmd, par)
-        @dic.report.each_pair { |key, value| set(key, value) } if cmd == STR_CMD_STATUS
-
-        # Jedes Control-Object ist auch Auslöser der Verarbeitung
-        if cmd == STR_CMD_RECORD || cmd == STR_CMD_EOF
-          @eof_handling = true
-          while number_of_valid_tokens_in_buffer > 1
-            process_buffer
-          end
-          forward_number_of_token( @buffer.size, false )
-          @eof_handling = false
-        end
+        @expected_tokens_in_buffer, @eof_handling = 2, false
       end
 
-      def process_buffer?
-        number_of_valid_tokens_in_buffer >= @number_of_expected_tokens_in_buffer
+      def control(cmd, param)
+        control_multi(cmd)
       end
 
       def process_buffer
-        if @buffer[0].is_a?(Word) &&
-          @buffer[0].form[-1..-1] == '-' &&
-          @buffer[1].is_a?(Word) &&
-          !(!( ttt = @buffer[1].get_class(/./) ).nil? &&
-          !@skip.index( ttt[0].attr ).nil?)
-
-          # Einfache Zusammensetzung versuchen
-          form = @buffer[0].form[0...-1] + @buffer[1].form
-          word = @dic.find_word(form)
-          word = @gra.find_compositum(form) unless word.identified?
-
-          unless word.identified? || (word.attr == WA_KOMPOSITUM && word.get_class('x+').empty?)
-            # Zusammensetzung mit Bindestrich versuchen
-            form = @buffer[0].form + @buffer[1].form
-            word = @dic.find_word(form)
-             word = @gra.find_compositum(form) unless word.identified?
-          end
+        a, b, h = *ab = @buffer.values_at(0, 1), '-'
 
-          unless word.identified? || (word.attr == WA_KOMPOSITUM && word.get_class('x+').empty?)
-            # Zusammensetzung mit Bindestrich versuchen
-            form = @buffer[0].form + @buffer[1].form
-            word = @dic.find_word(form)
-            word = @gra.find_compositum(form) unless word.identified?
-          end
+        if ab.all? { |i| i.is_a?(Word) } && a.form[-1, 1] == h && !(
+          (c = b.get_class(/./).first) && @skip.include?(c.attr)
+        )
+          a, b = ab.map!(&:form)
 
-          if word.identified? || (word.attr == WA_KOMPOSITUM && word.get_class('x+').empty?)
+          word = dehyphenize(a.chomp(h) + b)
+          word = dehyphenize(a          + b) unless dehyphenized?(word)
+
+          if dehyphenized?(word)
             @buffer[0] = word
-            @buffer.delete_at( 1 )
+            @buffer.delete_at(1)
           end
         end
 
-        # Buffer weiterschaufeln
-        forward_number_of_token( 1, false )
+        forward_number_of_token(1, false)
       end
 
       private
 
-      # Leitet 'len' Token weiter
-      def forward_number_of_token( len, count_punc = true )
-        begin
-          unless @buffer.empty?
-            forward( @buffer[0] )
-            len -= 1 unless count_punc && @buffer[0].form == CHAR_PUNCT
-            @buffer.delete_at( 0 )
-          end
-        end while len > 0
+      def dehyphenize(form)
+        find_word(form, &:identified?)
       end
 
-      # Liefert die Anzahl gültiger Token zurück
-      def number_of_valid_tokens_in_buffer
-        @buffer.collect { |token| (token.form == CHAR_PUNCT) ? nil : 1 }.compact.size
+      def dehyphenized?(word)
+        word.identified? || word.full_compound?
       end
 
     end

data/lib/lingo/attendee/formatter.rb

@@ -43,7 +43,7 @@ class Lingo
       end
 
       def process(obj)
-        if obj.is_a?(Word) || obj.is_a?(Token)
+        if obj.is_a?(WordForm)
           str = obj.form
 
           if obj.respond_to?(:lexicals)

data/lib/lingo/attendee/multi_worder.rb

@@ -48,9 +48,6 @@ class Lingo
     # <b>out</b>:: siehe allgemeine Beschreibung des Attendee
     # <b>source</b>:: siehe allgemeine Beschreibung des Dictionary
     # <b><i>mode</i></b>:: (Standard: all) siehe allgemeine Beschreibung des Dictionary
-    # <b><i>stopper</i></b>:: (Standard: TA_PUNCTUATION, TA_OTHER) Gibt die Begrenzungen an, zwischen
-    #                         denen der MultiWorder suchen soll, i.d.R. Satzzeichen und Sonderzeichen,
-    #                         weil sie kaum in einer Mehrwortgruppen vorkommen.
     #
     # === Beispiele
     # Bei der Verarbeitung einer normalen Textdatei mit der Ablaufkonfiguration <tt>t1.cfg</tt>
@@ -81,216 +78,131 @@ class Lingo
       protected
 
       def init
-        @stopper = get_array('stopper', TA_PUNCTUATION+','+TA_OTHER).map(&:upcase)
-        @mul_dic = dictionary(mul_src = get_array('source'), get_key('mode', 'all'))
-
         # combine lexical variants?
         #
         # false = old behaviour
         # true  = first match
         # 'all' = all matches
-        @combine  = get_key('combine', false)
-        @all_keys = @combine.is_a?(String) && @combine.downcase == 'all'
+        @combine = get_key('combine', false)
+        @all     = @combine.is_a?(String) && @combine.downcase == 'all'
+
+        lex_src, lex_mod, d = nil, nil, @lingo.dictionary_config['databases']
 
-        lex_src, lex_mod, databases = nil, nil, @lingo.dictionary_config['databases']
+        (mul_src = get_array('source')).each { |src|
+          s, m = d[src].values_at('use-lex', 'lex-mode')
 
-        mul_src.each { |src|
-          this_src, this_mod = databases[src].values_at('use-lex', 'lex-mode')
-          if lex_src.nil? || lex_src == this_src
-            lex_src, lex_mod = this_src, this_mod
+          if lex_src.nil? || lex_src == s
+            lex_src, lex_mod = s, m
           else
-            @lingo.warn "#{self.class}: Dictionaries don't match: #{mul_src.join(',')}"
+            warn "#{self.class}: Dictionaries don't match: #{mul_src.join(',')}"
           end
         }
 
-        lex_src = lex_src.split(STRING_SEPARATOR_RE)
+        lex_src = lex_src.split(SEP_RE)
         lex_mod = get_key('lex-mode', lex_mod || 'first')
 
+        @mul_dic = dictionary(mul_src, get_key('mode', 'all'))
         @lex_dic = dictionary(lex_src, lex_mod)
         @lex_gra = grammar(lex_src, lex_mod)
 
-        if @combine && has_key?('use-syn')
-          @syn_dic = dictionary(get_array('use-syn'), get_key('syn-mode', 'all'))
+        @syn_dic = if @combine && has_key?('use-syn')
+          dictionary(get_array('use-syn'), get_key('syn-mode', 'all'))
         end
 
-        @number_of_expected_tokens_in_buffer = 3
-        @eof_handling = false
+        @expected_tokens_in_buffer, @eof_handling = 3, false
       end
 
-      def control(cmd, par)
-        @mul_dic.report.each_pair { |key, value| set(key, value) } if cmd == STR_CMD_STATUS
-
-        # Jedes Control-Object ist auch Auslöser der Verarbeitung
-        if cmd == STR_CMD_RECORD || cmd == STR_CMD_EOF
-          @eof_handling = true
-          while number_of_valid_tokens_in_buffer > 1
-            process_buffer
-          end
-          forward_number_of_token( @buffer.size, false )
-          @eof_handling = false
-        end
-      end
-
-      def process_buffer?
-        number_of_valid_tokens_in_buffer >= @number_of_expected_tokens_in_buffer
+      def control(cmd, param)
+        control_multi(cmd, @mul_dic)
       end
 
       def process_buffer
-        unless @buffer[0].form == CHAR_PUNCT
-          # Prüfe 3er Schlüssel
-          result = check_multiword_key( 3 )
-          unless result.empty?
-            # 3er Schlüssel gefunden
-            lengths = sort_result_len( result )
-            unless lengths[0] > 3
-              # Längster erkannter Schlüssel = 3
-              create_and_forward_multiword( 3, result )
-              forward_number_of_token( 3 )
-              return
+        unless form_at(0) == CHAR_PUNCT
+          unless (res = check_multiword_key(3)).empty?
+            len = res.map { |r|
+              r.is_a?(Lexical) ? r.form.split(' ').size : r[/^\*(\d+)/, 1].to_i
+            }.sort!.reverse!
+
+            unless (max = len.first) > 3
+              create_and_forward_multiword(3, res)
+              forward_number_of_token(3)
             else
-              # Längster erkannter Schlüssel > 3, Buffer voll genug?
-              unless @buffer.size >= lengths[0] || @eof_handling
-                @number_of_expected_tokens_in_buffer = lengths[0]
-                return
+              unless @eof_handling || @buffer.size >= max
+                @expected_tokens_in_buffer = max
               else
-                # Buffer voll genug, Verarbeitung kann beginnen
-                catch( :forward_one ) do
-                  lengths.each do |len|
-                    result = check_multiword_key( len )
-                    unless result.empty?
-                      create_and_forward_multiword( len, result )
-                      forward_number_of_token( len )
-                      throw :forward_one
-                    end
-                  end
-
-                  # Keinen Match gefunden
-                  forward_number_of_token( 1 )
-                end
-
-                @number_of_expected_tokens_in_buffer = 3
+                forward_number_of_token(len.find { |l|
+                  r = check_multiword_key(l)
+                  create_and_forward_multiword(l, r) unless r.empty?
+                } || 1)
+
+                @expected_tokens_in_buffer = 3
                 process_buffer if process_buffer?
-                return
               end
             end
+
+            return
           end
 
-          # Prüfe 2er Schlüssel
-          result = check_multiword_key( 2 )
-          unless result.empty?
-            create_and_forward_multiword( 2, result )
-            forward_number_of_token( 1 )
+          unless (res = check_multiword_key(2)).empty?
+            create_and_forward_multiword(2, res)
+            forward_number_of_token(1)
           end
         end
 
-        # Buffer weiterschaufeln
-        forward_number_of_token( 1, false )
-        @number_of_expected_tokens_in_buffer = 3
+        forward_number_of_token(1, false)
+        @expected_tokens_in_buffer = 3
       end
 
       private
 
-      def create_and_forward_multiword( len, lexicals )
-        # Form aus Buffer auslesen und Teile markieren
-        pos = 0
-        form_parts = []
+      def create_and_forward_multiword(len, lex)
+        pos, parts = 0, []
+
         begin
-          if @buffer[pos].form == CHAR_PUNCT
-            @buffer.delete_at( pos )
-            form_parts[-1] += CHAR_PUNCT
+          if (form = form_at(pos)) == CHAR_PUNCT
+            @buffer.delete_at(pos)
+            parts[-1] += CHAR_PUNCT
           else
             @buffer[pos].attr = WA_UNKMULPART if @buffer[pos].unknown?
-            form_parts << @buffer[pos].form
+            parts << form
             pos += 1
           end
         end while pos < len
 
-        form = form_parts.join( ' ' )
-
-        # Multiword erstellen
-        word = Word.new( form, WA_MULTIWORD )
-        word << lexicals.collect { |lex| (lex.is_a?(Lexical)) ? lex : nil }.compact  # FIXME 1.60 - Ausstieg bei "*5" im Synonymer
-
-        # Forword Multiword
-        forward( word )
-      end
-
-      # Leitet 'len' Token weiter
-      def forward_number_of_token( len, count_punc = true )
-        begin
-          unless @buffer.empty?
-            forward( @buffer[0] )
-            len -= 1 unless count_punc && @buffer[0].form == CHAR_PUNCT
-            @buffer.delete_at( 0 )
-          end
-        end while len > 0
-      end
-
-      # Ermittelt die maximale Ergebnislänge
-      def sort_result_len( result )
-        result.collect do |res|
-          if res.is_a?( Lexical )
-            res.form.split( ' ' ).size
-          else
-            res =~ /^\*(\d+)/
-            $1.to_i
-          end
-        end.sort.reverse
+        forward(Word.new_lexicals(parts.join(' '),
+          WA_MULTIWORD, lex.select { |l| l.is_a?(Lexical) }))
       end
 
       # Prüft einen definiert langen Schlüssel ab Position 0 im Buffer
-      def check_multiword_key( len )
-        return [] if number_of_valid_tokens_in_buffer < len
+      def check_multiword_key(len)
+        return [] if valid_tokens_in_buffer < len
 
-        # Wortformen aus der Wortliste auslesen
-        sequence = @buffer.map { |obj|
+        seq = @buffer.map { |obj|
           next [obj] unless obj.is_a?(WordForm)
+          next if (form = obj.form) == CHAR_PUNCT
 
-          form = obj.form
-          next if form == CHAR_PUNCT
-
-          word = @lex_dic.find_word(form)
-          word = @lex_gra.find_compositum(form) if word.unknown?
-
-          lexicals = word.attr == WA_KOMPOSITUM ?
-            [word.lexicals.first] : word.lexicals.dup
-
-          lexicals << word if lexicals.empty?
-          lexicals += @syn_dic.find_synonyms(word) if @syn_dic
-
-          lexicals.map { |lex| lex.form }.uniq
-        }.compact[0, len]
-
-        if @combine
-          keys, muls = [], []
+          w = find_word(form, @lex_dic, @lex_gra)
+          l = w.lexicals
 
-          sequence.each { |forms|
-            keys = forms.map { |form|
-              keys.empty? ? form : keys.map { |key| "#{key} #{form}" }
-            }.flatten(1)
+          (w.attr == WA_COMPOUND ? [l.first] : l.empty? ? [w] : l.dup).tap { |i|
+            i.concat(@syn_dic.find_synonyms(w)) if @syn_dic
+            i.map! { |j| j.form.downcase }.uniq!
           }
+        }
 
-          keys.each { |key|
-            mul = @mul_dic.select(key.downcase)
-
-            unless mul.empty?
-              muls.concat(mul)
-              break unless @all_keys
-            end
-          }
+        seq.compact!
+        seq.slice!(len..-1)
 
-          muls.uniq
+        if @combine
+          [].tap { |mul| seq.shift.product(*seq) { |key|
+            mul.concat(@mul_dic.select(key.join(' ')))
+            break unless @all_keys || mul.empty?
+          } && mul.uniq! }
         else
-          key = sequence.map { |forms| forms.first }.join(' ')
-          @mul_dic.select(key.downcase)
+          @mul_dic.select(seq.map!(&:first).join(' '))
         end
       end
 
-      # Liefert die Anzahl gültiger Token zurück
-      def number_of_valid_tokens_in_buffer
-        @buffer.collect { |token| (token.form == CHAR_PUNCT) ? nil : 1 }.compact.size
-      end
-
     end
 
     # For backwards compatibility.