treetagger-ruby 0.0.1 → 0.1.0

data/CHANGELOG.rdoc

@@ -1,17 +1,27 @@
 == COMPLETED
+=== 0.1.0
+The inteface is now clear and stable.
+
+Tagging of big texts is possible since the TreeTagger is invoked as an
+external process through a pipe.
+
+Test suite improved.
 === 0.0.1
 Implemented simple tagging. The TreeTagger is invoked through the evn variable.
 === 0.0.1.prealpha
 Created the structure for this project, added documentation and a public repo.
 
 == PLANNED
-=== 0.1.0
-
 === 0.2.0
+Better tests. Support for all input types.
 === 0.3.0
+Lemmatizer.
 === 0.4.0
+File based FIFOs.
 === 0.5.0
+File based queues.
 === 0.6.0
+Full featured cmd interface.
 === 0.7.0
 === 0.8.0
 === 0.9.0

data/README.rdoc

@@ -1,37 +1,69 @@
 = TreeTagger for Ruby
 
 * {RubyGems}[http://rubygems.org/gems/treetagger-ruby]
-* Developers {Homepage}[http://bu.chsta.be/]
+* {Homepage}[http://bu.chsta.be/]
 * {RTT Project Page}[http://bu.chsta.be/projects/treetagger-ruby/]
 * {Source Code}[https://github.com/arbox/treetagger-ruby]
 * {Bug Tracker}[https://github.com/arbox/treetagger-ruby/issues]
 
 == DESCRIPTION
-The Ruby based wrapper for the TreeTagger by Helmut Schmid.
-Check it out if you are interested
-in Natural Language Processing (NLP) and Human Language Technology (HLT).
+A Ruby based wrapper for the TreeTagger by Helmut Schmid.
+
+Check it out if you are interested in Natural Language Processing (NLP)
+and/or Human Language Technology (HLT).
+
+This library provides comprehensive bindings for the
+{TreeTagger}[http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/],
+a statistical language independed POS tagging and chunking software.
+
+TreeTagger is language agnostic, it will never guess what language you're going
+to use. It 
+
+TODO:
+* References to Schmid's publications;
+* How to use TreeTagger in the wild;
+* Input and output format, tokenization;
+* ...
+* The actual german parameter file has been estimated on one byte encoded data.
+
 === Implemented Features
 Simple tagging.
 
+Please have a look at the CHANGELOG file for details on implemented and planned
+features. 
 
 == INSTALLATION
 Before you install the <tt>treetagger-ruby</tt> package please ensure
-you have downloaded and installe the <tt>TreeTagger</tt> itself.
+you have downloaded and installed the
+{TreeTagger}[http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/]
+itself.
 
 The {TreeTagger}[http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/]
-is a copyrighted software by Helmut Schmid and IMC, please read the license
-agreament befor you download the package.
+is a copyrighted software by Helmut Schmid and
+{IMS}[http://www.ims.uni-stuttgart.de/], please read the license
+agreament before you download the TreeTagger package and language models.
 
 After the installation of the <tt>TreeTagger</tt> set the environment variable
-<tt>TREETAGGERHOME</tt> to the location where you have the programm installed.
-Usually this directory contains subdirectories <tt>bin, cmd, lib</tt> and
-<tt>doc</tt>.
-For instance you may add the following line to your <tt>.profile</tt> file:
-  export TREETAGGERHOME='/path/to/your/TreeTagger/installation'
+<tt>TREETAGGER_BINARY</tt> to the location where the binary <tt>tree-tagger</tt>
+resides. Usually this binary is located under the <tt>bin</tt> directory in the
+main installation directory of the <tt>TreeTagger</tt>.
+
+Also you have to set the variable <tt>TREETAGGER_MODEL</tt> to the location of
+the appropriate language model you have acquired in the training step.
+
+For instance you may add the following lines to your <tt>.profile</tt> file:
+  export TREETAGGER_BINARY='/path/to/your/TreeTagger/bin/tree-tagger'
+  export TREETAGGER_MODEL='/path/to/your/TreeTagger/lib/german.par'
+
+It is convinient to work with a default language model, but you can change
+it every time during the instantiation of a new tagger instance.
+
+If you want to feed a lexicon file into your tagger you can do it globally
+through the environment variable <tt>TREETAGGER_LEXICON</tt>.
 
 <tt>treetagger-ruby</tt> is provided as a .gem package. Simply install it via
 {RubyGems}[http://rubygems.org/gems/treetagger-ruby].
-To install <tt>treetagger-ruby</tt> ussue the following command:
+To install <tt>treetagger-ruby</tt> issue the following command:
   $ gem install treetagger-ruby
 
 If you want to do a system wide installation, do this as root
@@ -41,14 +73,88 @@ Alternatively use your Gemfile for dependency management.
 
 
 == SYNOPSIS
-
+=== Basic Usage
 Basic usage is very simple:
   $ require 'treetagger-ruby'
+  $ # Instantiate a tagger instance with default values.
   $ tagger = TreeTagger::Tagger.new
-  $ api.process('Ich gehe in die Schule')
+  $ # Process an array of tokens.
+  $ tagger.process(%w{Ich gehe in die Schule})
+  $ # Flush the pipeline.
+  $ tagger.flush
+  $ # Get the processed data.
+  $ tagger.get_output
+
+=== Input Format
+Basically you have to provide a tokenized sequence with possibly some additional
+information on lexical classes of tokens and on their probabilities. Every token
+has to be on a separate line. Due to technical limitations SGML tags
+(i.e. sequences with heading < and trailing >) cannot be valid tokes since
+they are used internally for delimiting meaningful content from flush tokens.
+It implies the use of the <tt>-sgml</tt> option which cannot be changes by user.
+It is a limitation of <em>this</em> library. If you do need to process tags,
+fall back and use the TreeTagger as a standalone programm possibly employing
+temp files to store your input and output. This behaviour will be also
+implemented in futher versions of <tt>treetagger-ruby</tt>.
+
+Every token may occure alone on the line or be followed by additional
+information:
+* token;
+* token (\\tab tag)+;
+* token (\\tab tag \\space lemma)+;
+* token (\\tab tag \\space probability)+;
+* token (\\tab tag \\space probability \\space lemma)+.
+
+You input may look like the following sentence:
+  Die     ART 0.99
+  neuen   ADJA neu
+  Hunde  NN NP
+  stehen  VVFIN 0.99 stehen
+  an
+  den
+  Mauern  NN Mauer
+  .
+
+
+This wrapper accepts the input as <em>String</em> or <em>Array</em>.
+
+If you want to use strings, you are responsible for the proper delimiters inside
+the string: <tt>"Die\\tART 0.99\\nneuen\\tADJA neu\\nHunde\\tNN NP\\nstehen\\t
+VVFIN 0.99 stehen\\nan\\nden\\nMauern\\tNN Mauer\\n.\\n"</tt>
+Now <tt>treetagger-ruby</tt> does not check your markup for correctness and will
+possibly report a <tt>TreeTagger::ExternalError</tt> if the TreeTagger process
+die due to input errors.
+
+Using arrays is more convinient since they can be built programmatically.
+
+Arrays should have the following structure:
+* ['token', 'token', 'token'];
+* ['token', ['token', ['POS', 'lemma'], ['POS', 'lemma']], 'token'];
+* ['token', ['token', ['POS', prob], ['POS', 'prob']], 'token'];
+* ['token', ['token', ['POS', prob, 'lemma'], ['POS', 'prob', 'lemma']]].
+
+It is internally converted in the sequence <tt>token\\ntoken\\tPOS lemma\\t
+POS lemma\\ntoken\\n</tt>, i.e. in the enriched version alternatives are
+tab separated and entries a blank separated.
+
+Note that probabilities may be strings or integers.
+
+The lexicon lookup is +not+ implemented for now, that's the latter three forms
+of input arrays are not supported yet.
+
+=== Output Format
+For now you'll get an array with strings elements. However the precise string
+structure depends on the cmd arguments you've provided during the tagger
+instantiation.
+
+For instanse for the input <tt>["Veruntreute", "die", "AWO", "Spendengeld", "?"]
+</tt> you'll get the following output with default cmd argumetns:
+
+<tt>["Veruntreute\tNN\tVeruntreute", "die\tART\td", "AWO\tNN\t<unknown>",
+"Spendengeld\tNN\tSpendengeld", "?\t$.\t?"]</tt>
 
 See documentation in the TreeTagger::Tagger class for details
-on particular search methods.
+on particular methods.
 
 == EXCEPTION HIERARCHY
 While using TreeTagger you can face following errors:
@@ -56,10 +162,23 @@ While using TreeTagger you can face following errors:
 * <tt>TreeTagger::RuntimeError</tt>;
 * <tt>TreeTagger::ExternalError</tt>.
 
+This three kinds of errors all subclass <tt>TreeTagger::Error</tt>, which
+in turn is a subclass of <tt>StandardError</tt>. For an end user this means that
+it is possible to intercept all errors from <em>treetagger-ruby</em> with
+a simple <tt>rescue</tt> clause.
+
 == SUPPORT
-If you have question, bug reports or any suggestions, please drop me an email :) 
-Any help is deeply appreciated!
+If you have question, bug reports or any suggestions, please drop me an email :)
 
+== HOW TO CONTRIBUTE
+Please contact me and suggest your ideas, report bugs, talk to me, if you want
+to implement some features in the future releases of this library.
+
+Please don't feel offended if I cannot accept all your pull requests, I have
+to review them and find the appropriate time and place in the code base to
+incorporate your valuable changes.
+
+Any help is deeply appreciated!
 == CHANGELOG
 For details on future plan and working progress see CHANGELOG.
 

data/bin/rtt

@@ -8,24 +8,51 @@ options = TreeTagger::ARGVParser.parse(ARGV)
 
 tagger = TreeTagger::Tagger.new(options)
 
-while line = ARGF.gets
-  # [['token', 'tag', 'lemma'], ['token', 'tag', 'lemma']]
-  result_array = tagger.process(line.chomp)
-
-  # Adding some colors to the output.
-  # Using ANSI escape codes.
-  red   = "\e[31m"
-  green = "\e[32m"
-  blue  = "\e[34m"
-  reset = "\e[0m"
-  
-  result_array.each do |tuple|
-    if $stdout.tty?
-      tuple[0].insert(0, red).insert(-1, reset)
-      tuple[1].insert(0, green).insert(-1, reset)
-      tuple[2].insert(0, blue).insert(-1, reset)
+# Adding some colors to the output.
+# Using ANSI escape codes.
+red   = "\e[31m"
+green = "\e[32m"
+blue  = "\e[34m"
+reset = "\e[0m"
+
+reader = Thread.new do
+  beginning = true
+  loop do
+    result_array = tagger.get_output
+    if result_array.nil?
+      if beginning
+        sleep(0.1)
+        next
+      else
+        break
+      end
+    end
+    sleep(0.2) # Is useful!
+
+    beginning = false
+    result_array.each do |tuple|
+      tuple = tuple.split("\t")
+      
+      if $stdout.tty?
+        tuple[0].insert(0, red).insert(-1, reset)   if tuple[0]
+        tuple[1].insert(0, green).insert(-1, reset) if tuple[1]
+        tuple[2].insert(0, blue).insert(-1, reset)  if tuple[2]
+      end
+      
+      # [['token', 'tag', 'lemma'], ['token', 'tag', 'lemma']]`    
+      $stdout.puts tuple.join("\t")
     end
-    
-    $stdout.puts tuple.join("\t")
   end
 end
+
+# Read all lines from STDOUT or from files.
+while line = ARGF.gets
+  # Invoke tokenizer somehow here.
+  tagger.process(line)
+end
+
+tagger.flush
+
+reader.join
+
+STDOUT.flush

data/lib/tree_tagger/tagger.rb

@@ -1,22 +1,228 @@
 # -*- encoding: utf-8 -*-
+require 'thread'
+require 'tree_tagger/error'
 
+=begin
+TODO:
+- Observe the status of the reader thread.
+- Control the status of the pipe and recreate it.
+- Handle IO errors.
+- Handle errors while allocating the TT object.
+- Update the flush sentence, make it shorter.
+- Store the queue on a persistant medium, not in the memory.
+- Properly set the $ORS for all platforms.
+=end
+# :main: README.rdoc
+# :title: TreeTagger - Ruby based Wrapper for the TreeTagger by Helmut Schmid
+# Module comment
 module TreeTagger
+  # Class comment
   class Tagger
-    def initialize(
-                   lang = :de,
-                   opts = {
-                     :sgml => true,
-                     :token => true,
-                     :lemma => true
+
+    BEGIN_MARKER = '<BEGIN_OF_THE_TT_INPUT>'
+    END_MARKER   = '<END_OF_THE_TT_INPUT>'
+    # TT seems to hold only the last three tokens in the buffer.
+    # The flushing sentence can be shortened down to this size.
+    FLUSH_SENTENCE = "Das\nist\nein\nTestsatz\n,\num\ndas\nStossen\nder\nDaten\nsicherzustellen\n."
+    
+    # Initializer commet
+    def initialize(opts = {
+                     :binary => nil,
+                     :model => nil,
+                     :lexicon => nil,
+                     :options => '-token -lemma -sgml -quiet',
+                     :replace_blanks => true,
+                     :blank_tag => '<BLANK>',
+                     :lookup => false
                    }
                    )
-      @lang = lang
-      @opt = opts
+
+      @opts = validate_options(opts)
+      @blank_tag = @opts[:blank_tag]
+      @cmdline = "#{@opts[:binary]} #{@opts[:options]} #{@opts[:model]}"
+
+      @queue = Queue.new
+      @pipe = new_pipe
+      @pipe.sync = true
+      @reader = new_reader
+      @inside_output = false
+      @inside_input = false
+      @enqueued_tokens = 0
+      @mutex = Mutex.new
+      @queue_mutex = Mutex.new
+      # sleep(1) # Don't know if it's useful, no problems before.
+    end
+
+    # Send the string to the TreeTagger.
+    def process(input)
+
+      str = convert(input)
+      # Sanitize strings.
+      str = sanitize(str)
+      # Mark the beginning of the text.
+      if not @inside_input
+        str = "#{BEGIN_MARKER}\n#{str}\n"
+        @inside_input = true
+      else
+        str = str + "\n"
+      end
+      @mutex.synchronize { @enqueued_tokens += 1 }
+      @pipe.print(str)
     end
-    def process(str)
-      line = %x(echo '#{str}' | #{ENV['TREETAGGERHOME']}/cmd/tree-tagger-german)
-      arr = line.split("\n").collect { |el| el.split("\t") }
+
+    # Get processed tokens back.
+    # This method is not blocking. If some tokens have been sent,
+    # but not received from the pipe yet, it returns an empty array.
+    # If all sent tokens are in the queue it returns all of them.
+    # If no more tokens are awaited it returns <nil>.
+    def get_output
+      output = []
+      tokens = 0
+      @queue_mutex.synchronize do
+        tokens = @queue.size
+        tokens.times { output << @queue.shift }
+      end
+      @mutex.synchronize do
+        @enqueued_tokens -= tokens
+      end
+
+      # Nil if nothing to process in the pipe.
+      # Possible only after flushing the pipe.
+      if @enqueued_tokens > 0
+        output
+      else
+        output.any? ? output : nil
+      end
+    end
+    
+    # Get the rest of the text back.
+    # TT holds some meaningful parts in the buffer.
+    def flush
+      @inside_input = false
+      str = "#{END_MARKER}\n#{FLUSH_SENTENCE}\n"
+      @pipe.print(str)
+      # Here invoke the reader thread to ensure
+      # all output has been read.
+      #@reader.run
+    end
+    
+    private
+    # Return the options hash after validation.
+    #  {
+    #    :binary => nil,
+    #    :model => nil,
+    #    :lexicon => nil,
+    #    :options => '-token -lemma -sgml -quiet',
+    #    :replace_blanks => true,
+    #    :blank_tag => '<BLANK>',
+    #    :lookup => false
+    #  }
+    def validate_options(opts)
+      # Check if <:lookup> is boolean.
+
+      # Check if <:replace_blanks> is boolean.
+
+      # Check if <:options> is a string.
+
+      # Check if <:options> contains only allowed values.
+
+      # Ensure that <:options> contains <-sgml>.
+
+      # Check if <:blank_tag> is a string.
+
+      # Ensure that <:blank_tag> is a valid SGML sequence.
+      
+      # Set the model and binary paths if not provided.
+      [:binary, :model].each do |key|
+        if opts[key].nil?
+          opts[key] = ENV.fetch("TREETAGGER_#{key.to_s.upcase}") do |missing|
+            fail UserError, "Provide a value for <:#{key}>" +
+              " or set the environment variable <#{missing}>!"
+          end
+        end
+      end
+      
+      # Set the lexicon path if not provided but requested.
+      if opts[:lookup] && opts[:lexicon].nil?
+        opts[:lookup] = ENV.fetch('TREETAGGER_LEXICON') do |missing|
+          fail UserError, 'Provide a value for <:lexicon>' +
+            ' or set the environment variable <TREETAGGER_LEXICON>!'
+        end
+      end
+
+      # Check for existence and reedability of external files:
+      # * binary;
+      # * model;
+      # * lexicon (if applicable).
+      
+      opts
+    end
+    
+    # Starts the reader thread.
+    def new_reader
+      Thread.new do
+        while line = @pipe.gets
+          # The output strings must not contain "\n".
+          line.chomp!
+          case line
+          when BEGIN_MARKER
+            @inside_output = true
+            $stderr.puts 'Found the begin marker.' if $DEBUG
+          when END_MARKER
+            @inside_output = false
+            $stderr.puts 'Found the end marker.' if $DEBUG
+          else
+            if @inside_output
+              @queue_mutex.synchronize { @queue << line }
+              $stderr.puts "<#{line}> added to the queue." if $DEBUG
+            end
+          end
+        end
+      end # thread
+    end # start_reader
+    
+    # This method may be utilized to keep the TT process alive.
+    # Check here if TT returns the exit code 1 in case on invalide options.
+    def new_pipe
+      IO.popen(@cmdline, 'r+')
+    end
+
+    # Convert token arrays to delimited strings.
+    def convert(input)
+      unless input.is_a?(Array) || input.is_a?(String)
+        fail UserError, "Not a valid input format: <#{input.class}>!"
+      end
+
+      if input.empty?
+        fail UserError, "Empty input is not allowed!"
+      end
+      
+      if input.is_a?(Array)
+        input.each do |el|
+          unless el.is_a?(String)
+            fail UserError, "Input elements should be strings!"
+          end
+          el = sanitize(el)
+        end
+        input = input.join("\n")
+      end
+      
+      input
+    end
+    
+    def sanitize(str)
+      line = str.strip
+      if line.empty?
+        line = @blank_tag
+      end
+
+      line
     end
   end # class
 end # module
 
+__END__
+- tokenization
+- lexicon lookup
+- tagging
+- error correction

data/lib/tree_tagger/version.rb

@@ -1,3 +1,3 @@
 module TreeTagger
-  VERSION = '0.0.1'
+  VERSION = '0.1.0'
 end

data/test/test_tagger.rb

@@ -0,0 +1,154 @@
+require 'test/unit'
+require 'tree_tagger/tagger'
+require 'tree_tagger/error'
+require 'stringio'
+
+class TestTagger < Test::Unit::TestCase
+
+  PUBLIC_METHODS = [:process,
+                    :get_output,
+                    :flush
+                   ]
+  def setup
+#    ENV['TREETAGGER_BINARY'] = '/opt/TreeTagger/bin/tree-tagger'
+#    ENV['TREETAGGER_MODEL'] = '/opt/TreeTagger/lib/german.par'
+#    ENV['TREETAGGER_LEXICON'] = '/opt/TreeTagger/lib/german-lexicon.txt'
+
+    ENV['TREETAGGER_BINARY'] = 'test/tree-tagger/tree-tagger'
+    ENV['TREETAGGER_MODEL'] = 'test/tree-tagger/model_file.par'
+    ENV['TREETAGGER_LEXICON'] = 'test/tree-tagger/lexicon_file.txt'
+    
+    params = {} # dummy for now
+    @tagger = TreeTagger::Tagger.new
+  end
+  
+  def teardown
+  end
+
+  # It should have the following constants set.
+  def test_constants
+  end
+
+  # It should respond to valid methods
+  def test_public_methods
+    PUBLIC_METHODS.each do |m|
+      assert_respond_to(@tagger, m)
+    end
+  end
+
+  def test_tagger
+  end
+  
+  # It should accept only arrays and strings.
+  def test_input_for_its_class
+    assert_nothing_raised do
+      @tagger.process 'Ich\ngehe\nin\ndie\nSchule\n.\n'
+      @tagger.process %w{Ich gehe in die Schule .}
+    end
+  end
+
+  # It should reject non-string and non-array elements.
+  def test_rejecting_invalid_input
+    [{}, :input, 1, 1.0, Time.new].each do |input|
+      assert_raise(TreeTagger::UserError) do
+        @tagger.process(input)
+      end
+    end
+  end
+  
+  # It should reject empty input.
+  def test_for_empty_input
+    ['', []].each do |input|
+      assert_raise(TreeTagger::UserError) do
+        @tagger.process(input)
+      end
+    end
+  end
+
+  # It should reject arrays with wrong elements.
+  def test_for_elements_of_arrays
+    
+  end
+  
+  # It should accept valid input.
+  def test_accepting_vaild_input
+    input = ''
+  end
+  
+  # It should accept only valid input.
+  def test_input_validity
+    ['', [], {}, :input, [:one, :two]].each do |input|
+      assert_raise(TreeTagger::UserError) do
+        @tagger.process(input)
+      end
+    end
+  end
+
+  # It should instantiate a tagger instance only with valid options.
+  def test_for_binary_presence
+    ENV.delete('TREETAGGER_BINARY')
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new
+    end
+  end
+
+  # It should instantiate a tagger instance only with valid options.
+  def test_for_model_presence
+    ENV.delete('TREETAGGER_MODEL')
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new
+    end
+
+  end
+
+  # It should instantiate a tagger instance only with valid options.
+  def test_for_lexicon_presence
+    ENV.delete('TREETAGGER_LEXICON')
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new({:lookup => true, :options => '-quiet -sgml'})
+    end
+  end
+
+  # It should reject a non-boolean value for <:lookup>.
+  def test_rejecting_lookup_values
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new({:lookup => 'true', :options => '-quiet'})
+    end
+  end
+
+  # It should reject a non-boolean value for <:replace_blanks>.
+  def test_rejecting_blank_values
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new({:replace_blanks => 'true'})
+    end
+  end
+
+  # It should reject a non-string value for <:options>.
+  def test_rejecting_option_values
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new({:options => :quiet})
+    end
+  end
+
+  # It should reject invalid options for TreeTagger inside <:options>.
+  def test_rejecting_invalid_arguments
+    flunk 'Not implemented yet!'
+  end
+
+  # It should ensure the presense of the <-sgml> argument.
+  def test_presence_of_sgml_argument
+    flunk 'Not implemented yet!'
+  end
+
+  # It should reject a non-string value for <:blank_tag>.
+  def test_rejecting_blanktag_values
+    assert_raise(TreeTagger::UserError) do
+      TreeTagger::Tagger.new({:blank_tag => :blank})
+    end
+  end
+
+  # It should ensure that <:blang_tag> is a valid smgl sequence.
+  def test_sgml_form
+    flunk 'Not implemented yet!'
+  end
+end

data/test/tree-tagger/tree-tagger

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+
+ARGV.clear
+while gets
+  puts $_
+end
+
+#STDOUT.flush

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: treetagger-ruby
 version: !ruby/object:Gem::Version 
-  hash: 29
+  hash: 27
   prerelease: 
   segments: 
   - 0
-  - 0
   - 1
-  version: 0.0.1
+  - 0
+  version: 0.1.0
 platform: ruby
 authors: 
 - Andrei Beliankou
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-12-18 00:00:00 Z
+date: 2012-02-14 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: rdoc
@@ -95,6 +95,12 @@ files:
 - LICENCE.rdoc
 - CHANGELOG.rdoc
 - .yardopts
+- test/test_tagger.rb
+- test/tree-tagger/corrupted_lexicon_file.txt
+- test/tree-tagger/lexicon_file.txt
+- test/tree-tagger/corrupted_model_file.par
+- test/tree-tagger/model_file.par
+- test/tree-tagger/tree-tagger
 - bin/rtt
 homepage: http://www.uni-trier.de/index.php?id=34451
 licenses: []
@@ -132,6 +138,11 @@ rubygems_version: 1.8.10
 signing_key: 
 specification_version: 3
 summary: A wrapper for the TreeTagger by Helmut Schmid.
-test_files: []
-
+test_files: 
+- test/test_tagger.rb
+- test/tree-tagger/corrupted_lexicon_file.txt
+- test/tree-tagger/lexicon_file.txt
+- test/tree-tagger/corrupted_model_file.par
+- test/tree-tagger/model_file.par
+- test/tree-tagger/tree-tagger
 has_rdoc: