treat 0.2.0 → 0.2.5

data/TODO

@@ -1,6 +1,8 @@
 ## Todo
 
 - Chronic, ruby date
+- YAML in folder
+- Tag remove sentence
 - Stanford/Enju phrase in phrase
 - Date, time, roles structs
 - test wiki
@@ -19,6 +21,7 @@
 - Enju as a server
 - More information with categories
 - Gist languages convert
+-
 
 ## Personal
 

data/lib/economist/hungarys_troubles.txt

@@ -0,0 +1,46 @@
+Hungary's troubles
+Not just a rap on the knuckles
+
+THE pressure is piling up on the beleaguered Hungarian government. Today the European Commission threatened it with legal action over several new "cardinal" laws that would require a two-thirds majority in parliament to overturn.
+
+The commission is still considering the laws, but today it highlighted concerns over three issues:
+
+ - The independence of the central bank. Late last year the Hungarian parliament passed a law which expands the monetary council and takes the power to nominate deputies away from the governor and hands it to the prime minister. A separate law opens the door to a merger between the bank and the financial regulator.
+
+ - The judiciary. More than 200 judges over the age of 62 have been forced into retirement and hundreds more face the sack. The new National Judicial Authority is headed by Tünde Handó, a friend of the family of Viktor Orban, the prime minister.
+
+ - The independence of the national data authority.
+
+That wasn't all the commission had to say today. Hungary also received a ticking-off from Olli Rehn (pictured), the economic-affairs commissioner, for not doing enough to tackle its budget deficit. It may now lose access to EU funds.
+
+Slammed in Brussels, the Hungarian government is also under pressure at home. Earlier this week Gordon Bajnai, who served as Socialist prime minister from 2009-10, fired off a broadside that sent shockwaves through the political and media establishments.
+
+After a year and a half of government by the right-wing Fidesz party, wrote Mr Bajnai in a lengthy article on the website of the Patriotism and Progress Public Policy Foundation, democracy has been destroyed in Hungary. The country, he warned, is scarred by division and is drifting towards bankruptcy and away from Europe.
+
+Mr Bajnai called for a radical change of government and a complete political re-orientation. “A new government must have a programme readily at hand that can be applied without delay: a programme that promotes the republic, reconciliation, and recovery.” 
+
+Fidesz is rattled by Mr Bajnai, who since leaving office has been teaching at Columbia University in New York. Understandably so. He headed a technocratic administration which stabilised the economy. Unlike his Socialist predecessor, Ferenc Gyurcsany, he was neither part of the old Communist elite nor connected to it by marriage, and so cannot be smeared as a "Komcsi". He is modern in outlook and well regarded internationally.
+
+Moreover, say those how know him, Mr Bajnai has little patience for the narcissistic exceptionalism that shapes Fidesz’s worldview. Exhibit A: the plaintive cry of Janos Martonyi, the foreign minister, who lamented recently: “The world will never understand our pains and spiritual wounds.” Such self-pity is unlikely to endear the Hungarian government to Brussels or Washington DC (to where it has sent an envoy this week to negotiate with the IMF).
+
+Fidesz won a two-thirds majority in 2010. But its support is evaporating, and analysts say there is a gap in the political market for a centrist pro-business party committed to democratic norms. Mr Bajnai, who has not ruled out a return to politics, would be an obvious candidate to lead it.
+
+Meanwhile, as Hungarians watch the value of their assets vaporise, in large part thanks to the government’s increasingly erratic policies, Mr Orban smirks his way through press conferences. Here he is dodging questions from a reporter from HVG, an economics weekly, about his responsibility for the crisis and trying to shift the blame to his old enemy Andras Simor, president of the central bank. The interview ran as follows:
+
+hvg.hu: Do you feel responsible for the falling/weakening forint?
+
+Mr Orban: You mean the president of the central bank? He did not comment on it.
+
+hvg.hu: No, you, Mr prime minister!
+
+Mr Orban: The personal responsibility of the president of the central bank was not discussed over the meeting.
+
+hvg.hu: You, your personal…!
+
+Mr Orban: That neither.
+
+Surrounded by yes-men and grinning flunkies, Mr Orban seems increasingly out of touch. His future will likely be decided not in the gilded corridors of the Hungarian parliament, but in Brussels and Washington DC.
+
+What happens next? If his hand is forced Mr Orban can probably endure policy reversals on the independence of the central bank and the data ombudsman. Sorry, he would say to his loyal followers: national crisis, what can you do.
+
+The dismantling of the judiciary would be another matter. If outsiders keep up the pressure and the judicial changes are judged to be in breach of the EU treaty, Mr Orban would be in a tricky spot. It’s hard to see how he could declare the 200-plus judges his government has forced into retirement ready for office after all, and still sit in his own.

data/lib/economist/indias_slowdown.txt

@@ -0,0 +1,15 @@
+India’s slowdown
+The case for the defence
+Why officials think investors are too bearish about India’s economy
+
+THE SMOG is so bad in Delhi right now that it seeps indoors. In one government building the far end of the corridor seems hazy. But the view of the mandarin working there is clear: India’s economic miracle is not over, regardless of the chatter among investors and howls about government paralysis from industrialists. He pokes fun at the latter. A year ago they were swanning around Davos proclaiming India could grow in its sleep, he says. Now, with growth dipping to 6.9% last quarter, from a peak of 10% (see chart), they are pleading for government action.
+
+Bears in Mumbai, India’s financial capital, worry that GDP growth might slip below 6% as confidence and investment slip. That partly reflects global woes, and partly too the gumming up of the bureaucracy due to a wave of graft allegations. But it is also because no big reforms have taken place for years; and such is the dire state of India’s politics that it is hard to imagine any being imminent. Things reached a nadir at the end of last year when the ruling coalition announced it would allow foreign supermarkets into the country, only to do a U-turn in the face of protests from the opposition and its own coalition partners. Shortly afterwards it failed to carry a key anti-corruption bill through parliament.
+
+The government is not blind to these concerns—in his new-year address the prime minister, Manmohan Singh, conceded that “it would be wrong to conclude that India is now unshakeably set on a process of rapid growth.” But officials in Delhi are more optimistic than the financial markets, for three reasons. First, they argue that growth is bottoming out. Inflation is showing signs of falling, which should allow the central bank to reverse its long series of interest-rate hikes. The recent drop in the rupee is a healthy adjustment, not cause for panic, they say. Meanwhile the euro zone is vaguely getting its act together and there are hints of a recovery in America. Growth, it is thought, will be about 7% for the fiscal year ending in March, respectable enough, and will pick up from there.
+
+Second, the long-term drivers of India’s boom are intact. There is “not much reason to change your mind,” says the mandarin. The rise in the savings rate, which allows more investment, will continue, partly thanks to a demographic bulge of people reaching working age. Even if there is a drop in capital expenditure, it should remain above 30% of GDP—a “handsome level”, says another official, that will boost the country’s potential. The government’s 12th five-year plan, which is due out soon, was originally expected to forecast growth of 9% between 2012 and 2017. That might fall to 8.5%, officials say, but no further.
+
+The final strut of the argument is that the politics are not as bad as they seem. Pessimists, the mandarin says, reckon “we’re just going to fiddle around and miss our opportunities.” But after important elections in February in Uttar Pradesh, the most populous state, the politicians may stop posturing and even co-operate to pass less contentious reforms such as a new national value-added tax. That would cut red tape and the fiscal deficit. And even if parliament stays gridlocked, there are lots of nuts-and-bolts reforms that do not require legislation. The government will try harder to tackle the bottlenecks that choke the power industry, for instance, and the paperwork that is snarling up big projects.
+
+The nub of the official argument is “calm down—and trust us to do just enough.” The trouble is the government has been saying this for a year, and business folk and investors seem to have lost heart. Firms have cut investment and the stockmarket was one of the world’s worst performers in dollar terms last year. Perhaps they are being too jumpy, but India does not have the luxury of dismissing what firms and investors think. The fiscal deficit, including the states and off-balance-sheet items, is running at 9-10% of GDP for the fourth year in a row. The current-account deficit is drifting towards 4% of GDP, officials admit, well above the country’s traditional comfort zone. India needs to command the confidence of domestic and foreign investors. Unless the reform process starts moving there is a risk that the financing of these deficits will become an acute problem—and that India’s economic miracle recedes further into the Delhi haze.

data/lib/economist/merkozy_rides_again.txt

@@ -0,0 +1,24 @@
+Saving the euro, part 473
+Merkozy rides again
+
+ANGELA MERKEL and Nicolas Sarkozy kicked off the 2012 season of the euro soap opera with a summit meeting in Berlin today. Neither said anything startling; certainly nothing that would betoken a swift and happy conclusion to the long-running saga.
+
+The German chancellor and the French president muted their differences over such issues as how quickly to introduce a tax on financial transactions and what the role of the European Central Bank (ECB) should be in supporting shaky members of the euro zone. “Our analysis is the same,” said Mr Sarkozy at the post-summit press conference.
+
+This did not calm markets’ nerves. The euro dropped to its lowest level against the dollar since September 2010 ($1.266) before the summit and recovered marginally as the two leaders met. Currency traders’ biggest worry is Greece’s failure to meet its fiscal targets, which means it may not get the fresh money it needs to avoid defaulting on its debt.
+
+At the opposite end of the confidence spectrum, investors are so eager to finance Germany that they accepted a negative interest rate on an auction of six-month paper, in effect paying Germany’s government for the privilege of lending to it. Germans will see this as vindication of their prudent policies, but it also serves to underline the dangerous economic divergences within the euro zone.
+
+The main significance of the Merkozy summit is that it seemed to signal a shift in emphasis. True, the austerity agenda—promoted by the Germans and grudgingly accepted by the French—is still there. Indeed, Mr Sarkozy boasted that France’s fiscal deficit was smaller than expected in 2011. Europe is making swift progress towards a “fiscal pact” to limit deficits, proclaimed Mrs Merkel, including German-style “debt brakes”. A new treaty should be signed by March.
+
+But fiscal self-denial will now be supplemented by what Mrs Merkel called a “second leg”, meaning economic growth and job creation. This is partly meant to help Mr Sarkozy, who faces a tough re-election fight this spring.
+
+All euro-zone countries, including Germany, are “prepared to do their homework” in this area, the chancellor promised, but it is not clear that much new is on offer. A big German stimulus package to boost growth in neighbouring countries is not in prospect (that would nobble the fiscal leg).
+
+Mrs Merkel spoke of spreading best practice in labour-market regulation across the euro zone (which is German practice, Mr Sarkozy admits) and spending existing European funds more quickly and effectively. Both ideas make sense; neither will prevent further financial turmoil, or a European recession. In the latest sign of fragility, German industrial production dropped 1% in November.
+
+The leaders tried to seem anything but complacent. Mr Sarkozy called the situation “very tense” and Mrs Merkel said they had “understood the needs of the hour.” The intention is to keep Greece from dropping out of the euro zone, but whatever happens Greece is an exceptional case, the leaders said (perhaps fearing that a Greek default or even an exit from the euro could not be avoided). As always, the chancellor dampened expectations of a quick “one-dimensional” solution to the crisis. The problem would be solved, she said, “step by step.”
+
+The next steps involve Italy, an indebted giant that poses a far greater threat to the euro than Greece. Mrs Merkel will meet Italy’s unelected prime minister, Mario Monti, in Berlin on Wednesday; she and Mr Sarkozy will hold a three-way summit with him in Rome on January 20th. European heads of government are to gather, probably on January 30th, to put the finishing touches to the fiscal pact.
+
+Also on the agenda, no doubt, will be a proposed financial-transactions tax. Britain is threatening a veto; Mr Sarkozy has said France will go it alone at first, if need be. Mrs Merkel wants the tax but her junior coalition partner, the Free Democrats, do not unless the British get on board. As the crisis sharpens, disagreements are likely to re-emerge over the role of the ECB and how to strengthen the euro zone's bail-out funds. The soap opera has a long way to run.

data/lib/economist/prada_is_not_walmart.txt

@@ -0,0 +1,9 @@
+Prada is not Walmart
+
+INDIA, if you believe the government, will be a land in which Starbucks and Prada thrive but where foreign firms will be prohibited from selling onions. It does not seem like much of a cause for celebration, but the announcement on January 11th that foreign “single brand” retailers could own 100% of their operations in India was meant to show the reform process was on track. It followed a debacle late last year when the government first announced that not only would single brand retailing be opened up, but foreign supermarkets would be allowed to operate in India too—and then was quickly forced into a U-turn on the latter promise after facing a rebellion within its own ranks and from the coalition parties it relies on in parliament.
+
+By emphasising that at least the single brand bit of retail reform is still on track, the government hopes to show the world that India is still open for business. But this is a meek change indeed. Single brand retailers, such as fashion chains, were already allowed to own 51% of their operations. And the political stink of last month is likely to scare those who are not already present because swathes of the political class have been shown to be populist and hostile for foreign firms. Individual states may still choose to override the central government’s rules. Lastly, the reform comes with a large catch: 30% of what is sold must be supplied from cottage industries in India. If you are selling a uniform product worldwide—a sofa or handbag made in China—that is a major hassle.
+
+The hope must be that India is on a journey to the right place, stumbling along the way. Perhaps the supplier rule will eventually be dropped, the argument goes. Maybe reluctant states will learn the error of their ways and open up too, after seeing the success of single brand retailers in other states. And maybe, after seeing an influx of investment from single brand retailers, the political climate will change and it will be easier to pass a reform that lets in supermarkets in too.
+
+Interviewed in Delhi earlier in January a government mandarin insisted that the supermarket reform was not dead. Yet all of this seems half hearted. India is a hard enough place as it is for foreign firms to make profits. Adding in a fickle polity just makes things worse. And it is a rather sorry day for progress when a rule tweak to allow Starbucks or Prada to own not 51%, but 100%, of their shops is presented as a meaningful economic reform.

data/lib/treat/buildable.rb

@@ -81,7 +81,7 @@ module Treat
       end
       Treat::Entities::Number.new(numeric.to_s)
     end
-    def from_folder(folder)
+    def from_folder(folder, exclude = ['cfs'])
       unless FileTest.directory?(folder)
         raise Treat::Exception,
         "Path '#{folder}' does not point to a folder."
@@ -95,11 +95,15 @@ module Treat
         "Cannot create something else than a " +
         "collection from folder '#{folder}'."
       end
-      c = Treat::Entities::Collection.new
+      c = Treat::Entities::Collection.new(folder)
       folder += '/' unless folder[-1] == '/'
       Dir[folder + '*'].each do |f|
-        next if FileTest.directory?(f)
-        c << Treat::Entities::Document.from_file(f)
+        if FileTest.directory?(f)
+          c2 = Treat::Entities::Collection.from_folder(f)
+          c << c2
+        else
+          c << Treat::Entities::Document.from_file(f)
+        end
       end
       c
     end
@@ -127,6 +131,8 @@ module Treat
           else
             from_raw_file(file)
           end
+        elsif ext == 'cfs'
+          return
         else
           from_raw_file(file)
         end

data/lib/treat/categories.rb

@@ -36,7 +36,8 @@ module Treat
     require 'treat/formatters'
     require 'treat/processors'
     require 'treat/lexicalizers'
-    require 'treat/extractors'
     require 'treat/inflectors'
+    require 'treat/extractors'
+    require 'treat/retrievers'
   end
 end

data/lib/treat/delegatable.rb

@@ -68,6 +68,7 @@ module Treat
         end
         if group.type == :annotator
           f = postprocessor.nil? ? m : postprocessor
+          
           entity.features[f] = result unless result == nil
         end
         result
@@ -101,7 +102,7 @@ module Treat
       self.find_worker_for_language(entity.language, group) :
       group.default
       if worker == :none
-        raise NAT::Exception,
+        raise Treat::Exception,
         "There is intentionally no default worker for #{group}."
       end
       worker

data/lib/treat/doable.rb

@@ -18,6 +18,7 @@ module Treat
         end
       end
     end
+    DEBUG = true
     def do_task(task, worker, options)
       group = Categories.lookup(task)
       unless group
@@ -25,8 +26,8 @@ module Treat
       end
       entity_types = group.targets
       f = nil
-      entity_types.each do |t| 
-        f = true if Treat::Entities.match_types[type][t]
+      entity_types.each do |t|
+        f = true if Treat::Entities.match_types[t][type]
       end
       if f || entity_types.include?(:entity)
         send(task, worker, options)

data/lib/treat/entities/collection.rb

@@ -4,18 +4,11 @@ module Treat
     class Collection < Entity
       # Initialize the collection with a folder
       # containing the texts of the collection.
-      def initialize(folder = nil, id = nil)
+      def initialize(folder = nil)
         super('', id)
         @type = :collection
-        if folder
-          set :folder, folder
-          Dir.glob("#{folder}/*").each do |f|
-            next if FileTest.directory?(f)
-            self << Document.new(f)
-          end
-        end
+        set :folder, folder
       end
-      def type; :collection; end
     end
   end
 end

data/lib/treat/entities/entity.rb

@@ -33,6 +33,7 @@ module Treat
         id ||= object_id
         super(value, id)
         @type = :entity
+        # @match_types = Treat::Entities.match_types
       end
       # Catch missing methods to support method-like
       # access to features (e.g. entity.categoryinstead of
@@ -114,7 +115,7 @@ module Treat
             e.send($2.intern) == args[0]
           end
           a
-        elsif method =~ /^#{@@entities_regexp}s_with_([a-z]*)$/
+        elsif method =~ /^#{@@entities_regexp}_with_([a-z]*)$/
           a = []
           each_entity($1.intern) do |e|
             a << e if e.has?($2.intern) &&
@@ -187,21 +188,9 @@ module Treat
       #
       # This function NEEDS to be ported to C (see source).
       def each_entity(*types)
-=begin  
-     # Replace with:
-    inline do |builder|
-
-      builder.c_raw <<-EOS, :arity => -1
-      VALUE each_entity_c(int argc, VALUE *types, VALUE self)
-      {
-
-      }
-      EOS
-    end
-=end
         types = [:entity] if types.size == 0
         f = false
-        types.each { |t2| f = true if Treat::Entities.match_types[type][t2] }
+        types.each { |t2| f = true if Treat::Entities.match_types[t2][type] }
         yield self if f
         unless @children.size == 0
           @children.each do |child|
@@ -209,6 +198,16 @@ module Treat
           end
         end
       end
+
+      # Replace with:
+      #inline do |builder|
+        #  
+       # builder.c_raw <<-EOS, :arity => -1
+
+
+
+        #EOS
+      #end
       # Returns the first ancestor of this entity that has the given type.
       def ancestor_with_types(*types)
         ancestor = @parent

data/lib/treat/entities.rb

@@ -46,11 +46,12 @@ module Treat
       list = (Treat::Entities.list + [:entity])
       @@match_types = {}
       list.each do |type1|
-        @@match_types[type1] = {type1 => true}
         list.each do |type2|
-          if Treat::Entities.const_get(cc(type1)) <
-            Treat::Entities.const_get(cc(type2))
-            @@match_types[type1][type2] = true
+          @@match_types[type2] ||= {}
+          if (type1 == type2) || 
+            (Treat::Entities.const_get(cc(type1)) < 
+            Treat::Entities.const_get(cc(type2)))
+            @@match_types[type2][type1] = true
           end
         end
       end

data/lib/treat/extractors/coreferences/stanford.rb

@@ -5,6 +5,7 @@ module Treat
         require 'stanford-core-nlp'
         @@pipeline = nil
         def self.coreferences(entity, options = {})
+          val = entity.to_s
           if entity.has_children?
             warn "The Stanford Coreference Resolver currently requires " +
             "an unsegmented, untokenized block of text to work with. " +

data/lib/treat/extractors/topic_words/lda.rb

@@ -54,21 +54,8 @@ module Treat
             lda.load_vocabulary(options[:vocabulary])
           end
 
-          # Get the topic words and annotate the section.
-          topic_words = lda.top_words(options[:words_per_topic])
-
-          collection.each_word do |word|
-            topic_words.each do |i, words|
-              if words.include?(word)
-                word.set :is_topic_word?, true
-                word.set :topic_id, i
-              else
-                word.set :is_topic_word?, false
-              end
-            end
-          end
-
-          topic_words
+          # Get the topic words.
+          lda.top_words(options[:words_per_topic])
         end
       end
     end

data/lib/treat/formatters/readers/autoselect.rb

@@ -22,7 +22,6 @@ module Treat
           begin
             r = Treat::Formatters::Readers.const_get(cc(reader))
           rescue NameError
-            puts e.message
             raise Treat::Exception,
             "Cannot find a reader for format: '#{ext}'."
           end

data/lib/treat/formatters/unserializers/yaml.rb

@@ -6,6 +6,8 @@ module Treat
       class YAML
         # Require the Psych YAML parser.
         require 'psych'
+        # Require date to revive DateTime.
+        require 'date'
         # Unserialize a YAML file.
         # 
         # Options: none.

data/lib/treat/formatters.rb

@@ -31,13 +31,6 @@ module Treat
       self.targets = [:entity]
       self.default = :tree
     end
-    # Cleaners strip a text from its mark up.
-    module Cleaners
-      extend Group
-      self.type = :transformer
-      self.targets = [:document]
-      self.default = :html
-    end
     extend Treat::Category
   end
 end

data/lib/treat/group.rb

@@ -24,7 +24,7 @@ module Treat
           return @method if @method
           m = ucc(cl(self))
           if m[-3..-1] == 'ers'
-            if ['k', 't', 'm', 'd', 'g', 'n'].include? m[-4]
+            if ['k', 't', 'm', 'd', 'g', 'n', 'x', 'h'].include? m[-4]
               n = m[0..-4]
               n = n[0..-2] if n[-1] == n[-2]
             else
@@ -81,7 +81,9 @@ module Treat
     end
     # Get constants in this module, excluding those
     # defined by parent modules.
-    def const_get(const); super(const, false); end
+    def const_get(const)
+      super(const, false)
+    end
     # Lazy load the classes in the group.
     def const_missing(const)
       bits = self.ancestors[0].to_s.split('::')

data/lib/treat/languages/english.rb

@@ -20,7 +20,7 @@ module Treat
         :chunkers => [:txt],
         :parsers => [:stanford, :enju],
         :segmenters => [:tactful, :punkt, :stanford],
-        :tokenizers => [:macintyre, :multilingual, :perl, :punkt, :stanford, :tactful]
+        :tokenizers => [:macintyre, :multilingual, :perl, :punkt, :tactful, :stanford]
       }
       
       Lexicalizers = {

data/lib/treat/lexicalizers/tag/brill.rb

@@ -21,7 +21,7 @@ module Treat
       # http://rbtagger.rubyforge.org/
       # Original Perl module site:
       # http://search.cpan.org/~kwilliams/Lingua-BrillTagger-0.02/lib/Lingua/BrillTagger.pm
-      class Brill < Tagger
+      class Brill
         patch = false
         # Require the 'rbtagger' gem.
         require 'rbtagger'
@@ -50,9 +50,8 @@ module Treat
         end
         # Hold the tagger.
         @@tagger = nil
-        # Hold the user-set options
-        @@options = {}
         # Tag words using a native Brill tagger.
+        # Performs own tokenization.
         #
         # Options:
         #
@@ -60,24 +59,27 @@ module Treat
         # :lexical_rules => String (Lexical rule file to use)
         # :contextual_rules => String (Contextual rules file to use)
         def self.tag(entity, options = {})
-          r = super(entity, options)
-          return r if r && r != :isolated_word
-          # Reinitialize the tagger if the options have changed.
-          @@tagger = nil if options != @@options
+          if entity.has_children?
+            warn "The Brill tagger performs its own tokenization. " +
+                 "Removing all children of #{entity.type} with value #{entity.short_value}."
+            entity.remove_all!
+          end
           # Create the tagger if necessary
           @@tagger ||= ::Brill::Tagger.new(options[:lexicon],
           options[:lexical_rules], options[:contextual_rules])
-          words = (r == :isolated_word) ? [entity] : entity.tokens
-          res = @@tagger.tag(words.join(' '))[1..-1]
+          res = @@tagger.tag(entity.to_s)
           res ||= []
+          isolated_word = entity.is_a?(Treat::Entities::Token)
           res.each do |info|
-            words.each do |word|
-              if word.value == info[0]
-                word.set :tag_set, :penn
-                word.set :tag, info[1]
-                return info[1] if r == :isolated_word
-              end
+            next if info[1] == ')'
+            token = Treat::Entities::Token.from_string(info[0])
+            token.set :tag_set, :penn
+            token.set :tag, info[1]
+            if isolated_word
+              entity.set :tag_set, :penn
+              return info[1]
             end
+            entity << token
           end
           entity.set :tag_set, :penn
           return 'P' if entity.is_a?(Treat::Entities::Phrase)

data/lib/treat/lexicalizers/tag/lingua.rb

@@ -15,7 +15,7 @@ module Treat
       # Project website: http://engtagger.rubyforge.org/
       # Original Perl module site:
       # http://cpansearch.perl.org/src/ACOBURN/Lingua-EN-Tagger-0.15/
-      class Lingua < Tagger
+      class Lingua
         # Require the 'engtagger' gem.
         silence_warnings { require 'engtagger' }
         # Hold the tagger.
@@ -38,9 +38,11 @@ module Treat
         #   particularly words used polysemously.
         # - (String) :unknown_word_tag => Tag for unknown words.
         def self.tag(entity, options = {})
+          if !entity.has_children?
+            warn "The Lingua tagger requires prior tokenization."
+            warn "Tokenizing the entity #{entity.short_value}."
+          end
           options = DefaultOptions.merge(options)
-          r = super(entity, options)
-          return r if r && r != :isolated_word
           # Reinitialize the tagger if the options have changed.
           if options != @@options
             @@options = DefaultOptions.merge(options)
@@ -48,15 +50,18 @@ module Treat
           end
           @@tagger ||= ::EngTagger.new(@@options)
           left_tag = @@tagger.conf[:current_tag] = 'pp'
-          tokens = (r == :isolated_word) ? [entity] : entity.tokens
-          tokens.each do |token|
+          isolated_word = entity.is_a?(Treat::Entities::Token)
+          entity.tokens.each do |token|
             w = @@tagger.clean_word(token.to_s)
             t = @@tagger.assign_tag(left_tag, w)
             t = options[:unknown_word_tag] if t.nil? || t == ''
             @@tagger.conf[:current_tag] = left_tag = t
             token.set :tag, t.upcase
             token.set :tag_set, :penn
-            return t.upcase if r == :isolated_word
+            if isolated_word
+              entity.set :tag_set, :penn
+              return t.upcase
+            end
           end
           entity.set :tag_set, :penn
           return 'P' if entity.is_a?(Treat::Entities::Phrase)

data/lib/treat/lexicalizers/tag/stanford.rb

@@ -1,7 +1,7 @@
 module Treat
   module Lexicalizers
     module Tag
-      class Stanford < Tagger
+      class Stanford
         require 'stanford-core-nlp'
         # Hold one tagger per language.
         @@taggers = {}
@@ -21,54 +21,46 @@ module Treat
         def self.tag(entity, options = {})
           # Handle options and set models.
           options = DefaultOptions.merge(options)
-          r = super(entity, options)
-          return r if r && r != :isolated_word
+          if entity.has_children?
+            warn "The Stanford tagger performs its own tokenization." +
+                 "Removing all children of #{entity.type} with value #{entity.short_value}."
+            entity.remove_all!
+          end
           # Arrange options.
           lang = entity.language
-          @@tag_set = LanguageToTagSet[lang]
-          unless @@tag_set
-            warn "The tag set for the tagger you are requiring is not supported."
-          end
-          
-          if options[:tagger_model]
-            ::StanfordCoreNLP.set_model(
-              'pos.model', options[:tagger_model]
-            )
-          end
-          if options[:silence]
-            options[:log_to_file] = '/dev/null'
-          end
-          if options[:log_to_file]
-            ::StanfordCoreNLP.log_file = 
-              options[:log_to_file] 
-          end
+          tag_set = LanguageToTagSet[lang]
+          warn "The tag set for the Stanford tagger you are requiring is not supported." unless tag_set
+          ::StanfordCoreNLP.set_model('pos.model', options[:tagger_model]) if options[:tagger_model]
+          options[:log_to_file] = '/dev/null' if options[:silence]
+          ::StanfordCoreNLP.log_file = options[:log_to_file]  if options[:log_to_file]
           
           # Load the tagger.
           StanfordCoreNLP.use(lang)
           @@taggers[lang] ||= ::StanfordCoreNLP.load(:tokenize, :ssplit, :pos)
+          
           # Tag the text.
           text = ::StanfordCoreNLP::Text.new(entity.to_s)
+          isolated_word = entity.is_a?(Treat::Entities::Token)
           @@taggers[lang].annotate(text)
-          # Realign the tags.
-          entity.each_token do |t1|
-            text.get(:sentences).each do |sentence|
-              sentence.get(:tokens).each do |t2|
-                if t2.value == t1.value
-                  tag = t2.get(:part_of_speech).to_s
-                  tag_s, tag_opt = *tag.split('-')
-                  tag_s ||= ''
-                  t1.set :tag, tag_s
-                  t1.set :tag_opt, tag_opt
-                  t1.set :tag_set, @@tag_set if @@tag_set
-                  return tag_s if r == :isolated_word
-                  break
-                end
-              end
+          
+          text.get(:tokens).each do |token|
+            val = token.get(:value).to_s
+            tok = Treat::Entities::Token.from_string(val)
+            tag = token.get(:part_of_speech).to_s
+            tag_s, tag_opt = *tag.split('-')
+            tag_s ||= ''
+            tok.set :tag, tag_s
+            tok.set :tag_opt, tag_opt
+            tok.set :tag_set, tag_set if tag_set
+            if isolated_word
+              entity.set :tag_set, :penn
+              return tag_s
             end
+            entity << tok
           end
           
           # Handle tags for sentences and phrases.
-          entity.set :tag_set, @@tag_set if @@tag_set
+          entity.set :tag_set, tag_set if tag_set
           return 'P' if entity.is_a?(Treat::Entities::Phrase)
           return 'S' if entity.is_a?(Treat::Entities::Sentence)
         end

data/lib/treat/lexicalizers.rb

@@ -6,9 +6,8 @@ module Treat
     # Taggers return the part of speech tag of a word.
     module Tag
       extend Group
-      require 'treat/lexicalizers/tag/tagger'
       self.type = :annotator
-      self.targets = [:word]
+      self.targets = [:sentence, :phrase, :token]
     end
     
     # Return the general category of a word.