shalmaneser-fred 1.2.0.rc4 → 1.2.rc5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e1795de4d92cea5dee25e6840fc1080161aa1d6e
-  data.tar.gz: 8933ad415fc12fef76184e68e28757b2c6f79ec5
+  metadata.gz: 6f7ca2f794d9024633383d41b2092b9f250197bf
+  data.tar.gz: d8bc88eeda007ca746e39565709b5e3fd414f58e
 SHA512:
-  metadata.gz: 7efd1551dc7e902b2fed0dd717f9eb0b9ac7aa2c010ab2bf91472934f612c066b254175f7feac7d885f8953a8979872203c8f0d6eb040253949aea0090b98eb6
-  data.tar.gz: 4b46a404e0400483233cb196b3f2a41759db2c98936062a86a097bd404a7759884b4046b5f62f6223ad56d7c62599204238fdcf8e4852f2df59f091faf776822
+  metadata.gz: 0c81db071bbee52c8aa337cd9dfccbb7190cbc156cc2f38c580c2647196093b5e08b49170d891762ca1c0965a5e59695371a75a2b63a93cda179fd4f47ced219
+  data.tar.gz: e3e07f5a64ba00e6d1eca438a1106cd3a40b89167e0c318e7184c56448d7a6940a2bd0210773ff325989a2277555f016a6ecceb55f50943f10ce3b96bb0531c4

data/README.md

@@ -1,4 +1,4 @@
-# [SHALMANESER - a SHALlow seMANtic parSER](http://www.coli.uni-saarland.de/projects/salsa/shal/)
+# SHALMANESER
 
 [RubyGems](http://rubygems.org/gems/shalmaneser) |
 [Shalmanesers Project Page](http://bu.chsta.be/projects/shalmaneser/) |
@@ -7,9 +7,9 @@
 
 
 [![Gem Version](https://img.shields.io/gem/v/shalmaneser.svg")](https://rubygems.org/gems/shalmaneser)
-[![Gem Version](https://img.shields.io/gem/v/frprep.svg")](https://rubygems.org/gems/frprep)
-[![Gem Version](https://img.shields.io/gem/v/fred.svg")](https://rubygems.org/gems/fred)
-[![Gem Version](https://img.shields.io/gem/v/rosy.svg")](https://rubygems.org/gems/rosy)
+[![Gem Version](https://img.shields.io/gem/v/frprep.svg")](https://rubygems.org/gems/shalmaneser-prep)
+[![Gem Version](https://img.shields.io/gem/v/fred.svg")](https://rubygems.org/gems/shalmaneser-fred)
+[![Gem Version](https://img.shields.io/gem/v/rosy.svg")](https://rubygems.org/gems/shalmaneser-rosy)
 
 
 [![License GPL 2](http://img.shields.io/badge/License-GPL%202-green.svg)](http://www.gnu.org/licenses/gpl-2.0.txt)
@@ -17,12 +17,44 @@
 [![Code Climate](https://img.shields.io/codeclimate/github/arbox/shalmaneser.svg")](https://codeclimate.com/github/arbox/shalmaneser)
 [![Dependency Status](https://img.shields.io/gemnasium/arbox/shalmaneser.svg")](https://gemnasium.com/arbox/shalmaneser)
 
+[SHALMANESER](http://www.coli.uni-saarland.de/projects/salsa/shal/) is a SHALlow seMANtic parSER.
+
+The name Shalmaneser is borrowed from John Brunner. He describes in his novel
+"Stand on Zanzibar" an all knowing supercomputer baptized Shalmaneser.
+
+Shalmaneser also has other origins like the king [Shalmaneser III](https://en.wikipedia.org/wiki/Shalmaneser_III).
+
+> "SCANALYZER is the one single, the ONLY study of the news in depth
+> that’s processed by General Technics’ famed computer Shalmaneser,
+> who sees all, hears all, knows all save only that which YOU, Mr. and Mrs.
+> Everywhere, wish to keep to yourselves." <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
+> But Shalmaneser is a Micryogenic® computer bathed in liquid helium and it’s cold in his vault. <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
+> “Of course not. Shalmaneser’s main task is to achieve the impossible again, a routine undertaking here at GT.” <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
+> “They programmed Shalmaneser with the formula for this stiffener, see, and…” <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
+> What am I going to do now? <br/>
+> “All right, Shalmaneser!” <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
+> Shalmaneser is a Micryogenic® computer bathed in liquid helium and there’s no sign of Teresa. <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
+> Bathed in his currents of liquid helium, self-contained, immobile, vastly well informed by every mechanical sense: Shalmaneser. <br/>
+> John Brunner (1968) "Stand on Zanzibar"
+
 ## Description
 
 Please be careful, the whole thing is under construction! For now Shalmaneser it not intended to run on Windows systems since it heavily uses system calls for external invocations.
 Current versions of Shalmaneser have been tested on Linux only (other *NIX testers are welcome!).
 
-Shalmaneser is a supervised learning toolbox for shallow semantic parsing, i.e. the automatic assignment of semantic classes and roles to text. This technique is often called SRL (Semantic Role Labelling). The system was developed for Frame Semantics; thus we use Frame Semantics terminology and call the classes frames and the roles frame elements. However, the architecture is reasonably general, and with a certain amount of adaption, Shalmaneser should be usable for other paradigms (e.g., PropBank roles) as well. Shalmaneser caters both for end users, and for researchers.
+Shalmaneser is a supervised learning toolbox for shallow semantic parsing, i.e. the automatic assignment of semantic classes and roles to text. This technique is often called [SRL](https://en.wikipedia.org/wiki/Semantic_role_labeling) (Semantic Role Labelling). The system was developed for Frame Semantics; thus we use Frame Semantics terminology and call the classes frames and the roles frame elements. However, the architecture is reasonably general, and with a certain amount of adaption, Shalmaneser should be usable for other paradigms (e.g., PropBank roles) as well. Shalmaneser caters both for end users, and for researchers.
 
 For end users, we provide a simple end user mode which can simply apply the pre-trained classifiers
 for [English](http://www.coli.uni-saarland.de/projects/salsa/shal/index.php?nav=download) (FrameNet 1.3 annotation / Collins parser)
@@ -34,32 +66,27 @@ For researchers interested in investigating shallow semantic parsing, our system
 
 ## Origin
 
-The original version of Shalmaneser was written by Sebastian Padó, Katrin Erk and others during their work in the SALSA Project.
+The original version of Shalmaneser was written by Sebastian Padó, Katrin Erk, Alexander Koller, Ines Rehbein, Aljoscha Burchardt and others during their work in the SALSA Project.
 
 You can find original versions of Shalmaneser up to ``1.1`` on the [SALSA](http://www.coli.uni-saarland.de/projects/salsa/shal/) project page.
 
 ## Publications on Shalmaneser
 
 - K. Erk and S. Padó: Shalmaneser - a flexible toolbox for semantic role assignment. Proceedings of LREC 2006, Genoa, Italy. [Click here for details](http://www.nlpado.de/~sebastian/pub/papers/lrec06_erk.pdf).
+
 - TODO: add other works
 
 ## Documentation
 
-The project documentation can be found in our [doc](https://github.com/arbox/shalmaneser/blob/1.2/doc/index.md) folder.
+The project documentation can be found in our [doc](https://github.com/arbox/shalmaneser/blob/master/doc/index.md) folder.
 
 ## Development
 
-We are working now on two branches:
-
-- ``dev`` - our development branch incorporating actual changes, for now pointing to ``1.2``;
-
-- ``1.2`` - intermediate target;
-
-- ``2.0`` - final target.
+We are working now only on the `master` branch. For different intermediate versions see corresponding tags.
 
 ## Installation
 
-See the installation instructions in the [doc](https://github.com/arbox/shalmaneser/blob/1.2/doc/index.md#installation) folder.
+See the installation instructions in the [doc](https://github.com/arbox/shalmaneser/blob/master/doc/index.md#installation) folder.
 
 ### Tokenizers
 
@@ -75,7 +102,7 @@ See the installation instructions in the [doc](https://github.com/arbox/shalmane
 
 ### Parsers
 
-- [BerkeleyParser](https://code.google.com/p/berkeleyparser/downloads/list)
+- [BerkeleyParser](https://github.com/slavpetrov/berkeleyparser)
 - [Stanford Parser](http://nlp.stanford.edu/software/lex-parser.shtml)
 - [Collins Parser](http://www.cs.columbia.edu/~mcollins/code.html)
 
@@ -86,8 +113,10 @@ See the installation instructions in the [doc](https://github.com/arbox/shalmane
 
 ## License
 
-See the `LICENSE` file.
+Shalmaneser is released under the `GPL v. 2.0` license as of the initial authors.
+
+For a local copy of the full license text see the [LICENSE](LICENSE.md) file.
 
 ## Contributing
 
-See the `CONTRIBUTING` file.
+Feel free to contact me via Github. Open an issue if you see problems or need help.

data/bin/fred

@@ -10,7 +10,12 @@
 require 'fred/opt_parser'
 require 'fred/fred'
 
-options = Fred::OptParser.parse(ARGV)
+begin
+  options = ::Shalmaneser::Fred::OptParser.parse(ARGV)
 
-fred = Fred::Fred.new(options)
-fred.assign
+  fred = ::Shalmaneser::Fred::Fred.new(options)
+  fred.assign
+rescue => e
+  $stderr.puts 'Fred cannot serve you!'
+  $stderr.puts e.message, e.backtrace
+end

data/lib/fred/FredConventions.rb

@@ -4,229 +4,230 @@
 # several small things that should be uniform
 # throughout the system
 
-require "common/ruby_class_extensions"
+require 'monkey_patching/file.rb'
+require 'fred/answer_key_access'
 
-require "common/EnduserMode"
-class Object
+module Shalmaneser
+module Fred
 
-###
-# joining and breaking up senses
-def fred_join_senses(senses)
-  return senses.sort().join("++")
-end
-
-def fred_split_sense(joined_senses)
-  return joined_senses.split("++")
-end
+  module_function
 
-###
-# fred_dirname
-#
-# constructs a directory name:
-# fred data directory / experiment ID / maindir / subdir
-#
-# if is_existing == existing, the directory is checked for existence,
-# if is_existing == new, it is created if necessary
-#
-# returns: a string
-def fred_dirname(exp,             # FredConfigData object
-                 maindir,         # string: main part of directory name
-                 subdir,          # string: subpart of directory name
-                 is_existing = "existing")  # string: "existing" or "new", default: existing
-
-  case is_existing
-  when "existing"
-    return File.existing_dir(exp.get("fred_directory"),
-                         exp.get("experiment_ID"),
-                         maindir,
-                         subdir)
-  when "new"
-    return File.new_dir(exp.get("fred_directory"),
-                         exp.get("experiment_ID"),
-                         maindir,
-                         subdir)
-  else
-    raise "Shouldn't be here: #{is_existing}"
-  end
-end
+  def determine_training_senses(lemma, exp, lemmas_and_senses_obj, split_id)
+    if split_id
+      # oh no, we're splitting the dataset into random training and test portions.
+      # this means that we actually have to look into the training part of the data to
+      # determine the number of training senses
 
-####
-# filenames for feature files
-def fred_feature_filename(lemma, sense = nil, 
-			  do_binary = false)
-  if do_binary
-    return "fred.features.#{lemma}.SENSE.#{sense}"
-  else
-    return "fred.features.#{lemma}"
-  end
-end
+      senses_hash = {}
 
-####
-# filenames for split files
-def fred_split_filename(lemma)
-  return "fred.split.#{lemma}"
-end
+      reader = AnswerKeyAccess.new(exp, "train", lemma, "r", split_id, "train")
+      reader.each do |_lemma, _pos, _ids, _sids, gold_senses, _transf_gold_senses|
+        gold_senses.each { |s| senses_hash[s] = true }
+      end
 
-###
-# deconstruct split filename
-# returns: lemma
-def deconstruct_fred_split_filename(filename)
-  basename = File.basename(filename)
-  if basename =~ /^fred\.split\.(.*)/
-    return $1
-  else
-    return nil
-  end
-end
+      return senses_hash.keys
 
-###
-# deconstruct feature file name
-# returns: hash with keys
-# "lemma"
-# "sense
-def deconstruct_fred_feature_filename(filename)
-
-  basename = File.basename(filename)
-  retv = Hash.new()
-  # binary: 
-  # fred.features.#{lemma}.SENSE.#{sense}
-  if basename =~ /^fred\.features\.(.*)\.SENSE\.(.*)$/
-    retv["lemma"] = $1
-    retv["sense"] = $2
-  elsif basename =~ /^fred\.features\.(.*)/
-    # fred.features.#{lemma}
-    retv["lemma"] = $1
-
-  else
-    # complete mismatch
-    return nil
+    else
+      # we're using separate test data.
+      # so we can just look up the number of training senses
+      # in the lemmas_and_senses object
+      senses = lemmas_and_senses_obj.get_senses(lemma)
+      if senses
+        return senses
+      else
+        return []
+      end
+    end
   end
 
-  return retv
-end
 
-####
-# filename for answer key files
-def fred_answerkey_filename(lemma)
-  return "fred.answerkey.#{lemma}"
-end
+  ###
+  # joining and breaking up senses
+  # @note Used only in FredFeatures.
+  def fred_join_senses(senses)
+    senses.sort.join("++")
+  end
 
-###
-# classifier directory
-def fred_classifier_directory(exp,     # FredConfigData object
-                              splitID = nil) # string or nil
+  # @note Used only in FredEval.
+  def fred_split_sense(joined_senses)
+    joined_senses.split("++")
+  end
 
-  if exp.get("classifier_dir")
-    # user-specified classifier directory
+  ###
+  # fred_dirname
+  #
+  # @note Used on multiple positions.
+  # constructs a directory name:
+  # fred data directory / experiment ID / maindir / subdir
+  #
+  # if is_existing == existing, the directory is checked for existence,
+  # if is_existing == new, it is created if necessary
+  #
+  # @return [String]
+  def fred_dirname(exp,             # FredConfigData object
+                   maindir,         # string: main part of directory name
+                   subdir,          # string: subpart of directory name
+                   is_existing = "existing")  # string: "existing" or "new", default: existing
+
+    case is_existing
+    when "existing"
+      return File.existing_dir(exp.get("fred_directory"),
+                               exp.get("experiment_ID"),
+                               maindir,
+                               subdir)
+    when "new"
+      return File.new_dir(exp.get("fred_directory"),
+                          exp.get("experiment_ID"),
+                          maindir,
+                          subdir)
+    else
+      raise "Shouldn't be here: #{is_existing}"
+    end
+  end
 
-    if splitID
-      return File.new_dir(exp.get("classifier_dir"), splitID)
+  ####
+  # filenames for feature files
+  # @note Used on multiple points.
+  def fred_feature_filename(lemma, sense = nil,
+                            do_binary = false)
+    if do_binary
+      return "fred.features.#{lemma}.SENSE.#{sense}"
     else
-      return File.new_dir(exp.get("classifier_dir"))
+      return "fred.features.#{lemma}"
     end
+  end
 
-  else
-    # my classifier directory
-    if splitID
-      return fred_dirname(exp, "classifiers", splitID, "new")
+  ###
+  # deconstruct split filename
+  # returns: lemma
+  # @note Not used anywhere.
+  def deconstruct_fred_split_filename(filename)
+    basename = File.basename(filename)
+    if basename =~ /^fred\.split\.(.*)/
+      return $1
     else
-      return fred_dirname(exp, "classifiers", "all", "new")
+      return nil
     end
   end
-end
 
-###
-# classifier file
-def fred_classifier_filename(classifier, lemma, sense=nil)
-  if sense
-    return "fred.classif.#{classifier}.LEMMA.#{lemma}.SENSE.#{sense}"
-  else
-    return "fred.classif.#{classifier}.LEMMA.#{lemma}"
+  ###
+  # deconstruct feature file name
+  # returns: hash with keys
+  # "lemma"
+  # "sense
+  # @note Used only in FredFeatures.
+  def deconstruct_fred_feature_filename(filename)
+    basename = File.basename(filename)
+    retv = {}
+
+    # binary:
+    # fred.features.#{lemma}.SENSE.#{sense}
+    if basename =~ /^fred\.features\.(.*)\.SENSE\.(.*)$/
+      retv["lemma"] = $1
+      retv["sense"] = $2
+    elsif basename =~ /^fred\.features\.(.*)/
+      # fred.features.#{lemma}
+      retv["lemma"] = $1
+
+    else
+      # complete mismatch
+      return nil
+    end
+
+    return retv
   end
-end
 
-def deconstruct_fred_classifier_filename(filename)
-  retv = Hash.new()
-  if filename =~ /^fred\.classif\.(.*)\.LEMMA\.(.*)\.SENSE\.(.*)$/
-    retv["lemma"] = $2
-    retv["sense"] = $3
-  elsif filename =~ /^fred\.classif\.(.*)\.LEMMA\.(.*)$/
-    retv["lemma"] = $2
+  ####
+  # filename for answer key files
+  # @note Used only in FredFeatures.
+  def fred_answerkey_filename(lemma)
+    return "fred.answerkey.#{lemma}"
   end
-  return retv
-end
 
-###
-# result file
-def fred_result_filename(lemma)
-  return "fred.result.#{lemma.gsub(/\./, "_")}"
-end
+  ###
+  # classifier directory
+  # @note Used on multiple points.
+  def fred_classifier_directory(exp,     # FredConfigData object
+                                splitID = nil) # string or nil
 
-##########
-# lemma and POS: combine into string separated by 
-# a separator character
-#
-# fred_lemmapos_combine: take two strings, return combined string
-#      if POS is nil, returns lemma<separator character>
-# fred_lemmapos_separate: take one string, return two strings
-#      if no POS could be retrieved, returns nil as POS and the whole string as lemma
-def fred_lemmapos_combine(lemma, # string
-			  pos)   # string
-  return lemma.to_s + "." + pos.to_s.gsub(/\./, "DOT")
-end
+    if exp.get("classifier_dir")
+      # user-specified classifier directory
 
-###
-def fred_lemmapos_separate(lemmapos)  # string
-  pieces = lemmapos.split(".")
-  if pieces.length() > 1
-	return [ pieces[0..-2].join("."), pieces[-1] ]
-  else
-    # no POS found, treat all of lemmapos as lemma
-    return [ lemmapos, nil ]
-  end
-end
-end
+      if splitID
+        return File.new_dir(exp.get("classifier_dir"), splitID)
+      else
+        return File.new_dir(exp.get("classifier_dir"))
+      end
 
-########################################
-# given a SynNode object representing a terminal,
-# return:
-# - the word
-# - the lemma
-# - the part of speech
-# - the named entity (if any)
-#
-# as a tuple
-#
-# WARNING: word and lemma are turned to lowercase
-module WordLemmaPosNe
-  def word_lemma_pos_ne(syn_obj, # SynNode object
-                        i)       # SynInterpreter class
-    unless syn_obj.is_terminal?
-      $stderr.puts "Featurization warning: unexpectedly received non-terminal"
-      return [ nil, nil, nil, nil ]
+    else
+      # my classifier directory
+      if splitID
+        return fred_dirname(exp, "classifiers", splitID, "new")
+      else
+        return fred_dirname(exp, "classifiers", "all", "new")
+      end
     end
+  end
 
-    word = syn_obj.word()
-    if word
-      word.downcase!
+  ###
+  # classifier file
+  # @note Used on multiple points.
+  def fred_classifier_filename(classifier, lemma, sense = nil)
+    if sense
+      return "fred.classif.#{classifier}.LEMMA.#{lemma}.SENSE.#{sense}"
+    else
+      return "fred.classif.#{classifier}.LEMMA.#{lemma}"
     end
+  end
 
-    lemma = i.lemma_backoff(syn_obj)
-    if lemma and SalsaTigerXMLHelper.unescape(lemma) == "<unknown>"
-      lemma = nil
-    end
-    if lemma
-      lemma.downcase!
+  # @note Used only in Baseline.
+  def deconstruct_fred_classifier_filename(filename)
+    retv = {}
+
+    if filename =~ /^fred\.classif\.(.*)\.LEMMA\.(.*)\.SENSE\.(.*)$/
+      retv["lemma"] = $2
+      retv["sense"] = $3
+    elsif filename =~ /^fred\.classif\.(.*)\.LEMMA\.(.*)$/
+      retv["lemma"] = $2
     end
 
-    pos = syn_obj.part_of_speech()
+    retv
+  end
 
-    ne = syn_obj.get_attribute("ne")
-    unless ne
-      ne = syn_obj.get_attribute("headof_ne")
-    end
+  ###
+  # result file
+  # @note Used on multiple points.
+  def fred_result_filename(lemma)
+    "fred.result.#{lemma.gsub(/\./, "_")}"
+  end
 
-    return [word, lemma, pos, ne]
+  ##########
+  # lemma and POS: combine into string separated by
+  # a separator character
+  #
+  # fred_lemmapos_combine: take two strings, return combined string
+  #      if POS is nil, returns lemma<separator character>
+  # fred_lemmapos_separate: take one string, return two strings
+  #      if no POS could be retrieved, returns nil as POS and the whole string as lemma
+  # @param lemma [String]
+  # @param pos [String]
+  # @note Used on multiple points.
+  def fred_lemmapos_combine(lemma, pos)
+    lemma.to_s + "." + pos.to_s.gsub(/\./, "DOT")
   end
-end
 
+  ###
+  # @param lemmapos [String]
+  # @note Used only in FredDetermineTargets.
+  def fred_lemmapos_separate(lemmapos)
+    pieces = lemmapos.split(".")
+
+    if pieces.length > 1
+      return [pieces[0..-2].join("."), pieces[-1]]
+    else
+      # no POS found, treat all of lemmapos as lemma
+      return [lemmapos, nil]
+    end
+  end
+end
+end