parts 0.1.0 → 0.1.1

data/README.md

@@ -0,0 +1,75 @@
+# Parts: a probabilistic part of speech tagger
+
+Parts is a simple to use probabilistic part of speech tagger. At its core, parts is an adapted [Viterbi](http://en.wikipedia.org/wiki/Viterbi_algorithm) [bi-gram](http://en.wikipedia.org/wiki/Bigram) classifier. As such it looks to classify a sentence's parts of speech by identifying the most probable sequence of tags, given the sentence's words. This is done by training the tagger with a pre-tagged corpora. By default the tagger is trained using the [Treebank 3](http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC99T42) corpora. 
+
+Any questions please do get in contact via [email](mailto:joe@onlysix.co.uk).
+
+## Basics use
+
+Parts is packaged as a [gem](https://rubygems.org/pages/download) and thus installed accordingly. 
+
+	gem install parts
+
+With the gem installed, we must first `require` it within any code making use of it.
+
+	require 'parts'
+
+In order to create a tagger with parts, we must first initialise a new `Parts::Tagger`.
+
+	tagger = Parts::Tagger.new
+
+This will create a new tagger, and assuming no arguments are passed in, will train it with the default Treebank 3 corpora. With our tagger now created and trained, we can classify a sentence using the tagger's `classify` method. For example, if we wish to classify the string, `Hello world, this is a sentence`, we would write the following.
+
+	tagger.classify ["Hello", "world", ",", "this", "is", "a", "sentence"]
+
+The tagger requires you to split a sentence up into its appropriate tokens, thus when calling the `classify` method, an array of tokens must be passed in rather than the sentence string itself.
+
+As the tagger is trained by default using the Penn Treebank 3 corpora, sentences are tagged with the [Penn Treebank tags](http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). 
+
+## Training parts
+
+Training and evaluating parts with your own corpora is simple. In order to train a tagger with your own corpora, parts requires you to pass in an array of tagged sentences.
+
+	tagger = Parts::Tagger.new sentences
+
+Sentences are stored as array's of word-tag pairs, where each sentence will be [{:word => w1, :tag => t1},...,{:word => wn, :tag => tn}]. For example, were we to train it with one sentence, we might create a `sentences` array as such.
+
+	sentences = [
+	  [
+	    {:word => "Rolls-Royce"", :tag => "NNP"}
+	    {:word => "said", :tag => "VBD"},
+	    {:word => "it", :tag => "PRP"},
+		{:word => "expects", :tag => "VBZ"},
+		{:word => "to", :tag => "TO"},
+		{:word => "remain", :tag => "VB"},
+		{:word => "steady", :tag => "JJ"}
+ 	  ]
+	]
+
+Parts aims to stay out of your way as much as possible, thus you are free to use whatever tags you want within your corpora. It is worth noting that we automatically prepend and append `$start` and `$end` tags to all sentence arrays when training, thus full-stops need *not* be included in each sentence in the `sentences` array.
+
+## TODO
+
+There is still significant work to be done on parts, in particular looking at:
+
+* integrate the k-fold tester such that it can be used with user built corporas
+* noun phrase grouping, e.g. *The British Broadcasting Company*
+* exploring mechanisms for automatically splitting sentences into their tokens
+* introducing tri-gram tagging as an option
+* writing a full featured test suite
+
+## Contributing to parts
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2012 Joe Root. See LICENSE.txt for
+further details.
+

data/lib/parts.rb

@@ -9,7 +9,8 @@ module Parts
 
     attr_accessor :bigrams, :words, :tags, :bigram_smoothing, :suffixes
 
-    def initialize sentences
+    def initialize sentences=nil
+      sentences = Treebank.new.sentences if sentences.nil?
       # Tag-bigrams are stored such that P(T2|T1) = @bigrams[T1][T2].
       # Word-tag pairs are stored such that P(W|T) = @words[W][T].
       # Tags are stored such that @tags[T] = no. of occurences of T.
@@ -149,3 +150,4 @@ module Parts
 end
 
 require 'parts/tester'
+require 'parts/treebank'

data/lib/parts/treebank.rb

@@ -0,0 +1,36 @@
+class Parts::Treebank
+  
+  def initialize path="#{File.dirname(__FILE__)}/treebank3.2.txt"
+    # Sentences are stored as array's of word-tag pairs, where each sentence
+    # will be [{:word => w1, :tag => t1},...,{:word => wn, :tag => tn}].
+    @sentences = []
+    self.load path
+  end
+  
+  def sentences
+    @sentences
+  end
+
+  def load path
+    # For each sentence we split on empty space, and then use regex to split
+    # each word/tag pair into its word and tag constituents. Whenever a full
+    # stop is encountered we create a new sentence.
+    File.open(path, "r") do |file|
+      sentence  = []
+      while (line = file.gets)
+        line.split(' ').each do |part|
+          md = /(.+)+(\/){1}(.+)+/.match part
+          if md
+            if md[3] == "."
+              @sentences << sentence if not sentence.empty?
+              sentence = []
+            else
+              sentence << {:word => md[1].downcase, :tag => md[3]}
+            end
+          end
+        end
+      end
+    end
+  end
+  
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: parts
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
   prerelease: 
 platform: ruby
 authors:
@@ -13,7 +13,7 @@ date: 2012-01-16 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: shoulda
-  requirement: &70189868506800 !ruby/object:Gem::Requirement
+  requirement: &70208468895780 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70189868506800
+  version_requirements: *70208468895780
 - !ruby/object:Gem::Dependency
   name: bundler
-  requirement: &70189868506320 !ruby/object:Gem::Requirement
+  requirement: &70208468895300 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -32,10 +32,10 @@ dependencies:
         version: 1.0.0
   type: :development
   prerelease: false
-  version_requirements: *70189868506320
+  version_requirements: *70208468895300
 - !ruby/object:Gem::Dependency
   name: jeweler
-  requirement: &70189868505820 !ruby/object:Gem::Requirement
+  requirement: &70208468894820 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: 1.6.4
   type: :development
   prerelease: false
-  version_requirements: *70189868505820
+  version_requirements: *70208468894820
 - !ruby/object:Gem::Dependency
   name: rcov
-  requirement: &70189868492780 !ruby/object:Gem::Requirement
+  requirement: &70208468894300 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,20 +54,21 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70189868492780
+  version_requirements: *70208468894300
 description: ''
 email: joe@onlysix.co.uk
 executables: []
 extensions: []
 extra_rdoc_files:
 - LICENSE.txt
-- README.rdoc
+- README.md
 files:
 - lib/parts.rb
 - lib/parts/tester.rb
+- lib/parts/treebank.rb
 - lib/parts/treebank3.2.txt
 - LICENSE.txt
-- README.rdoc
+- README.md
 homepage: http://github.com/joeroot/parts
 licenses:
 - MIT
@@ -83,7 +84,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1093201021619199354
+      hash: -447277508490023829
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:

data/README.rdoc

@@ -1,19 +0,0 @@
-= parts
-
-Description goes here.
-
-== Contributing to parts
- 
-* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
-* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
-* Fork the project
-* Start a feature/bugfix branch
-* Commit and push until you are happy with your contribution
-* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
-* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
-
-== Copyright
-
-Copyright (c) 2012 Joe Root. See LICENSE.txt for
-further details.
-