natural 0.5.4 → 0.6.1

package/README.md

@@ -3,6 +3,7 @@ natural
 
 [![NPM version](https://img.shields.io/npm/v/natural.svg)](https://www.npmjs.com/package/natural)
 [![Build Status](https://travis-ci.org/NaturalNode/natural.png?branch=master)](https://travis-ci.org/NaturalNode/natural)
+[![Slack](https://slack.bri.im/badge.svg)](https://slack.bri.im)
 
 "Natural" is a general natural language facility for nodejs. Tokenizing,
 stemming, classification, phonetics, tf-idf, WordNet, string similarity,
@@ -14,8 +15,7 @@ contributions and the like.
 Note that many algorithms from Rob Ellis's [node-nltools](https://github.com/NaturalNode/node-nltools) are
 being merged into this project and will be maintained from here onward.
 
-At the moment, most of the algorithms are English-specific, but in the long-term, some diversity
-will be in order. Thanks to Polyakov Vladimir, Russian stemming has been added!, Thanks to David Przybilla, Spanish stemming has been added!.
+While most of the algorithms are English-specific, contributors have implemented support for other languages. Thanks to Polyakov Vladimir, Russian stemming has been added! Thanks to David Przybilla, Spanish stemming has been added! Thanks to [even more contributors](https://github.com/NaturalNode/natural/graphs/contributors), stemming and tokenizing in more languages have been added.
 
 Aside from this README, the only documentation is [this DZone article](http://www.dzone.com/links/r/using_natural_a_nlp_module_for_nodejs.html), [this free course on Egghead.io](https://egghead.io/courses/natural-language-processing-in-javascript-with-natural), and [here on my blog](http://www.chrisumbel.com/article/node_js_natural_language_porter_stemmer_lancaster_bayes_naive_metaphone_soundex), which is a bit older.
 
@@ -24,8 +24,12 @@ Aside from this README, the only documentation is [this DZone article](http://ww
 * [Installation](#installation)
 * [Tokenizers](#tokenizers)
 * [String Distance](#string-distance)
+* [Approximate String Matching](#approximate-string-matching)
 * [Stemmers](#stemmers)
 * [Classifiers](#classifiers)
+  * [Bayesian and logistic regression](#bayesian-and-logistic-regression)
+  * [Maximum Entropy Classifier](#maximum-entropy-classifier)
+* [Sentiment Analysis](#sentiment-analysis)
 * [Phonetics](#phonetics)
 * [Inflectors](#inflectors)
 * [N-Grams](#n-grams)
@@ -37,7 +41,6 @@ Aside from this README, the only documentation is [this DZone article](http://ww
 * [WordNet](#wordnet)
 * [Spellcheck](#spellcheck)
 * [POS Tagger](#pos-tagger)
-* [Acknowledgements/references](#acknowledgements-and-references)
 * [Development](#development)
 * [License](#license)
 
@@ -78,16 +81,64 @@ console.log(tokenizer.tokenize("flea-dog"));
 tokenizer = new natural.WordPunctTokenizer();
 console.log(tokenizer.tokenize("my dog hasn't any fleas."));
 // [ 'my',  'dog',  'hasn',  '\'',  't',  'any',  'fleas',  '.' ]
-```
+
+tokenizer = new natural.OrthographyTokenizer({language: "fi"});
+console.log(tokenizer.tokenize("Mikä sinun nimesi on?"));
+// [ 'Mikä', 'sinun', 'nimesi', 'on' ]
+```
+
+Overview of available tokenizers:
+
+| Tokenizer              | Language    | Explanation                                                             |
+|:-----------------------|:------------|:------------------------------------------------------------------------|
+| WordTokenizer          | Any         | Splits on anything except alphabetic characters, digits and underscore  |
+| WordPunctTokenizer     | Any         | Splits on anything except alphabetic characters, digits, punctuation and underscore  |
+| SentenceTokenizer      | Any         | Break string up in to parts based on punctation and quotation marks     |
+| CaseTokenizer          | Any?        | If lower and upper case are the same, the character is assumed to be whitespace or something else (punctuation) |
+| RegexpTokenizer        | Any         | Splits on a regular expression that either defines sequences of word characters or gap characters |
+| OrthographyTokenizer   | Finnish     | Splits on anything except alpabetic characters, digits and underscore   |
+| TreebankWordTokenizer  | Any         |  |
+| AggressiveTokenizer    | English     |  |
+| AggressiveTokenizerFa  | Farsi       |  |
+| AggressiveTokenizerFr  | French      |  |
+| AggressiveTokenizerRu  | Russian     |  |
+| AggressiveTokenizerEs  | Spanish     |  |
+| AggressiveTokenizerIt  | Italian     |  |
+| AggressiveTokenizerPl  | Polish      |  |
+| AggressiveTokenizerPt  | Portugese   |  |
+| AggressiveTokenizerNo  | Norwegian   |  |
+| AggressiveTokenizerSv  | Swedish     |  |
+| AggressiveTokenizerVi  | Vietnamese  |  |
+| TokenizerJa            | Japanese    |  |  |
+
+
 
 ## String Distance
 
-Natural provides an implementation of the [Jaro–Winkler](http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance) string distance measuring algorithm.
-This will return a number between 0 and 1 which tells how closely the strings match (0 = not at all, 1 = exact match):
+Natural provides an implementation of three algorithms for calculating string distance: Hamming distance, Jaro-Winkler, Levenshtein distance, and Dice coefficient.
 
+[Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) measures the distance between two strings of equal length by counting the number of different characters. The third parameter indicates whether case should be ignored. By default the algorithm is case sensitive.
 ```javascript
 var natural = require('natural');
-console.log(natural.JaroWinklerDistance("dixon","dicksonx"))
+console.log(natural.HammingDistance("karolin", "kathrin", false));
+console.log(natural.HammingDistance("karolin", "kerstin", false));
+// If strings differ in length -1 is returned
+console.log(natural.HammingDistance("short string", "longer string", false));
+```
+
+Output:
+```javascript
+3
+3
+-1
+```
+
+
+The [Jaro–Winkler](http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance) string distance measuring algorithm will return a number between 0 and 1 which tells how closely the strings match (0 = not at all, 1 = exact match):
+
+```javascript
+var natural = require('natural');
+console.log(natural.JaroWinklerDistance("dixon","dicksonx"));
 console.log(natural.JaroWinklerDistance('not', 'same'));
 ```
 
@@ -98,7 +149,14 @@ Output:
 0
 ```
 
-Natural also offers support for Levenshtein distances:
+If the distance between the strings is already known you can pass it as a third parameter. And you can force the algorithm to ignore case by passing a fourth parameter as follows:
+```javascript
+var natural = require('natural');
+console.log(natural.JaroWinklerDistance("dixon","dicksonx", undefined, true));
+```
+
+
+Natural also offers support for [Levenshtein](https://en.wikipedia.org/wiki/Levenshtein_distance) distances:
 
 ```javascript
 var natural = require('natural');
@@ -129,7 +187,46 @@ Output:
 1
 ```
 
-And Dice's co-efficient:
+Full Damerau-Levenshtein matching can be used if you want to consider character transpositions as a valid edit operation.
+
+```javascript
+console.log(natural.DamerauLevenshteinDistance("az", "za"));
+```
+
+Output:
+```javascript
+1
+```
+
+The transposition cost can be modified as well:
+
+```javascript
+console.log(natural.DamerauLevenshteinDistance("az", "za", { transposition_cost: 0 }))
+```
+
+Output:
+```javascript
+0
+```
+
+A restricted form of Damerau-Levenshtein (Optimal String Alignment) is available.
+
+This form of matching is more space efficient than unrestricted Damerau-Levenshtein, by only considering a transposition if there are no characters between the transposed characters.
+
+Comparison:
+
+```javascript
+// Optimal String Alignment
+console.log(natural.DamerauLevenshteinDistance('ABC', 'ACB'), { restricted: true });
+1
+console.log(natural.DamerauLevenshteinDistance('CA', 'ABC', { restricted: true }));
+2
+// Unrestricted Damerau-Levenshtein
+console.log(natural.DamerauLevenshteinDistance('CA', 'ABC', { restricted: false }));
+1
+```
+
+And [Dice's co-efficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient):
 
 ```javascript
 var natural = require('natural');
@@ -144,9 +241,28 @@ Output:
 0
 ```
 
+## Approximate String Matching
+Currently matching is supported via the Levenshtein algorithm.
+
+```javascript
+var natural = require('natural');
+var source = 'The RainCoat BookStore';
+var target = 'All the best books are here at the Rain Coats Book Store';
+
+console.log(natural.LevenshteinDistance(source, target, {search: true}));
+```
+
+Output:
+
+```javascript
+{ substring: 'the Rain Coats Book Store', distance: 4 }
+```
+
+The following
+
 ## Stemmers
 
-Currently stemming is supported via the [Porter](http://tartarus.org/martin/PorterStemmer/index.html) and [Lancaster](http://www.comp.lancs.ac.uk/computing/research/stemming/) (Paice/Husk) algorithms.
+Currently stemming is supported via the [Porter](http://tartarus.org/martin/PorterStemmer/index.html) and [Lancaster](http://www.comp.lancs.ac.uk/computing/research/stemming/) (Paice/Husk) algorithms. The Indonesian and Japanese stemmers do not follow a known algorithm.
 
 ```javascript
 var natural = require('natural');
@@ -170,6 +286,24 @@ console.log(natural.PorterStemmerRu.stem("падший"));
 console.log(natural.PorterStemmerEs.stem("jugaría"));
 ```
 
+The following stemmers are available:
+
+| Language      | Porter      | Lancaster | Other     | Module            |
+|:------------- |:-----------:|:---------:|:---------:|:------------------|
+| Dutch         | X           |           |           | `PorterStemmerNl` |
+| English       | X           |           |           | `PorterStemmer`   |
+| English       |             |  X        |           | `LancasterStemmer` |
+| Farsi (in progress) |  X    |           |           | `PorterStemmerFa` |
+| French        | X           |           |           | `PorterStemmerFr` |
+| Indonesian    |             |           | X         | `StemmerId`       |
+| Italian       | X           |           |           | `PorterStemmerIt` |
+| Japanese      |             |           | X         | `StemmerJa`       |
+| Norwegian     | X           |           |           | `PorterStemmerNo` |
+| Portugese     | X           |           |           | `PorterStemmerPt` |
+| Russian       | X           |           |           | `PorterStemmerRu` |
+| Swedish       | X           |           |           | `PorterStemmerSv` |
+
+
 `attach()` patches `stem()` and `tokenizeAndStem()` to String as a shortcut to
 `PorterStemmer.stem(token)`. `tokenizeAndStem()` breaks text up into single words
 and returns an array of stemmed tokens.
@@ -180,7 +314,7 @@ console.log("i am waking up to the sounds of chainsaws".tokenizeAndStem());
 console.log("chainsaws".stem());
 ```
 
-the same thing can be done with a Lancaster stemmer:
+The same thing can be done with a Lancaster stemmer:
 
 ```javascript
 natural.LancasterStemmer.attach();
@@ -190,6 +324,8 @@ console.log("chainsaws".stem());
 
 ## Classifiers
 
+### Bayesian and logistic regression
+
 Two classifiers are currently supported, [Naive Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) and [logistic regression](http://en.wikipedia.org/wiki/Logistic_regression).
 The following examples use the BayesClassifier class, but the
 LogisticRegressionClassifier class could be substituted instead.
@@ -247,7 +383,7 @@ classifier.addDocument(['sell', 'gold'], 'sell');
 ```
 
 The training process can be monitored by subscribing to the event `trainedWithDocument` that's emitted by the classifier, this event's emitted each time a document is finished being trained against:
-
+```javascript
     classifier.events.on('trainedWithDocument', function (obj) {
        console.log(obj);
        /* {
@@ -257,7 +393,7 @@ The training process can be monitored by subscribing to the event `trainedWithDo
        *  }
        */
     });
-
+```
 A classifier can also be persisted and recalled so you can reuse a training
 
 ```javascript
@@ -298,8 +434,215 @@ const PorterStemmerRu = require('./node_modules/natural/lib/natural/stemmers/por
 var classifier = new natural.BayesClassifier(PorterStemmerRu);
 ```
 
-## Phonetics
+### Maximum Entropy Classifier
+This module provides a classifier based on maximum entropy modelling. The central idea to maximum entropy modelling is to estimate a probability distribution that that has maximum entropy subject to the evidence that is available. This means that the distribution follows the data it has "seen" but does not make any assumptions beyond that.
+
+The module is not specific to natural language processing, or any other application domain. There are little requirements with regard to the data structure it can be trained on. For training, it needs a sample that consists of elements. These elements have two parts:
+* part a: the class of the element
+* part b: the context of the element
+The classifier will, once trained, return the most probable class for a particular context.
+
+We start with an explanation of samples and elements. You have to create your own specialisation of the Element class. Your element class should implement the generateFeatures method for inferring feature functions from the sample.
+
+#### Samples and elements
+Elements and contexts are created as follows:
+
+```javascript
+var MyElement = require('MyElementClass');
+var Context = require('Context');
+var Sample = require('Sample');
+
+var x = new MyElementClass("x", new Context("0"));
+// A sample is created from an array of elements
+var sample = new Sample();
+sample.addElement(x);
+```
+A class is a string, contexts may be as complex as you want (as long as it can be serialised).
+
+A sample can be saved to and loaded from a file:
+```javascript
+sample.save('sample.json', function(error, sample) {
+  ...
+});
+```
+A sample can be read from a file as follows.
+
+```javascript
+sample.load('sample.json', MyElementClass, function(err, sample) {
+
+});
+```
+You have to pass the element class to the load method so that the right element objects can be created from the data.
+
+#### Features and feature sets
+Features are functions that map elements to zero or one. Features are defined as follows:
+```javascript
+var Feature = require('Feature');
+
+function f(x) {
+  if (x.b === "0") {
+    return 1;
+  }
+  return 0;
+}
+
+var feature = new Feature(f, name, parameters);
+```
+<code>name</code> is a string for the name of the feature function, <code>parameters</code> is an array of strings for the parameters of the feature function. The combination of name and parameters should uniquely distinguish features from each other. Features that are added to a feature set are tested for uniqueness using these properties.
 
+A feature set is created like this
+```javascript
+var FeatureSet = require('FeatureSet');
+
+var set = new FeatureSet();
+set.addFeature(f, "f", ["0"]);
+```
+
+In most cases you will generate feature functions using closures. For instance, when you generate feature functions in a loop that iterates through an array
+```javascript
+var FeatureSet = require('FeatureSet');
+var Feature = require('Feature');
+
+var listOfTags = ['NN', 'DET', 'PREP', 'ADJ'];
+var featureSet = new FeatureSet();
+
+listofTags.forEach(function(tag) {
+  function isTag(x) {
+    if (x.b.data.tag === tag) {
+      return 1
+    }
+    return 0;
+  }
+  featureSet.addFeature(new Feature(f, "f", [tag]));
+});
+```
+In this example you create feature functions that each have a different value for <code>tag</code> in their closure.
+
+#### Setting up and training the classifier
+A classifier needs the following parameter:
+* Classes: an array of classes (strings)
+* Features: an array of feature functions
+* Sample: a sample of elements for training the classifier
+
+A classifier can be created as follows:
+```javascript
+var Classifier = require('Classifier');
+var classifier = new Classifier(classes, featureSet, sample);
+```
+And it starts training with:
+```javascript
+var maxIterations = 100;
+var minImprovement = .01;
+var p = classifier.train(maxIterations, minImprovement);
+```
+Training is finished when either <code>maxIterations</code> is reached or the improvement in likelihood (of the sample) becomes smaller than <code>minImprovement</code>. It returns a probability distribution that can be stored and retrieved for later usage:
+```javascript
+classifier.save('classifier.json', function(err, c) {
+  if (err) {
+    console.log(err);
+  }
+  else {
+    // Continue using the classifier
+  }
+});
+
+classifier.load('classifier.json', function(err, c) {
+  if (err) {
+    console.log(err);
+  }
+  else {
+    // Use the classifier
+  }
+});
+```
+
+The training algorithm is based on Generalised Iterative Scaling.
+
+#### Applying the classifier
+The classifier can be used to classify contexts in two ways. To get the probabilities for all classes:
+```javascript
+var classifications = classifier.getClassifications(context);
+classifications.forEach(function(classPlusProbability) {
+  console.log('Class ' + classPlusProbability.label + ' has score ' + classPlusProbability.value);
+});
+```
+This returns a map from classes to probabilities.
+To get the highest scoring class:
+```javascript
+var class = classifier.classify(context);
+console.log(class);
+```
+
+#### Simple example of maximum entropy modelling
+A  test is added to the spec folder based on simple elements that have contexts that are either "0" or "1", and classes are "x" and "y".
+```javascript
+{
+  "a": "x",
+  "b": {
+    "data": "0"
+  }
+}
+```
+In the SE_Element class that inherits from Element, the method generateFeatures is implemented. It creates a feature function that tests for context "0".
+
+After setting up your own element class, the classifier can be created and trained.
+
+#### Application to POS tagging
+A more elaborate example of maximum entropy modelling is provided for part of speech tagging. The following steps are taken to create a classifier and apply it to a test set:
+* A new element class POS_Element is created that has a word window and a tag window around the word to be tagged.
+* From the Brown corpus a sample is generated consisting of POS elements.
+* Feature functions are generated from the sample.
+* A classifier is created and trained.
+* The classifier is applied to a test set. Results are compared to a simple lexicon-based tagger.  
+
+#### References
+* Adwait RatnaParkhi, Maximum Entropy Models For Natural Language Ambiguity Resolution, University of Pennsylvania, 1998, URL: http://repository.upenn.edu/cgi/viewcontent.cgi?article=1061&context=ircs_reports
+* Darroch, J.N.; Ratcliff, D. (1972). Generalized iterative scaling for log-linear models, The Annals of Mathematical Statistics, Institute of Mathematical Statistics, 43 (5): 1470–1480.
+
+## Sentiment Analysis
+This is a simple sentiment analysis algorithm based on a vocabulary that assigns polarity to words. The algorithm calculates the sentiment of a piece of text by summing the polarity of each word and normalizing with the length of the sentence. If a negation occurs the result is made negative. It is used as follows:
+```javascript
+var Analyzer = require('natural').SentimentAnalyzer;
+var stemmer = require('natural').PorterStemmer;
+var analyzer = new Analyzer("English", stemmer, "afinn")
+console.log(analyzer.getSentiment("I like cherries"));
+// 0.6666666666666666
+```
+The constructor has three parameters:
+* Language: see below for supported languages.
+* Stemmer: to increase the coverage of the sentiment analyzer a stemmer may be provided. May be `null`.
+* Vocabulary: sets the type of vocabulary, `"afinn"`, `"senticon"` or `"pattern"` are valid values.
+
+Currently, the following languages are supported with type of vocabulary and availability of negations (in alphabetic order):
+
+| Language      | AFINN       | Senticon  | Pattern   | Negations |
+| ------------- |:-----------:|:---------:|:---------:|:---------:|
+| Basque        |             |  X        |           |           |
+| Catalan       |             |  X        |           |           |
+| Dutch         |             |           | X         | X         |
+| English       | X           |  X        | X         | X         |
+| French        |             |           | X         |           |
+| Galician      |             |  X        |           |           |   
+| Italian       |             |           | X         |           |
+| Spanish       | X           |  X        |           | X         |     
+
+More languages can be added by adding vocabularies and extending the map `languageFiles` in `SentimentAnalyzer.js`. In the tools folder below `lib/natural/sentiment` some tools are provided for transforming vocabularies in Senticon and Pattern format into a JSON format.
+
+
+
+### Acknowledgements and References
+Thanks to Domingo Martín Mancera for providing the basis for this sentiment analyzer in his repo [Lorca](https://github.com/dmarman/lorca).
+
+AFINN is a list of English words rated for valence with an integer
+between minus five (negative) and plus five (positive). The words have
+been manually labeled by Finn Årup Nielsen in 2009-2011. Scientific reference can be found [here](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010). We used [afinn-165](https://github.com/words/afinn-165) which is available as nodejs module.
+
+The senticon vocabulary is based on work by Fermin L. Cruz and others:
+Cruz, Fermín L., José A. Troyano, Beatriz Pontes, F. Javier Ortega. Building layered, multilingual sentiment lexicons at synset and lemma levels, Expert Systems with Applications, 2014.
+
+The Pattern vocabularies are from the [Pattern project](https://github.com/clips/pattern) of the CLiPS Research Center of University of Antwerpen. These have a PDDL license.
+
+## Phonetics
 Phonetic matching (sounds-like) matching can be done with the [SoundEx](http://en.wikipedia.org/wiki/Soundex),
 [Metaphone](http://en.wikipedia.org/wiki/Metaphone) or [DoubleMetaphone](http://en.wikipedia.org/wiki/Metaphone#Double_Metaphone) algorithms
 
@@ -565,7 +908,11 @@ console.log(NGramsZH.ngrams(['一', '个', '中', '文', '测',
 ## tf-idf
 
 [Term Frequency–Inverse Document Frequency (tf-idf)](http://en.wikipedia.org/wiki/Tf%E2%80%93idf) is implemented to determine how important a word (or words) is to a
-document relative to a corpus. The following example will add four documents to
+document relative to a corpus. The following formulas are used for calculating tf and idf:
+* tf(t, d) is a so-called raw count, so just the count of the term in the document
+* idf(t, D) uses the following formula: 1 + ln(N / (1 + n_t)) where N is the number of documents, and n_t the number of documents in which the term appears. The 1 + in the denominator is for handling the possibility that n_t is 0. 
+
+The following example will add four documents to
 a corpus and determine the weight of the word "node" and then the weight of the
 word "ruby" in each document.
 
@@ -648,7 +995,7 @@ document #1 is 1
 document #2 is 2
 ```
 
-The examples above all use strings, which case natural to automatically tokenize the input.
+The examples above all use strings, which causes natural to automatically tokenize the input.
 If you wish to perform your own tokenization or other kinds of processing, you
 can do so, then pass in the resultant arrays later. This approach allows you to bypass natural's
 default preprocessing.
@@ -790,14 +1137,14 @@ get the number of vertexes:
 ```javascript
 console.log(digraph.v());
 ```
-you will get 5.
+you will get 7.
 
 get the number of edges:
 
 ```javascript
 console.log(digraph.e());
 ```
-you will get 5.
+you will get 6.
 
 
 
@@ -825,22 +1172,9 @@ console.log(spt.getDistTo(4));
 ```
 the output will be: 0.35
 
-### hasDistTo(vertex)
-
-```javascript
-console.log(spt.hasDistTo(4));
-console.log(spt.hasDistTo(5));
-```
-
-output will be:
-
-```javascript
-true
-false
-```
-
 ### pathTo(vertex)
-this will return a shortest path:
+
+Will return the shortest path:
 
 ```javascript
 console.log(spt.pathTo(4));
@@ -862,8 +1196,8 @@ hasPathTo(vertex),
 pathTo(vertex).
 
 ```javascript
-var ShortestPathTree = natural.ShortestPathTree;
-var spt = new ShortestPathTree(digraph, 5);
+var LongestPathTree = natural.LongestPathTree;
+var lpt = new LongestPathTree(digraph, 5);
 ```
 digraph is an instance of EdgeWeightedDigraph, the second param is the start vertex of DAG.
 
@@ -872,29 +1206,16 @@ digraph is an instance of EdgeWeightedDigraph, the second param is the start ver
 Will return the dist to vertex.
 
 ```javascript
-console.log(spt.getDistTo(4));
+console.log(lpt.getDistTo(4));
 ```
 the output will be: 2.06
 
-### hasDistTo(vertex)
-
-```javascript
-console.log(spt.hasDistTo(4));
-console.log(spt.hasDistTo(5));
-```
-
-output will be:
-
-```javascript
-true
-false
-```
-
 ### pathTo(vertex)
-this will return a longest path:
+
+Will return the longest path:
 
 ```javascript
-console.log(spt.pathTo(4));
+console.log(lpt.pathTo(4));
 ```
 
 output will be:
@@ -1000,9 +1321,19 @@ var rules = new natural.RuleSet(rulesFilename);
 var tagger = new natural.BrillPOSTagger(lexicon, rules);
 
 var sentence = ["I", "see", "the", "man", "with", "the", "telescope"];
-console.log(JSON.stringify(tagger.tag(sentence)));
-// [["I","NN"],["see","VB"],["the","DT"],["man","NN"],["with","IN"],["the","DT"],["telescope","NN"]]
-
+console.log(tagger.tag(sentence));
+```
+This outputs the following:
+```
+Sentence {
+  taggedWords:
+   [ { token: 'I', tag: 'NN' },
+     { token: 'see', tag: 'VB' },
+     { token: 'the', tag: 'DT' },
+     { token: 'man', tag: 'NN' },
+     { token: 'with', tag: 'IN' },
+     { token: 'the', tag: 'DT' },
+     { token: 'telescope', tag: 'NN' } ] }
 ```
 
 ### Lexicon
@@ -1043,34 +1374,30 @@ VBD NN PREV-TAG DT
 Here the category of the previous word must be <code>DT</code> for the rule to be applied.
 
 ### Algorithm
-The tagger applies transformation rules that may change the category of words. The input sentence must be split into words which are assigned with categories. The tagged sentence is then processed from left to right. At each step all rules are applied once; rules are applied in the order in which they are specified. Algorithm:
+The tagger applies transformation rules that may change the category of words. The input sentence is a Sentence object with tagged words. The tagged sentence is processed from left to right. At each step all rules are applied once; rules are applied in the order in which they are specified. Algorithm:
 ```javascript
-function(sentence) {
-  var tagged_sentence = new Array(sentence.length);
-
-  // snip
-
-  // Apply transformation rules
-  for (var i = 0, size = sentence.length; i < size; i++) {
-    this.transformation_rules.forEach(function(rule) {
-      rule.apply(tagged_sentence, i);
+Brill_POS_Tagger.prototype.applyRules = function(sentence) {
+  for (var i = 0, size = sentence.taggedWords.length; i < size; i++) {
+    this.ruleSet.getRules().forEach(function(rule) {
+      rule.apply(sentence, i);
     });
   }
-  return(tagged_sentence);
-}
+  return sentence;
+};
 ```
+The output is a Sentence object just like the input sentence.
 
 ### Adding a predicate
 Predicates are defined in module <code>lib/RuleTemplates.js</code>. In that file
 predicate names are mapped to metadata for generaring transformation rules. The following properties must be supplied:
 * Name of the predicate
 * A function that evaluates the predicate (should return a boolean)
-* A window <code>[i, j]</code> that defines the span of the predicate in the 
+* A window <code>[i, j]</code> that defines the span of the predicate in the
 sentence relative to the current position
 * The number of parameter the predicate needs: 0, 1 or 2
-* If relevant, a function for parameter 1 that returns its possible values 
+* If relevant, a function for parameter 1 that returns its possible values
 at the current position in the sentence (for generating rules in training)
-* If relevant, a function for parameter 2 that returns its possible values 
+* If relevant, a function for parameter 2 that returns its possible values
 at the current position in the sentence (for training)
 
 A typical entry for a rule templates looks like this:
@@ -1086,13 +1413,13 @@ A typical entry for a rule templates looks like this:
     "parameter1Values": nextTagParameterValues
   }
 ```
-A predicate function accepts a tagged sentence, the current position in the 
-sentence that should be tagged, and the outcome(s) of the predicate. 
+A predicate function accepts a Sentence object, the current position in the
+sentence that should be tagged, and the outcome(s) of the predicate.
 An example of a predicate that checks the category of the current word:
 ```javascript
-function next_tag_is(tagged_sentence, i, parameter) {
-  if (i < tagged_sentence.length - 1) {
-    return(tagged_sentence[i+1][1] === parameter);
+function next_tag_is(sentence, i, parameter) {
+  if (i < sentence.taggedWords.length - 1) {
+    return(sentence.taggedWords[i + 1][1] === parameter);
   }
   else {
     return(false);
@@ -1100,7 +1427,7 @@ function next_tag_is(tagged_sentence, i, parameter) {
 }
 ```
 
-A values function for a parameter returns an array all possible parameter 
+A values function for a parameter returns an array all possible parameter
 values given a location in a tagged sentence.
 ```javascript
 function nextTagParameterValues(sentence, i) {
@@ -1112,15 +1439,11 @@ function nextTagParameterValues(sentence, i) {
   }
 }
 ```
-Please note that these functions work with a different data type. Here, a 
-sentence is an array of tokens and tokens are maps that have at least a 
-token (word) and a tag. 
-
 
 ### Training
-The trainer allows to learn a new set of transformation rules from a corpus. 
-It takes as input a tagged corpus and a set of rule templates. The algorithm 
-generates positive rules (rules that apply at some location in the corpus) 
+The trainer allows to learn a new set of transformation rules from a corpus.
+It takes as input a tagged corpus and a set of rule templates. The algorithm
+generates positive rules (rules that apply at some location in the corpus)
 from the templates and iteratively extends and optimises the rule set.
 
 First, a corpus should be loaded. Currently, the format of Brown corpus is supported. Then a lexicon can be created from the corpus. The lexicon is needed for tagging the sentences before the learning algorithm is applied.
@@ -1130,8 +1453,8 @@ var text = fs.readFileSync(brownCorpusFile, 'utf8');
 var corpus = new natural.Corpus(text, 1);
 var lexicon = corpus.buildLexicon();
 ```
-The next step is to create a set of rule templates from which the learning 
-algorithm can generate transformation rules. Rule templates are defined in 
+The next step is to create a set of rule templates from which the learning
+algorithm can generate transformation rules. Rule templates are defined in
 <code>PredicateMapping.js</code>.
 ```javascript
 var natural require('natural');
@@ -1152,17 +1475,17 @@ var Tester = require('natural.BrillPOSTrainer');
 var trainer = new Trainer(/* optional threshold */);
 var ruleSet = trainer.train(corpus, templates, lexicon);
 ```
-A threshold value can be passed to constructor. Transformation rules with 
+A threshold value can be passed to constructor. Transformation rules with
 a score below the threshold are removed after training.
-The train method returns a set of transformation rules that can be used to 
-create a POS tagger as usual. Also you can output the rule set in the right 
+The train method returns a set of transformation rules that can be used to
+create a POS tagger as usual. Also you can output the rule set in the right
 format for later usage.
 ```javascript
 console.log(ruleSet.prettyPrint());
 ```
 
 ### Testing
-Now we can apply the lexicon and rule set to a test set. 
+Now we can apply the lexicon and rule set to a test set.
 ```javascript
 var tester = new natural.BrillPOSTester();
 var tagger = new natural.BrillPOSTagger(lexicon, ruleSet);