opener-tokenizer 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: fcf7f5ea8023ba15ef71ca6fb82839f7404cf1bf
+  data.tar.gz: 021f76b37d483bea54bc3f41ccb1c85a35432003
+SHA512:
+  metadata.gz: d359e3fdfc7f5792958df137725fcdb02192f52e8a321654628bb5ed4a954b09ef461f5f6a69ed2215e813cad663b945c61cf8a262a80775d882dba98cc185a3
+  data.tar.gz: 77d3ab3fd86a8dd12d8fe4210f36d40e1fb1185cfaab0e5fe40eba34b44b84e62ffe80e042e14b6c90aa67f2b58feaf4fcffd6e8aac85339d6dae9ba7564d22a

data/README.md

@@ -0,0 +1,165 @@
+Introduction
+------------
+
+The tokenizer tokenizes a text into sentences and words. 
+
+### Confused by some terminology?
+
+This software is part of a larger collection of natural language processing
+tools known as "the OpeNER project". You can find more information about the
+project at (the OpeNER portal)[http://opener-project.github.io]. There you can
+also find references to terms like KAF (an XML standard to represent linguistic
+annotations in texts), component, cores, scenario's and pipelines.
+
+Quick Use Example
+-----------------
+
+Installing the tokenizer can be done by executing:
+
+    gem install tokenizer
+
+Please bare in mind that all components in OpeNER take KAF as an input and
+output KAF by default.
+
+
+### Command line interface
+
+You should now be able to call the tokenizer as a regular shell
+command: by its name. Once installed the gem normalyl sits in your path so you can call it directly from anywhere.
+
+Tokenizing some text:
+
+    echo "This is English text" | tokenizer -l en --no-kaf
+    
+Will result in
+
+    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
+    <KAF version="v1.opener" xml:lang="en">
+      <kafHeader>
+        <linguisticProcessors layer="text">
+          <lp name="opener-sentence-splitter-en" timestamp="2013-05-31T11:39:31Z" version="0.0.1"/>
+          <lp name="opener-tokenizer-en" timestamp="2013-05-31T11:39:32Z" version="1.0.1"/>
+        </linguisticProcessors>
+      </kafHeader>
+      <text>
+        <wf length="4" offset="0" para="1" sent="1" wid="w1">This</wf>
+        <wf length="2" offset="5" para="1" sent="1" wid="w2">is</wf>
+        <wf length="7" offset="8" para="1" sent="1" wid="w3">English</wf>
+        <wf length="4" offset="16" para="1" sent="1" wid="w4">text</wf>
+      </text>
+    </KAF>
+
+  The available languages for tokenization are: English (en), German (de), Dutch (nl), French (fr), Spanish (es), Italian (it)
+
+#### KAF input format
+
+The tokenizer is capable of taking KAF as input, and actually does so by
+default. You can do so like this:
+
+    echo "<?xml version='1.0' encoding='UTF-8' standalone='no'?><KAF version='v1.opener' xml:lang='en'><raw>This is what I call, a test!</raw></KAF>" | tokenizer
+
+Will result in
+
+    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
+    <KAF version="v1.opener" xml:lang="en">
+      <kafHeader>
+        <linguisticProcessors layer="text">
+          <lp name="opener-sentence-splitter-en" timestamp="2013-05-31T11:39:31Z" version="0.0.1"/>
+          <lp name="opener-tokenizer-en" timestamp="2013-05-31T11:39:32Z" version="1.0.1"/>
+        </linguisticProcessors>
+      </kafHeader>
+      <text>
+        <wf length="4" offset="0" para="1" sent="1" wid="w1">this</wf>
+        <wf length="2" offset="5" para="1" sent="1" wid="w2">is</wf>
+        <wf length="2" offset="8" para="1" sent="1" wid="w3">an</wf>
+        <wf length="7" offset="11" para="1" sent="1" wid="w4">english</wf>
+        <wf length="4" offset="19" para="1" sent="1" wid="w5">text</wf>
+      </text>
+    </KAF>
+
+If the argument -k (--kaf) is passed, then the argument -l (--language) is ignored.
+
+### Webservices
+
+You can launch a language identification webservice by executing:
+
+    tokenizer-server
+
+This will launch a mini webserver with the webservice. It defaults to port 9292,
+so you can access it at <http://localhost:9292>.
+
+To launch it on a different port provide the `-p [port-number]` option like
+this:
+
+    tokenizer-server -p 1234
+
+It then launches at <http://localhost:1234>
+
+Documentation on the Webservice is provided by surfing to the urls provided
+above. For more information on how to launch a webservice run the command with
+the ```-h``` option.
+
+
+### Daemon
+
+Last but not least the tokenizer comes shipped with a daemon that
+can read jobs (and write) jobs to and from Amazon SQS queues. For more
+information type:
+
+    tokenizer-daemon -h
+
+Description of dependencies
+---------------------------
+
+This component runs best if you run it in an environment suited for OpeNER
+components. You can find an installation guide and helper tools in the (OpeNER
+installer)[https://github.com/opener-project/opener-installer] and (an
+installation guide on the Opener
+Website)[http://opener-project.github.io/getting-started/how-to/local-installation.html]
+
+At least you need the following system setup:
+
+### Depenencies for normal use:
+
+* Perl 5
+* MRI 1.9.3
+
+### Dependencies if you want to modify the component:
+
+* Maven (for building the Gem)
+
+
+Language Extension
+------------------
+
+  TODO
+
+The Core
+--------
+
+The component is a fat wrapper around the actual language technology core. You
+can find the core technolies in the following repositories:
+
+* (tokenizer-base)[http://github.com/opener-project/tokenizer-base]
+
+Where to go from here
+---------------------
+
+* Check (the project websitere)[http://opener-project.github.io]
+* (Checkout the webservice)[http://opener.olery.com/tokenizer]
+
+Report problem/Get help
+-----------------------
+
+If you encounter problems, please email support@opener-project.eu or leave an
+issue in the (issue tracker)[https://github.com/opener-project/tokenizer/issues].
+
+
+Contributing
+------------
+
+1. Fork it ( http://github.com/opener-project/tokenizer/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/bin/tokenizer

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require_relative '../lib/opener/tokenizer'
+
+cli = Opener::Tokenizer::CLI.new(:args => ARGV)
+
+cli.run(STDIN.tty? ? nil : STDIN.read)

data/bin/tokenizer-daemon

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+#
+require 'rubygems'
+require 'opener/daemons'
+
+exec_path = File.expand_path("../../exec/tokenizer.rb", __FILE__)
+Opener::Daemons::Controller.new(:name=>"tokenizer",
+                                :exec_path=>exec_path)
+

data/bin/tokenizer-server

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+
+require 'rack'
+
+# Without calling `Rack::Server#options` manually the CLI arguments will never
+# be passed, thus the application can't be specified as a constructor argument.
+server                  = Rack::Server.new
+server.options[:config] = File.expand_path('../../config.ru', __FILE__)
+
+server.start

data/config.ru

@@ -0,0 +1,4 @@
+require File.expand_path('../lib/opener/tokenizer', __FILE__)
+require File.expand_path('../lib/opener/tokenizer/server', __FILE__)
+
+run Opener::Tokenizer::Server

data/exec/tokenizer.rb

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+#
+require 'opener/daemons'
+require 'opener/tokenizer'
+
+options = Opener::Daemons::OptParser.parse!(ARGV)
+daemon = Opener::Daemons::Daemon.new(Opener::Tokenizer, options)
+daemon.start

data/lib/opener/tokenizer/cli.rb

@@ -0,0 +1,117 @@
+module Opener
+  class Tokenizer
+    ##
+    # CLI wrapper around {Opener::Tokenizer} using OptionParser.
+    #
+    # @!attribute [r] options
+    #  @return [Hash]
+    # @!attribute [r] option_parser
+    #  @return [OptionParser]
+    #
+    class CLI
+      attr_reader :options, :option_parser
+
+      ##
+      # @param [Hash] options
+      #
+      def initialize(options = {})
+        @options = DEFAULT_OPTIONS.merge(options)
+
+        @option_parser = OptionParser.new do |opts|
+          opts.program_name   = 'tokenizer'
+          opts.summary_indent = '  '
+
+          opts.on('-h', '--help', 'Shows this help message') do
+            show_help
+          end
+
+          opts.on('-v', '--version', 'Shows the current version') do
+            show_version
+          end
+
+          opts.on(
+            '-l',
+            '--language [VALUE]',
+            'Uses this specific language'
+          ) do |value|
+            @options[:language] = value
+            @options[:kaf] = false
+          end
+
+          opts.on('-k', '--kaf', 'Treats the input as a KAF document') do
+            @options[:kaf] = true
+          end
+
+          opts.on('-p', '--plain', 'Treats the input as plain text') do
+            @options[:kaf] = false
+          end
+
+          opts.separator <<-EOF
+
+Examples:
+
+  cat example.txt | #{opts.program_name} -l en # Manually specify the language
+  cat example.kaf | #{opts.program_name}       # Uses the xml:lang attribute
+
+Languages:
+
+  * Dutch (nl)
+  * English (en)
+  * French (fr)
+  * German (de)
+  * Italian (it)
+  * Spanish (es)
+
+KAF Input:
+
+  If you give a KAF file as an input (-k or --kaf) the language is taken from
+  the xml:lang attribute inside the file. Else it expects that you give the
+  language as an argument (-l or --language)
+
+Sample KAF syntax:
+
+  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
+  <KAF version="v1.opener" xml:lang="en">
+    <raw>This is some text.</raw>
+  </KAF>
+          EOF
+        end
+      end
+
+      ##
+      # @param [String] input
+      #
+      def run(input)
+        option_parser.parse!(options[:args])
+
+        tokenizer = Tokenizer.new(options)
+
+        stdout, stderr, process = tokenizer.run(input)
+
+        if process.success?
+          puts stdout
+
+          STDERR.puts(stderr) unless stderr.empty?
+        else
+          abort stderr
+        end
+      end
+
+      private
+
+      ##
+      # Shows the help message and exits the program.
+      #
+      def show_help
+        abort option_parser.to_s
+      end
+
+      ##
+      # Shows the version and exits the program.
+      #
+      def show_version
+        abort "#{option_parser.program_name} v#{VERSION} on #{RUBY_DESCRIPTION}"
+      end
+    end # CLI
+  end # Tokenizer
+end # Opener

data/lib/opener/tokenizer/public/markdown.css

@@ -0,0 +1,283 @@
+input[type="text"], textarea
+{
+  width: 500px;
+}
+
+body {
+  font-family: Helvetica, arial, sans-serif;
+  font-size: 14px;
+  line-height: 1.6;
+  padding-top: 10px;
+  padding-bottom: 10px;
+  background-color: white;
+  padding: 30px; }
+
+body > *:first-child {
+  margin-top: 0 !important; }
+body > *:last-child {
+  margin-bottom: 0 !important; }
+
+a {
+  color: #4183C4; }
+a.absent {
+  color: #cc0000; }
+a.anchor {
+  display: block;
+  padding-left: 30px;
+  margin-left: -30px;
+  cursor: pointer;
+  position: absolute;
+  top: 0;
+  left: 0;
+  bottom: 0; }
+
+h1, h2, h3, h4, h5, h6 {
+  margin: 20px 0 10px;
+  padding: 0;
+  font-weight: bold;
+  -webkit-font-smoothing: antialiased;
+  cursor: text;
+  position: relative; }
+
+h1:hover a.anchor, h2:hover a.anchor, h3:hover a.anchor, h4:hover a.anchor, h5:hover a.anchor, h6:hover a.anchor {
+  background: url("../../images/modules/styleguide/para.png") no-repeat 10px center;
+  text-decoration: none; }
+
+h1 tt, h1 code {
+  font-size: inherit; }
+
+h2 tt, h2 code {
+  font-size: inherit; }
+
+h3 tt, h3 code {
+  font-size: inherit; }
+
+h4 tt, h4 code {
+  font-size: inherit; }
+
+h5 tt, h5 code {
+  font-size: inherit; }
+
+h6 tt, h6 code {
+  font-size: inherit; }
+
+h1 {
+  font-size: 28px;
+  color: black; }
+
+h2 {
+  font-size: 24px;
+  border-bottom: 1px solid #cccccc;
+  color: black; }
+
+h3 {
+  font-size: 18px; }
+
+h4 {
+  font-size: 16px; }
+
+h5 {
+  font-size: 14px; }
+
+h6 {
+  color: #777777;
+  font-size: 14px; }
+
+p, blockquote, ul, ol, dl, li, table, pre {
+  margin: 15px 0; }
+
+hr {
+  background: transparent url("../../images/modules/pulls/dirty-shade.png") repeat-x 0 0;
+  border: 0 none;
+  color: #cccccc;
+  height: 4px;
+  padding: 0; }
+
+body > h2:first-child {
+  margin-top: 0;
+  padding-top: 0; }
+body > h1:first-child {
+  margin-top: 0;
+  padding-top: 0; }
+body > h1:first-child + h2 {
+  margin-top: 0;
+  padding-top: 0; }
+body > h3:first-child, body > h4:first-child, body > h5:first-child, body > h6:first-child {
+  margin-top: 0;
+  padding-top: 0; }
+
+a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
+  margin-top: 0;
+  padding-top: 0; }
+
+h1 p, h2 p, h3 p, h4 p, h5 p, h6 p {
+  margin-top: 0; }
+
+li p.first {
+  display: inline-block; }
+
+ul, ol {
+  padding-left: 30px; }
+
+ul :first-child, ol :first-child {
+  margin-top: 0; }
+
+ul :last-child, ol :last-child {
+  margin-bottom: 0; }
+
+dl {
+  padding: 0; }
+dl dt {
+  font-size: 14px;
+  font-weight: bold;
+  font-style: italic;
+  padding: 0;
+  margin: 15px 0 5px; }
+dl dt:first-child {
+  padding: 0; }
+dl dt > :first-child {
+  margin-top: 0; }
+dl dt > :last-child {
+  margin-bottom: 0; }
+dl dd {
+  margin: 0 0 15px;
+  padding: 0 15px; }
+dl dd > :first-child {
+  margin-top: 0; }
+dl dd > :last-child {
+  margin-bottom: 0; }
+
+blockquote {
+  border-left: 4px solid #dddddd;
+  padding: 0 15px;
+  color: #777777; }
+blockquote > :first-child {
+  margin-top: 0; }
+blockquote > :last-child {
+  margin-bottom: 0; }
+
+table {
+  padding: 0; }
+table tr {
+  border-top: 1px solid #cccccc;
+  background-color: white;
+  margin: 0;
+  padding: 0; }
+table tr:nth-child(2n) {
+  background-color: #f8f8f8; }
+table tr th {
+  font-weight: bold;
+  border: 1px solid #cccccc;
+  text-align: left;
+  margin: 0;
+  padding: 6px 13px; }
+table tr td {
+  border: 1px solid #cccccc;
+  text-align: left;
+  margin: 0;
+  padding: 6px 13px; }
+table tr th :first-child, table tr td :first-child {
+  margin-top: 0; }
+table tr th :last-child, table tr td :last-child {
+  margin-bottom: 0; }
+
+img {
+  max-width: 100%; }
+
+span.frame {
+  display: block;
+  overflow: hidden; }
+span.frame > span {
+  border: 1px solid #dddddd;
+  display: block;
+  float: left;
+  overflow: hidden;
+  margin: 13px 0 0;
+  padding: 7px;
+  width: auto; }
+span.frame span img {
+  display: block;
+  float: left; }
+span.frame span span {
+  clear: both;
+  color: #333333;
+  display: block;
+  padding: 5px 0 0; }
+span.align-center {
+  display: block;
+  overflow: hidden;
+  clear: both; }
+span.align-center > span {
+  display: block;
+  overflow: hidden;
+  margin: 13px auto 0;
+  text-align: center; }
+span.align-center span img {
+  margin: 0 auto;
+  text-align: center; }
+span.align-right {
+  display: block;
+  overflow: hidden;
+  clear: both; }
+span.align-right > span {
+  display: block;
+  overflow: hidden;
+  margin: 13px 0 0;
+  text-align: right; }
+span.align-right span img {
+  margin: 0;
+  text-align: right; }
+span.float-left {
+  display: block;
+  margin-right: 13px;
+  overflow: hidden;
+  float: left; }
+span.float-left span {
+  margin: 13px 0 0; }
+span.float-right {
+  display: block;
+  margin-left: 13px;
+  overflow: hidden;
+  float: right; }
+span.float-right > span {
+  display: block;
+  overflow: hidden;
+  margin: 13px auto 0;
+  text-align: right; }
+
+code, tt {
+  margin: 0 2px;
+  padding: 0 5px;
+  white-space: nowrap;
+  border: 1px solid #eaeaea;
+  background-color: #f8f8f8;
+  border-radius: 3px; }
+
+pre code {
+  margin: 0;
+  padding: 0;
+  white-space: pre;
+  border: none;
+  background: transparent; }
+
+.highlight pre {
+  background-color: #f8f8f8;
+  border: 1px solid #cccccc;
+  font-size: 13px;
+  line-height: 19px;
+  overflow: auto;
+  padding: 6px 10px;
+  border-radius: 3px; }
+
+pre {
+  background-color: #f8f8f8;
+  border: 1px solid #cccccc;
+  font-size: 13px;
+  line-height: 19px;
+  overflow: auto;
+  padding: 6px 10px;
+  border-radius: 3px; }
+pre code, pre tt {
+  background-color: transparent;
+  border: none; }
+

data/lib/opener/tokenizer/server.rb

@@ -0,0 +1,16 @@
+require 'sinatra/base'
+require 'httpclient'
+require 'opener/webservice'
+
+module Opener
+  class Tokenizer
+    ##
+    # Text tokenizer server powered by Sinatra.
+    #
+    class Server < Webservice
+      set :views, File.expand_path('../views', __FILE__)
+      text_processor Tokenizer
+      accepted_params :input, :kaf, :language
+    end # Server
+  end # Tokenizer
+end # Opener

data/lib/opener/tokenizer/version.rb

@@ -0,0 +1,5 @@
+module Opener
+  class Tokenizer
+    VERSION = "1.0.0"
+  end
+end

data/lib/opener/tokenizer/views/index.erb

@@ -0,0 +1,162 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <link type="text/css" rel="stylesheet" charset="UTF-8" href="markdown.css"/>
+    <title>Tokenizer Webservice</title>
+  </head>
+  <body>
+    <h1>Tokenizer Web Service</h1>
+
+    <h2>Example Usage</h2>
+
+    <p>
+      <pre>tokenizer-server start</pre>
+      <pre>curl -d "input=this is an english text&amp;language=en" http://localhost:9393 -XPOST</pre>
+
+      outputs:
+
+      <pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;no&quot;?&gt;
+&lt;KAF version=&quot;v1.opener&quot; xml:lang=&quot;en&quot;&gt;
+  &lt;kafHeader&gt;
+    &lt;linguisticProcessors layer=&quot;text&quot;&gt;
+      &lt;lp name=&quot;opener-sentence-splitter-en&quot; timestamp=&quot;2013-06-11T13:29:21Z&quot; version=&quot;0.0.1&quot;/&gt;
+      &lt;lp name=&quot;opener-tokenizer-en&quot; timestamp=&quot;2013-06-11T13:29:22Z&quot; version=&quot;1.0.1&quot;/&gt;
+    &lt;/linguisticProcessors&gt;
+  &lt;/kafHeader&gt;
+  &lt;text&gt;
+    &lt;wf length=&quot;4&quot; offset=&quot;0&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w1&quot;&gt;this&lt;/wf&gt;
+    &lt;wf length=&quot;2&quot; offset=&quot;5&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w2&quot;&gt;is&lt;/wf&gt;
+    &lt;wf length=&quot;2&quot; offset=&quot;8&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w3&quot;&gt;an&lt;/wf&gt;
+    &lt;wf length=&quot;7&quot; offset=&quot;11&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w4&quot;&gt;english&lt;/wf&gt;
+    &lt;wf length=&quot;4&quot; offset=&quot;19&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w5&quot;&gt;text&lt;/wf&gt;
+  &lt;/text&gt;
+&lt;/KAF&gt;</pre>
+
+
+
+      <pre>curl -d &#39;text=&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;yes&quot;?&gt;&lt;KAF xml:lang=&quot;en&quot;&gt;&lt;raw&gt;this is an english text&lt;/raw&gt;&lt;/KAF&gt;&amp;kaf=true&#39; http://localhost:9292 -XPOST</pre>
+
+      outputs:
+
+      <pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot; standalone=&quot;no&quot;?&gt;
+&lt;KAF version=&quot;v1.opener&quot; xml:lang=&quot;en&quot;&gt;
+  &lt;kafHeader&gt;
+    &lt;linguisticProcessors layer=&quot;text&quot;&gt;
+      &lt;lp name=&quot;opener-sentence-splitter-en&quot; timestamp=&quot;2013-06-11T13:26:15Z&quot; version=&quot;0.0.1&quot;/&gt;
+      &lt;lp name=&quot;opener-tokenizer-en&quot; timestamp=&quot;2013-06-11T13:26:16Z&quot; version=&quot;1.0.1&quot;/&gt;
+    &lt;/linguisticProcessors&gt;
+  &lt;/kafHeader&gt;
+  &lt;text&gt;
+    &lt;wf length=&quot;4&quot; offset=&quot;0&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w1&quot;&gt;this&lt;/wf&gt;
+    &lt;wf length=&quot;2&quot; offset=&quot;5&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w2&quot;&gt;is&lt;/wf&gt;
+    &lt;wf length=&quot;2&quot; offset=&quot;8&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w3&quot;&gt;an&lt;/wf&gt;
+    &lt;wf length=&quot;7&quot; offset=&quot;11&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w4&quot;&gt;english&lt;/wf&gt;
+    &lt;wf length=&quot;4&quot; offset=&quot;19&quot; para=&quot;1&quot; sent=&quot;1&quot; wid=&quot;w5&quot;&gt;text&lt;/wf&gt;
+  &lt;/text&gt;
+&lt;/KAF&gt;</pre>
+    </p>
+
+    <h2>Try the webservice</h2>
+
+    <p>* required</p>
+    <p>** When entering a value no response will be displayed in the browser.</p>
+
+    <form action="<%=url("/")%>" method="POST">
+      <div>
+        <label for="input"/>Type your text here*</label>
+        <br/>
+
+        <textarea name="input" id="input" rows="10" cols="50"/></textarea>
+      </div>
+
+      <% 10.times do |t| %>
+      <div>
+        <label for="callbacks">Callback URL <%=t+1%>(**)</label>
+        <br />
+
+        <input id="callbacks" type="text" name="callbacks[]" />
+      </div>
+      <% end %>
+
+
+      <div>
+        <label for="error_callback">Error Callback</label>
+        <br />
+
+        <input id="error_callback" type="text" name="error_callback" />
+      </div>
+      <div>
+        <label for="kaf">
+          <input type="checkbox" name="kaf" value="false" id="kaf"/>
+          The input is in raw text (as opposed to kaf) format.
+        </label>
+
+        <br/>
+
+        <label for="language">
+          Choose the language of the text from the list.
+          <select name="language" id="language">
+            <option value="en">English</option>
+            <option value="de">German</option>
+            <option value="nl">Dutch</option>
+            <option value="fr">French</option>
+            <option value="es">Spanish</option>
+            <option value="it">Italian</option>
+          </select>
+        </label>
+      </div>
+
+      <input type="submit" value="Submit" />
+    </form>
+
+    <h2>Actions</h2>
+
+    <p>
+      <dl>
+        <dt>POST /</dt>
+        <dd>Tokenize the input text. See arguments listing for more options.</dd>
+        <dt>GET /</dt>
+        <dd>Show this page</dd>
+      </dl>
+    </p>
+
+    <h2>Arguments</h2>
+
+    <p> The webservice takes the following arguments: </p>
+    <p>* required</p>
+
+    <dl>
+      <dt>text*</dt>
+      <dd>The input text</dd>
+      <dt>kaf [true | false]</dt>
+      <dd>The input is in KAF format.</dd>
+      <dt>language [English | German | Dutch | French | Spanish | Italian]</dt>
+      <dd>The language of the provided text</dt>
+      <dt>callbacks</dt>
+      <dd>
+            You can provide a list of callback urls. If you provide callback urls
+            the tokenizer will run as a background job and a callback
+            with the results will be performed (POST) to the first url in the callback
+            list. The other urls in callback list will be provided in the "callbacks"
+            argument.<br/><br/>
+            Using callback you can chain together several OpeNER webservices in
+        one call. The first, will call the second, which will call the third, etc.
+        See for more information the <a href="http://opener-project.github.io">
+          webservice documentation online</a>.
+      </dd>
+      <dt>error_callback</dt>
+      <dd>URL to notify if errors occur in the background process. The error
+      callback will do a POST with the error message in the 'error' field.</dd>
+      </dt>
+
+
+
+    </dl>
+
+
+    <p>
+
+    </p>
+
+  </body>
+</html>

data/lib/opener/tokenizer/views/result.erb

@@ -0,0 +1,15 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <link type="text/css" rel="stylesheet" charset="UTF-8" href="markdown.css"/>
+    <title>Language Detector Webservice</title>
+  </head>
+  <body>
+    <h1>Output URL</h1>
+    <p>
+      When ready, you can view the result
+      <a href=<%= output_url %>>here</a>
+    </p>
+
+  </body>
+</html>

data/lib/opener/tokenizer.rb

@@ -0,0 +1,109 @@
+require 'opener/tokenizers/base'
+require 'nokogiri'
+require 'open3'
+require 'optparse'
+
+require_relative 'tokenizer/version'
+require_relative 'tokenizer/cli'
+
+module Opener
+  ##
+  # Primary tokenizer class that delegates the work to the various language
+  # specific tokenizers.
+  #
+  # @!attribute [r] options
+  #  @return [Hash]
+  #
+  class Tokenizer
+    attr_reader :options
+
+    ##
+    # The default language to use when no custom one is specified.
+    #
+    # @return [String]
+    #
+    DEFAULT_LANGUAGE = 'en'.freeze
+
+    ##
+    # Hash containing the default options to use.
+    #
+    # @return [Hash]
+    #
+    DEFAULT_OPTIONS = {
+      :args     => [],
+      :kaf      => true,
+      :language => DEFAULT_LANGUAGE
+    }.freeze
+
+    ##
+    # @param [Hash] options
+    #
+    # @option options [Array] :args Collection of arbitrary arguments to pass
+    #  to the individual tokenizer commands.
+    # @option options [String] :language The language to use for the
+    #  tokenization process.
+    # @option options [TrueClass|FalseClass] :kaf When set to `true` the input
+    #  is assumed to be KAF.
+    #
+    def initialize(options = {})
+      @options = DEFAULT_OPTIONS.merge(options)
+    end
+
+    ##
+    # Processes the input and returns an array containing the output of STDOUT,
+    # STDERR and an object containing process information.
+    #
+    # @param [String] input
+    # @return [Array]
+    #
+    def run(input)
+
+      if options[:kaf]
+        language, input = kaf_elements(input)
+      else
+        language = options[:language]
+      end
+
+      unless valid_language?(language)
+        raise ArgumentError, "The specified language (#{language}) is invalid"
+      end
+
+      kernel = language_constant(language).new(:args => options[:args])
+
+      return Open3.capture3(*kernel.command.split(" "), :stdin_data => input)
+    end
+
+    alias tokenize run
+
+    private
+
+    ##
+    # Returns an Array containing the language an input from a KAF document.
+    #
+    # @param [String] input The KAF document.
+    # @return [Array]
+    #
+    def kaf_elements(input)
+      document = Nokogiri::XML(input)
+      language = document.at('KAF').attr('xml:lang')
+      text     = document.at('raw').text
+
+      return language, text
+    end
+
+    ##
+    # @param [String] language
+    # @return [Class]
+    #
+    def language_constant(language)
+      Opener::Tokenizers.const_get(language.upcase)
+    end
+
+    ##
+    # @return [TrueClass|FalseClass]
+    #
+    def valid_language?(language)
+      return Opener::Tokenizers.const_defined?(language.upcase)
+    end
+  end # Tokenizer
+end # Opener

data/opener-tokenizer.gemspec

@@ -0,0 +1,36 @@
+require File.expand_path('../lib/opener/tokenizer/version', __FILE__)
+
+Gem::Specification.new do |gem|
+  gem.name        = 'opener-tokenizer'
+  gem.version     = Opener::Tokenizer::VERSION
+  gem.authors     = ['development@olery.com']
+  gem.summary     = 'Gem that wraps up the the tokenizer cores'
+  gem.description = gem.summary
+  gem.homepage    = 'http://opener-project.github.com/'
+  gem.has_rdoc    = "yard"
+
+  gem.required_ruby_version = '>= 1.9.2'
+
+  gem.files = Dir.glob([
+    'exec/**/*',
+    'lib/**/*',
+    'config.ru',
+    '*.gemspec',
+    'README.md'
+  ]).select { |file| File.file?(file) }
+
+  gem.executables = Dir.glob('bin/*').map { |file| File.basename(file) }
+
+  gem.add_dependency 'opener-tokenizer-base', '>= 0.3.1'
+  gem.add_dependency 'opener-webservice'
+
+  gem.add_dependency 'nokogiri'
+  gem.add_dependency 'sinatra', '~>1.4.2'
+  gem.add_dependency 'httpclient'
+  gem.add_dependency 'opener-daemons'
+
+  gem.add_development_dependency 'rspec'
+  gem.add_development_dependency 'cucumber'
+  gem.add_development_dependency 'pry'
+  gem.add_development_dependency 'rake'
+end

metadata

@@ -0,0 +1,200 @@
+--- !ruby/object:Gem::Specification
+name: opener-tokenizer
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- development@olery.com
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-05-20 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: opener-tokenizer-base
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.3.1
+- !ruby/object:Gem::Dependency
+  name: opener-webservice
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sinatra
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.4.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.4.2
+- !ruby/object:Gem::Dependency
+  name: httpclient
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: opener-daemons
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: cucumber
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Gem that wraps up the the tokenizer cores
+email: 
+executables:
+- tokenizer
+- tokenizer-daemon
+- tokenizer-server
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- bin/tokenizer
+- bin/tokenizer-daemon
+- bin/tokenizer-server
+- config.ru
+- exec/tokenizer.rb
+- lib/opener/tokenizer.rb
+- lib/opener/tokenizer/cli.rb
+- lib/opener/tokenizer/public/markdown.css
+- lib/opener/tokenizer/server.rb
+- lib/opener/tokenizer/version.rb
+- lib/opener/tokenizer/views/index.erb
+- lib/opener/tokenizer/views/result.erb
+- opener-tokenizer.gemspec
+homepage: http://opener-project.github.com/
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 1.9.2
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.2.2
+signing_key: 
+specification_version: 4
+summary: Gem that wraps up the the tokenizer cores
+test_files: []
+has_rdoc: yard