opener-tokenizer-base 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 52254f5259f0ae95c92f14b961c4c56699da2ae0
+  data.tar.gz: 5c303cf8e7fd2d876e3f88dd66a2da9b6dc29b64
+SHA512:
+  metadata.gz: e1da706e735d5e3a872aa1249e29b5e612b16a8773341d496448ebccc2c114946b93fe46967e5cc8fd230ea9871496dd646b23da0bde6929a4ec8b740ab61f66
+  data.tar.gz: 10052efe22ae37be19137afe01c8b1770389eb8bd5ab94614783b384d605ea67a5bd7ba4927e24f8ca6347a91d95303a5f1c1a2fd367eff18d9e24e0412f1b14

data/README.md

@@ -0,0 +1,148 @@
+[![Build Status](https://drone.io/github.com/opener-project/tokenizer-base/status.png)](https://drone.io/github.com/opener-project/tokenizer-base/latest)
+# Opener::Tokenizer::Base
+
+Base tokenizer for various languages such as English, German and Italian. Keep
+in mind that this tokenizer supports multiple languages and as such requires
+you to specify said language in a commandline option. The language is specified
+using the `-l` option. The following languages are supported:
+
+* en
+* es
+* it
+* nl
+* de
+* fr
+
+More languages may be supported in the future.
+
+## Quick Use Overview
+
+Install the Gem using Specific Install
+
+    gem specific_install opener-tokenizer-base \
+        -l https://github.com/opener-project/tokenizer-base.git
+
+If you dont have specific\_install already, install it first:
+
+    gem intall specific_install
+
+You should now be able to call the tokenizer as a regular shell command, by its
+name. Once installed as a gem you can access the gem from anywhere. This aplication
+reads a text from standard input in order to tokenize.
+
+    echo "This is an English text." | tokenizer-base -l en
+
+For more information about the available CLI options run the following:
+
+    tokenizer-base --help
+
+## Requirements
+
+* Perl 5.14.2 or newer.
+* Ruby 1.9.3 or newer (1.9.2 should work too but 1.9.3. is recommended). Ruby
+  2 is supported.
+
+## Installation
+
+To set up the project run the following commands:
+
+    bundle install
+    bundle exec rake compile
+
+This will install all the dependencies and generate the Java files. To run all
+the tests (including the process of building the files first) you can run the
+following:
+
+    bundle exec rake
+
+or:
+
+    bundle exec rake test
+
+Building a new Gem can be done as following:
+
+    bundle exec rake build
+
+For more information invoke `rake -T` or take a look at the Rakefile.
+
+
+## Gem Installation
+
+Add the following to your Gemfile (use Git for now):
+
+    gem 'opener-tokenizer-base',
+      :git=>"git@github.com:opener-project/tokenizer-base.git"
+
+
+## Usage
+
+Once installed, the tokenizer can be called as a shell command. It reads the
+standard input, and writes to standard output.
+
+It is mandatory to set the language as a parameter (there is no default
+language nor automatic detection inside). Providing no language parameter will
+raise an error.  To set a language, it has to be preceded by -l
+
+    echo "Tokenizer example." | tokenizer-base -l en
+
+or you can use the convenience option
+
+    echo "Tokenizer example." | tokenizer-it
+
+The output should be:
+
+```xml
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<KAF version="v1.opener" xml:lang="en">
+  <kafHeader>
+    <linguisticProcessors layer="text">
+      <lp name="opener-sentence-splitter-en" timestamp="2013-05-31T11:39:31Z" version="0.0.1"/>
+      <lp name="opener-tokenizer-en" timestamp="2013-05-31T11:39:32Z" version="1.0.1"/>
+    </linguisticProcessors>
+  </kafHeader>
+  <text>
+    <wf length="9" offset="0" para="1" sent="1" wid="w1">Tokenizer</wf>
+    <wf length="7" offset="10" para="1" sent="1" wid="w2">example</wf>
+    <wf length="1" offset="17" para="1" sent="1" wid="w3">.</wf>
+  </text>
+</KAF>
+```
+
+If you need a static timestamp you can use the -t param.
+
+    echo "Tokenizer example." | tokenizer-base -l en -t
+
+The output will be something along the lines of the following:
+
+```xml
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<KAF version="v1.opener" xml:lang="en">
+  <kafHeader>
+    <linguisticProcessors layer="text">
+      <lp name="opener-sentence-splitter-en" timestamp="0000-00-00T00:00:00Z" version="0.0.1"/>
+      <lp name="opener-tokenizer-en" timestamp="0000-00-00T00:00:00Z" version="1.0.1"/>
+    </linguisticProcessors>
+  </kafHeader>
+  <text>
+    <wf length="9" offset="0" para="1" sent="1" wid="w1">Tokenizer</wf>
+    <wf length="7" offset="10" para="1" sent="1" wid="w2">example</wf>
+    <wf length="1" offset="17" para="1" sent="1" wid="w3">.</wf>
+  </text>
+</KAF>
+```
+
+## Possible bugs
+
+The merging of all tokenizers in one has been done quite quickly. It seems to
+work so far, but there can be bugs, or lack of some functionality. As the
+tokenizer is the first step of the chain, any error will affect to the analysis
+of the rest of the layers.
+
+
+## Contributing
+
+1. Pull it
+2. Create your feature branch (`git checkout -b features/my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin features/my-new-feature`)
+5. If you're confident, merge your changes into master.

data/bin/tokenizer-base

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::Base.new
+puts kernel.run

data/bin/tokenizer-de

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::DE.new(:language=>"de")
+puts kernel.run

data/bin/tokenizer-en

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::EN.new(:language=>"en")
+puts kernel.run

data/bin/tokenizer-es

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::ES.new(:language=>"es")
+puts kernel.run

data/bin/tokenizer-fr

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::FR.new(:language=>"fr")
+puts kernel.run

data/bin/tokenizer-it

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::IT.new(:language=>"it")
+puts kernel.run

data/bin/tokenizer-nl

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+require_relative '../lib/opener/tokenizers/base'
+
+kernel = Opener::Tokenizers::NL.new(:language=>"nl")
+puts kernel.run

data/core/lib/Data/OptList.pm

@@ -0,0 +1,256 @@
+use strict;
+use warnings;
+package Data::OptList;
+BEGIN {
+  $Data::OptList::VERSION = '0.107';
+}
+# ABSTRACT: parse and validate simple name/value option pairs
+
+use List::Util ();
+use Params::Util ();
+use Sub::Install 0.921 ();
+
+
+my %test_for;
+BEGIN {
+  %test_for = (
+    CODE   => \&Params::Util::_CODELIKE,  ## no critic
+    HASH   => \&Params::Util::_HASHLIKE,  ## no critic
+    ARRAY  => \&Params::Util::_ARRAYLIKE, ## no critic
+    SCALAR => \&Params::Util::_SCALAR0,   ## no critic
+  );
+}
+
+sub __is_a {
+  my ($got, $expected) = @_;
+
+  return List::Util::first { __is_a($got, $_) } @$expected if ref $expected;
+
+  return defined (
+    exists($test_for{$expected})
+    ? $test_for{$expected}->($got)
+    : Params::Util::_INSTANCE($got, $expected) ## no critic
+  );
+}
+
+sub mkopt {
+  my ($opt_list) = shift;
+
+  my ($moniker, $require_unique, $must_be); # the old positional args
+  my $name_test;
+
+  if (@_ == 1 and Params::Util::_HASHLIKE($_[0])) {
+    my $arg = $_[0];
+    ($moniker, $require_unique, $must_be, $name_test)
+      = @$arg{ qw(moniker require_unique must_be name_test) };
+  } else {
+    ($moniker, $require_unique, $must_be) = @_;
+  }
+
+  $moniker = 'unnamed' unless defined $moniker;
+
+  return [] unless $opt_list;
+
+  $name_test ||= sub { ! ref $_[0] };
+
+  $opt_list = [
+    map { $_ => (ref $opt_list->{$_} ? $opt_list->{$_} : ()) } keys %$opt_list
+  ] if ref $opt_list eq 'HASH';
+
+  my @return;
+  my %seen;
+
+  for (my $i = 0; $i < @$opt_list; $i++) { ## no critic
+    my $name = $opt_list->[$i];
+    my $value;
+
+    if ($require_unique) {
+      Carp::croak "multiple definitions provided for $name" if $seen{$name}++;
+    }
+
+    if    ($i == $#$opt_list)               { $value = undef;            }
+    elsif (not defined $opt_list->[$i+1])   { $value = undef; $i++       }
+    elsif ($name_test->($opt_list->[$i+1])) { $value = undef;            }
+    else                                    { $value = $opt_list->[++$i] }
+
+    if ($must_be and defined $value) {
+      unless (__is_a($value, $must_be)) {
+        my $ref = ref $value;
+        Carp::croak "$ref-ref values are not valid in $moniker opt list";
+      }
+    }
+
+    push @return, [ $name => $value ];
+  }
+
+  return \@return;
+}
+
+
+sub mkopt_hash {
+  my ($opt_list, $moniker, $must_be) = @_;
+  return {} unless $opt_list;
+
+  $opt_list = mkopt($opt_list, $moniker, 1, $must_be);
+  my %hash = map { $_->[0] => $_->[1] } @$opt_list;
+  return \%hash;
+}
+
+
+BEGIN {
+  *import = Sub::Install::exporter {
+    exports => [qw(mkopt mkopt_hash)],
+  };
+}
+
+1;
+
+__END__
+=pod
+
+=head1 NAME
+
+Data::OptList - parse and validate simple name/value option pairs
+
+=head1 VERSION
+
+version 0.107
+
+=head1 SYNOPSIS
+
+  use Data::OptList;
+
+  my $options = Data::OptList::mkopt([
+    qw(key1 key2 key3 key4),
+    key5 => { ... },
+    key6 => [ ... ],
+    key7 => sub { ... },
+    key8 => { ... },
+    key8 => [ ... ],
+  ]);
+
+...is the same thing, more or less, as:
+
+  my $options = [
+    [ key1 => undef,        ],
+    [ key2 => undef,        ],
+    [ key3 => undef,        ],
+    [ key4 => undef,        ],
+    [ key5 => { ... },      ],
+    [ key6 => [ ... ],      ],
+    [ key7 => sub { ... },  ],
+    [ key8 => { ... },      ],
+    [ key8 => [ ... ],      ],
+  ]);
+
+=head1 DESCRIPTION
+
+Hashes are great for storing named data, but if you want more than one entry
+for a name, you have to use a list of pairs.  Even then, this is really boring
+to write:
+
+  $values = [
+    foo => undef,
+    bar => undef,
+    baz => undef,
+    xyz => { ... },
+  ];
+
+Just look at all those undefs!  Don't worry, we can get rid of those:
+
+  $values = [
+    map { $_ => undef } qw(foo bar baz),
+    xyz => { ... },
+  ];
+
+Aaaauuugh!  We've saved a little typing, but now it requires thought to read,
+and thinking is even worse than typing... and it's got a bug!  It looked right,
+didn't it?  Well, the C<< xyz => { ... } >> gets consumed by the map, and we
+don't get the data we wanted.
+
+With Data::OptList, you can do this instead:
+
+  $values = Data::OptList::mkopt([
+    qw(foo bar baz),
+    xyz => { ... },
+  ]);
+
+This works by assuming that any defined scalar is a name and any reference
+following a name is its value.
+
+=head1 FUNCTIONS
+
+=head2 mkopt
+
+  my $opt_list = Data::OptList::mkopt($input, \%arg);
+
+Valid arguments are:
+
+  moniker        - a word used in errors to describe the opt list; encouraged
+  require_unique - if true, no name may appear more than once
+  must_be        - types to which opt list values are limited (described below)
+  name_test      - a coderef used to test whether a value can be a name
+                   (described below, but you probably don't want this)
+
+This produces an array of arrays; the inner arrays are name/value pairs.
+Values will be either "undef" or a reference.
+
+Positional parameters may be used for compability with the old C<mkopt>
+interface:
+
+  my $opt_list = Data::OptList::mkopt($input, $moniker, $req_uni, $must_be);
+
+Valid values for C<$input>:
+
+ undef    -> []
+ hashref  -> [ [ key1 => value1 ] ... ] # non-ref values become undef
+ arrayref -> every name followed by a non-name becomes a pair: [ name => ref ]
+             every name followed by undef becomes a pair: [ name => undef ]
+             otherwise, it becomes [ name => undef ] like so:
+             [ "a", "b", [ 1, 2 ] ] -> [ [ a => undef ], [ b => [ 1, 2 ] ] ]
+
+By default, a I<name> is any defined non-reference.  The C<name_test> parameter
+can be a code ref that tests whether the argument passed it is a name or not.
+This should be used rarely.  Interactions between C<require_unique> and
+C<name_test> are not yet particularly elegant, as C<require_unique> just tests
+string equality.  B<This may change.>
+
+The C<must_be> parameter is either a scalar or array of scalars; it defines
+what kind(s) of refs may be values.  If an invalid value is found, an exception
+is thrown.  If no value is passed for this argument, any reference is valid.
+If C<must_be> specifies that values must be CODE, HASH, ARRAY, or SCALAR, then
+Params::Util is used to check whether the given value can provide that
+interface.  Otherwise, it checks that the given value is an object of the kind.
+
+In other words:
+
+  [ qw(SCALAR HASH Object::Known) ]
+
+Means:
+
+  _SCALAR0($value) or _HASH($value) or _INSTANCE($value, 'Object::Known')
+
+=head2 mkopt_hash
+
+  my $opt_hash = Data::OptList::mkopt_hash($input, $moniker, $must_be);
+
+Given valid C<L</mkopt>> input, this routine returns a reference to a hash.  It
+will throw an exception if any name has more than one value.
+
+=head1 EXPORTS
+
+Both C<mkopt> and C<mkopt_hash> may be exported on request.
+
+=head1 AUTHOR
+
+Ricardo Signes <rjbs@cpan.org>
+
+=head1 COPYRIGHT AND LICENSE
+
+This software is copyright (c) 2006 by Ricardo Signes.
+
+This is free software; you can redistribute it and/or modify it under
+the same terms as the Perl 5 programming language system itself.
+
+=cut
+