opener-language-identifier 3.0.3 → 3.0.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dcf50a5847ebd111af672d41d6f2a40dfc6e5c2b
-  data.tar.gz: de82ac2bcc8485d334cfd9a8b84c373781abda89
+  metadata.gz: ce2775b5964868c6ad0e00519dde29c5dc1654a4
+  data.tar.gz: be35d8f78c6a32b39a77e41b9ba673c70e37c728
 SHA512:
-  metadata.gz: 45a785402359135646b38000ad9502ebbe44389d38f91c92eed73b315d2397524a3522b9d8fa9f2a6a42626bad9ae30971ed90f6301af38d0f843b197beaca58
-  data.tar.gz: 571cbb5c6a46acabedbb572832a46c3cfe565fdc0d2ceeb78168d7e58843b88659f764fcfa810bb6f418d3b054b21b3d9f5aa9556b50d3a9430e2fac704e3e2e
+  metadata.gz: 527280005269de7dadc0e7a4c8169c8f9da4f922e596e04e2765f8e69b0d5873c96e1f2c5f4d902ff7ca5fd4fa85fa9758c693f6fa1eb0cc5a905c5b41365337
+  data.tar.gz: d4cf50e2110aa86c9c908068e9fc339e4124523edeef301fcd49b42c0776cc41d3e5100681b2f12f6e480ebef4a2abb743036ac430f7abe86882a27fe38f1586

data/README.md

@@ -2,41 +2,48 @@
 
 # Language Identifier
 
-The language identifier takes raw text and tries to figure out what language it was written in. The output can either be a plain-text i18n language code or a basic KAF document containing the language and raw input text.
+The language identifier takes raw text and tries to figure out what language it
+was written in. The output can either be a plain-text i18n language code or a
+basic KAF document containing the language and raw input text.
 
-The output of the language identifier can then be used to drive further text analysis of for example sentiments and or entities.
+The output of the language identifier can then be used to drive further text
+analysis of for example sentiments and or entities.
 
-### Confused by some terminology?
+## Confused by some terminology?
 
-This software is part of a larger collection of natural language processing tools known as "the OpeNER project". You can find more information about the project at [the OpeNER portal](http://opener-project.github.io). There you can also find references to terms like KAF (an XML standard to represent linguistic annotations in texts), component, cores, scenario's and pipelines.
+This software is part of a larger collection of natural language processing
+tools known as "the OpeNER project". You can find more information about the
+project at [the OpeNER portal](http://opener-project.github.io). There you can
+also find references to terms like KAF (an XML standard to represent linguistic
+annotations in texts), component, cores, scenario's and pipelines.
 
-Quick Use Example
------------------
+## Quick Use Example
 
 Install the Gem:
 
     gem install opener-language-identifier
 
-Make sure you run ```jruby``` since the language-identifier uses Java.
+Make sure you run `jruby` since the language-identifier uses Java.
 
 ### Command line interface
 
-You should now be able to call the language indentifier as a regular shell command: by its name. Once installed the gem normally sits in your path so you can call it directly from anywhere.
+You should now be able to call the language indentifier as a regular shell
+command: by its name. Once installed the gem normally sits in your path so you
+can call it directly from anywhere.
 
-This aplication reads a text from standard input in order to identify the language.
+This aplication reads a text from standard input in order to identify the
+language.
 
     echo "This is an English text." | language-identifier
 
 This will output:
 
-~~~~
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<KAF xml:lang="en" version="2.1">
-    <raw>This is an English text.</raw>
-</KAF>
-~~~~
+    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+    <KAF xml:lang="en" version="2.1">
+        <raw>This is an English text.</raw>
+    </KAF>
 
-If you just want the language code returned add the ```--no-kaf``` option like this
+If you just want the language code returned add the `--no-kaf` option like this
 
     echo "This is an English text." | language-identifier --no-kaf
 
@@ -50,7 +57,8 @@ You can launch a language identification webservice by executing:
 
     $ language-identifier-server
 
-This will launch a mini webserver with the webservice. It defaults to port 9292, so you can access it at <http://localhost:9292/>.
+This will launch a mini webserver with the webservice. It defaults to port
+9292, so you can access it at <http://localhost:9292/>.
 
 To launch it on a different port provide the `-p [port-number]` option like
 this:
@@ -61,61 +69,58 @@ It then launches at <http://localhost:1234/>
 
 Documentation on the Webservice is provided by surfing to the urls provided
 above. For more information on how to launch a webservice run the command with
-the ```-h``` option.
+the `-h` option.
 
 ### Daemon
 
-Last but not least the language identifier comes shipped with a daemon that
-can read jobs (and write) jobs to and from Amazon SQS queues. For more
-information type:
+Last but not least the language identifier comes shipped with a daemon that can
+read jobs (and write) jobs to and from Amazon SQS queues. For more information
+type:
 
     $ language-identifier-daemon -h
 
-Description of dependencies
----------------------------
+## Description of dependencies
 
 This component runs best if you run it in an environment suited for OpeNER
-components. You can find an installation guide and helper tools in the [OpeNER installer](https://github.com/opener-project/opener-installer) and [an installation guide on the OpenerWebsite](http://opener-project.github.io/getting-started/how-to/local-installation.html)
+components. You can find an installation guide and helper tools in the
+[OpeNER installer](https://github.com/opener-project/opener-installer) and
+[an installation guide on the OpenerWebsite](http://opener-project.github.io/getting-started/how-to/local-installation.html).
 
 At least you need the following system setup:
 
 ### Dependencies for normal use:
 
-* Python 2.6 - PIP, possibly VirtualEnv
-* Jruby
-* Java 1.7 or newer (There are problems with encoding in older versions).
+* JRuby 1.7 or newer
+* Java 1.7 or newer (there are problems with encodings in older versions).
 
 ### Dependencies if you want to modify the component:
 
 * Maven (for building the Gem)
 
-Language Extension
-------------------
+## Language Extension
 
-The internal library that actually performs the language identification already supports a lot of languages. 
-For more information about how to extends it for more languages or functionalities, please, visit the website of the tool at <a href="https://code.google.com/p/language-detection/">https://code.google.com/p/language-detection/</a>
+The internal library that actually performs the language identification already
+supports a lot of languages.  For more information about how to extends it for
+more languages or functionalities, please, visit the website of the tool at
+<https://code.google.com/p/language-detection/>.
+
+## The Core
 
-The Core
---------
-  
 The component is a fat wrapper around the actual language technology core.
 Written in Java. Checkout the core/src directory of the package to get to the
 actual working component.
 
-Where to go from here
----------------------
+## Where to go from here
 
 * [Check the project website](http://opener-project.github.io)
 * [Checkout the webservice](http://opener.olery.com/language-identifier)
 
-Report problem/Get help
------------------------
+## Report problem/Get help
 
 If you encounter problems, please email support@opener-project.eu or leave an
-issue in the [issue tracker](https://github.com/opener-project/language-identifier/issues). 
+issue in the [issue tracker](https://github.com/opener-project/language-identifier/issues).
 
-Contributing
-------------
+## Contributing
 
 1. Fork it <http://github.com/opener-project/language-identifier/fork>
 2. Create your feature branch (`git checkout -b my-new-feature`)

data/lib/opener/language_identifier.rb

@@ -9,6 +9,7 @@ import 'org.vicomtech.opennlp.LanguageDetection.CybozuDetector'
 require_relative 'language_identifier/version'
 require_relative 'language_identifier/kaf_builder'
 require_relative 'language_identifier/cli'
+require_relative 'language_identifier/error_layer'
 require_relative 'language_identifier/detector.rb'
 
 module Opener
@@ -57,14 +58,19 @@ module Opener
     # @return [Array]
     #
     def run(input)
-      if options[:probs]
-        output = @detector.probabilities(input)
-      else
-        output = @detector.detect(input)
-        output = build_kaf(input, output) if @options[:kaf]
-      end
+      begin
+        if options[:probs]
+            output = @detector.probabilities(input)
+          else
+            output = @detector.detect(input)
+            output = build_kaf(input, output) if @options[:kaf]
+        end
 
-      return output
+        return output
+        
+      rescue Exception => error
+        return ErrorLayer.new(input, error.message, self.class).add
+      end
     end
 
     alias identify run

data/lib/opener/language_identifier/error_layer.rb

@@ -0,0 +1,91 @@
+require 'nokogiri'
+
+module Opener
+  class LanguageIdentifier
+    ##
+    # Add Error Layer to KAF file instead of throwing an error.
+    #
+    class ErrorLayer
+      attr_accessor :input, :document, :error, :klass
+      
+      def initialize(input, error, klass)
+        @input    = input.to_s
+        # Make sure there is always a document, even if it is empty.
+        @document = Nokogiri::XML(input) rescue Nokogiri::XML(nil)
+        @error    = error
+        @klass    = klass
+      end
+      
+      def add
+        if is_xml?
+          unless has_errors_layer?
+            add_errors_layer
+          end
+        else
+          add_root
+          add_text
+          add_errors_layer
+        end
+        add_error
+        
+        xml = !!document.encoding ? document.to_xml : document.to_xml(:encoding => "UTF-8")
+        
+        return xml
+      end
+      
+      ##
+      # Check if the document is a valid XML file.
+      #
+      def is_xml?
+        !!document.root
+      end
+      
+      ##
+      # Add root element to the XML file.
+      #
+      def add_root
+        root = Nokogiri::XML::Node.new "KAF", document
+        document.add_child(root)
+      end
+      
+      ##
+      # Check if the document already has an errors layer.
+      #
+      def has_errors_layer?
+        !!document.at('errors')
+      end
+      
+      ##
+      # Add errors element to the XML file.
+      #
+      def add_errors_layer
+        node = Nokogiri::XML::Node.new "errors", document
+        document.root.add_child(node)
+      end
+      
+      ##
+      # Add the text file incase it is not a valid XML document. More
+      # info for debugging.
+      #
+      def add_text
+        node = Nokogiri::XML::Node.new "raw", document
+        node.inner_html = input
+        document.root.add_child(node)
+        
+      end
+      
+      ##
+      # Add the actual error to the errors layer.
+      #
+      def add_error
+        node = document.at('errors')
+        error_node = Nokogiri::XML::Node.new "error", node
+        error_node['class']   = klass.to_s
+        error_node['version'] = klass::VERSION
+        error_node.inner_html = error
+        node.add_child(error_node)
+      end     
+      
+    end # ErrorLayer
+  end # LanguageIdentifier
+end # Opener

data/lib/opener/language_identifier/version.rb

@@ -1,5 +1,5 @@
 module Opener
   class LanguageIdentifier
-    VERSION = "3.0.3"
+    VERSION = "3.0.4"
   end
 end

data/opener-language-identifier.gemspec

@@ -12,7 +12,6 @@ Gem::Specification.new do |gem|
 
   gem.files = Dir.glob([
     'core/target/LanguageDetection-*.jar',
-    'core/target/classes/**/*.*',
     'core/target/classes/**/*',
     'exec/**/*',
     'lib/**/*',
@@ -28,11 +27,12 @@ Gem::Specification.new do |gem|
   gem.add_dependency 'sinatra', '~>1.4.2'
   gem.add_dependency 'httpclient'
   gem.add_dependency 'uuidtools'
-  gem.add_dependency 'opener-build-tools'
   gem.add_dependency 'opener-webservice'
   gem.add_dependency 'opener-daemons'
+  gem.add_dependency 'nokogiri'
 
-  gem.add_development_dependency 'rspec'
+  gem.add_development_dependency 'rspec', '~> 3.0'
   gem.add_development_dependency 'cucumber'
   gem.add_development_dependency 'rake'
+  gem.add_development_dependency 'cliver'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: opener-language-identifier
 version: !ruby/object:Gem::Version
-  version: 3.0.3
+  version: 3.0.4
 platform: ruby
 authors:
 - development@olery.com
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2014-05-24 00:00:00.000000000 Z
+date: 2014-06-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: builder
@@ -81,7 +81,7 @@ dependencies:
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
-  name: opener-build-tools
+  name: opener-webservice
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -95,7 +95,7 @@ dependencies:
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
-  name: opener-webservice
+  name: opener-daemons
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -109,7 +109,7 @@ dependencies:
   prerelease: false
   type: :runtime
 - !ruby/object:Gem::Dependency
-  name: opener-daemons
+  name: nokogiri
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -124,6 +124,20 @@ dependencies:
   type: :runtime
 - !ruby/object:Gem::Dependency
   name: rspec
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  prerelease: false
+  type: :development
+- !ruby/object:Gem::Dependency
+  name: cucumber
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -137,7 +151,7 @@ dependencies:
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
-  name: cucumber
+  name: rake
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -151,7 +165,7 @@ dependencies:
   prerelease: false
   type: :development
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: cliver
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
@@ -253,6 +267,7 @@ files:
 - lib/opener/language_identifier.rb
 - lib/opener/language_identifier/cli.rb
 - lib/opener/language_identifier/detector.rb
+- lib/opener/language_identifier/error_layer.rb
 - lib/opener/language_identifier/kaf_builder.rb
 - lib/opener/language_identifier/public/markdown.css
 - lib/opener/language_identifier/server.rb