opener-webservice 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 02a6433aa01c06b1251256710c706a59e45586b2
-  data.tar.gz: 6dc2b0270835171445461234f8123fb13e267d36
+  metadata.gz: 32302b9a2aac215c69670cd55e4c28d22901df9c
+  data.tar.gz: d24fb9f1fc3113ff70685bd1c2ce440eced844c6
 SHA512:
-  metadata.gz: 733000a6cd389bcfbdfd0865432cc45c4896d4564a653fa7ad5a14a123d1cf5eb28ab0202e13fa47daa4e5f2b4a64f88f753abf3046d71fc11d7198329468c02
-  data.tar.gz: cfd032408fe8d59f391577d029eb96880684eb93ff61f4920f6264ac9ac214bed9d869b52b81d612c95ac386f552e43d8c5055e2024aa82e52a4597a778285fc
+  metadata.gz: 5c96ce12b9449ee865ee208180fed338a7bdb9077741cecc23a618cd31fe499570a0fa750f905ca1f17ebebb09294a9a431fa83d7c43910a6890a83779c68285
+  data.tar.gz: 602204c799cca06cf486f1fb44c1f6d2ea2a8f53417916313a08e7e80a035ef2e6f4c2eb75f784ce0f3ac7389c121851b041ef8b7a628493f2e1aaec2334663a

data/README.md

@@ -1,29 +1,147 @@
-# Opener::Webservice
+# Opener Webservice
 
-TODO: Write a gem description
+This Gem makes it possible for OpeNER components to be used as a webservice.
+Input can be passed directly or using an URL, the latter allows for greater data
+sizes to be processed. Webservices can be chained together using callback URLs,
+each passing its output to the next callback. Output can either be passed
+directly, or as a URL pointing to a document in Amazon S3.
 
-## Installation
+## Usage
 
-Add this line to your application's Gemfile:
+Create an executable file `bin/<component>-server`, for example
+`bin/language-identifier-server`, with the following content:
 
-    gem 'opener-webservice'
+    #!/usr/bin/env ruby
 
-And then execute:
+    require 'opener/webservice'
 
-    $ bundle
+    parser = Opener::Webservice::OptionParser.new(
+      'opener-<component>',
+      File.expand_path('../../config.ru', __FILE__)
+    )
 
-Or install it yourself as:
+    parser.run
 
-    $ gem install opener-webservice
+Replace `<component>` with the name of the component. For example, for the
+language identifier this would result in the following:
 
-## Usage
+    #!/usr/bin/env ruby
+
+    require 'opener/webservice'
+
+    parser = Opener::Webservice::OptionParser.new(
+      'opener-language-identifier',
+      File.expand_path('../../config.ru', __FILE__)
+    )
+
+    parser.run
+
+Next, create a `config.ru` file in the root directory of the component. It
+should have the following content:
+
+    require File.expand_path('../lib/opener/<component>', __FILE__)
+    require File.expand_path('../lib/opener/<component>/server', __FILE__)
+
+    run Opener::<constant>::Server
+
+Replace `<component>` with the component name, replace `<constant>` with the
+corresponding constant. For example, for the language identifier:
+
+    require File.expand_path('../lib/opener/language_identifier', __FILE__)
+    require File.expand_path('../lib/opener/language_identifier/server', __FILE__)
+
+    run Opener::LanguageIdentifier::Server
+
+## Input
+
+To submit data, send a POST request to the root URL of a webservice. The request
+body can either be a set of POST fields, or a JSON object. In both cases the
+following fields can be set:
+
+* `input`: direct input to process
+* `input_url`: a URL to a document to download and process
+* `callbacks`: an array of callback URLs to send output to
+* `error_callback`: a URL to send errors to
+* `request_id`: a custom request ID/identifier to associate with the document
+* `metadata`: an arbitrary metadata object to associate with a document, only
+  supported when using JSON input as POST fields can't represent key/values.
+
+Any other parameters are ignored _but_ passed along to the next callback (if
+any).
+
+To use JSON input, set the `Content-Type` header to `application/json` when
+submitting data.
+
+If no callback URLs are specified the data is processed synchronously, the
+response will be whatever output the underlying component returned (usually
+KAF).
+
+When using a callback URL the response will be a JSON object containing:
+
+* `request_id`: the generated (or manually specified) request ID/identifier
+* `output_url`: the URL that will contain the end output after all callbacks
+  have been processed
+
+If an error occurs the output URL will _not_ contain the document, instead a
+POST request is executed using the URL in the `error_callback` field. This URL
+receives the following parameters:
+
+* `request_id`: The ID of the request/document that failed
+* `error`: the error message
+
+## Requirements
+
+* A supported Ruby version (see below)
+* Amazon S3 (only when one wants to store ouput in S3)
+
+The following Ruby versions are supported:
+
+| Ruby     | Required      | Recommended |
+|:---------|:--------------|:------------|
+| MRI      | >= 1.9.3      | >= 2.1.4    |
+| Rubinius | >= 2.2        | >= 2.3.0    |
+| JRuby    | >= 1.7        | >= 1.7.16   |
+
+Note that various components use JRuby, thus they won't work on MRI and
+Rubinius.
+
+## S3 Support
+
+To enable storing of output on Amazon S3, specify the `--bucket` option when
+running the CLI. Also make sure that the following environment variables are
+set:
+
+* `AWS_ACCESS_KEY_ID`
+* `AWS_SECRET_ACCESS_KEY`
+* `AWS_REGION`
+
+If you're running this daemon on an EC2 instance then the first two environment
+variables will be set automatically if the instance has an associated IAM
+profile. The `AWS_REGION` variable must _always_ be set.
+
+Output files are named `<identifier>.xml` where `<identifier>` is the unique
+identifier of the document. The content type of these documents is set to
+`application/xml`.  Metadata associated with the job (as specified in the
+`metadata` field) is saved as metadata of the S3 object.
+
+The S3 URLs are only valid for a limited time (currently 1 hour) so callbacks
+must ensure they can process the input within that time limit.
+
+To use custom identifiers for documents, specify a unique value in the
+`request_id` parameter when submitting data. Existing documents using the same
+identifier will be _overwritten_, so make sure your identifiers are truly
+unique. Default identifiers are generated using Ruby's `SecureRandom.hex`
+method.
+
+## Monitoring
 
-TODO: Write usage instructions here
+Components using this Gem can measure performance using New Relic and report
+errors using Rollbar. To support this the following two environment variables
+must be set:
 
-## Contributing
+* `NEWRELIC_TOKEN`
+* `ROLLBAR_TOKEN`
 
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+For New Relic the application names will be `opener-<component>` where
+`<component>` is the component name, as defined by a component itself. If one of
+these environment variables is not set the corresponding feature is disabled.

data/lib/opener/webservice/configuration.rb

@@ -0,0 +1,90 @@
+module Opener
+  module Webservice
+    ##
+    # Module for storing global configuration settings such as whether or not to
+    # enable authentication.
+    #
+    module Configuration
+      ##
+      # Returns `true` if authentication should be enabled.
+      #
+      # @return [TrueClass|FalseClass]
+      #
+      def self.authentication?
+        return !!authentication_endpoint && !authentication_endpoint.empty?
+      end
+
+      ##
+      # Returns the authentication endpoint to use.
+      #
+      # @return [String]
+      #
+      def self.authentication_endpoint
+        return ENV['AUTHENTICATION_ENDPOINT']
+      end
+
+      ##
+      # Returns the field name of the authentication secret.
+      #
+      # @return [String]
+      #
+      def self.authentication_secret
+        return ENV['AUTHENTICATION_SECRET'] || 'secret'
+      end
+
+      ##
+      # Returns the field name of the authentication token.
+      #
+      # @return [String]
+      #
+      def self.authentication_token
+        return ENV['AUTHENTICATION_TOKEN'] || 'token'
+      end
+
+      ##
+      # Name of the S3 bucket to store output in.
+      #
+      # @return [String]
+      #
+      def self.output_bucket
+        return ENV['OUTPUT_BUCKET']
+      end
+
+      ##
+      # Returns `true` if Syslog should be enabled.
+      #
+      # @return [TrueClass|FalseClass]
+      #
+      def self.syslog?
+        return !!ENV['ENABLE_SYSLOG'] && !ENV['ENABLE_SYSLOG'].empty?
+      end
+
+      ##
+      # Returns `true` if Rollbar error tracking should be enabled.
+      #
+      # @return [TrueClass|FalseClass]
+      #
+      def self.rollbar?
+        return !!ENV['ROLLBAR_TOKEN']
+      end
+
+      ##
+      # Configures Rollbar.
+      #
+      def self.configure_rollbar
+        Rollbar.configure do |config|
+          config.access_token = ENV['ROLLBAR_TOKEN']
+          config.enabled      = rollbar?
+          config.environment  = environment
+        end
+      end
+
+      ##
+      # @return [String]
+      #
+      def self.environment
+        return ENV['RACK_ENV'] || ENV['RAILS_ENV']
+      end
+    end # Configuration
+  end # Webservice
+end # Opener

data/lib/opener/webservice/error_handler.rb

@@ -0,0 +1,29 @@
+module Opener
+  module Webservice
+    ##
+    # Class for handling error messages that occur when processing a document.
+    #
+    # @!attribute [r] http
+    #  @return [HTTPClient]
+    #
+    class ErrorHandler
+      attr_reader :http
+
+      def initialize
+        @http = HTTPClient.new
+      end
+
+      ##
+      # @param [StandardError] error
+      # @param [String] request_id
+      # @param [String] url
+      #
+      def submit(error, request_id, url)
+        http.post(
+          url,
+          :body => {:error => error.message, :request_id => request_id}
+        )
+      end
+    end # ErrorHandler
+  end # Webservice
+end # Opener

data/lib/opener/webservice/input_extractor.rb

@@ -0,0 +1,43 @@
+module Opener
+  module Webservice
+    ##
+    # Extracts the KAF/text input to use from a set of input parameters.
+    #
+    # @!attribute [r] http
+    #  @return [HTTPClient]
+    #
+    class InputExtractor
+      attr_reader :http
+
+      def initialize
+        @http = HTTPClient.new
+      end
+
+      ##
+      # @param [Hash] options
+      #
+      # @option options [String] input_url A URL to download input from.
+      # @option options [String] input The direct input to process.
+      #
+      # @return [String]
+      #
+      # @raise [RuntimeError] Raised when the input could not be downloaded.
+      #
+      def extract(options)
+        if options['input_url']
+          resp = http.get(options['input_url'], :follow_redirect => true)
+
+          unless resp.ok?
+            raise "Failed to download input from #{options['input_url']}"
+          end
+
+          input = resp.body
+        else
+          input = options['input']
+        end
+
+        return input
+      end
+    end # InputExtractor
+  end # Webservice
+end # Opener

data/lib/opener/webservice/input_sanitizer.rb

@@ -0,0 +1,65 @@
+module Opener
+  module Webservice
+    ##
+    # Sanitizes raw Sinatra input and component options.
+    #
+    class InputSanitizer
+      ##
+      # Returns a Hash containing cleaned up pairs based on the input
+      # parameters. The keys of the returned Hash are String instances to
+      # prevent Symbol DOS attacks.
+      #
+      # @param [Hash] input
+      # @return [Hash]
+      #
+      def prepare_parameters(input)
+        sanitized = {}
+
+        input.each do |key, value|
+          # Sinatra/Rack uses "on" for checked checkboxes.
+          if value == 'true' or value == 'on'
+            value = true
+          elsif value == 'false'
+            value = false
+          end
+
+          sanitized[key.to_s] = value
+        end
+
+        # Strip empty callback URLs (= default form values).
+        if sanitized['callbacks']
+          sanitized['callbacks'].reject! { |url| url.nil? || url.empty? }
+        end
+
+        if sanitized['error_callback'] and sanitized['error_callback'].empty?
+          sanitized.delete('error_callback')
+        end
+
+        return sanitized
+      end
+
+      ##
+      # Returns a Hash containing the whitelisted options to pass to a
+      # component. Since components use Symbols for their options this Hash uses
+      # Symbols for its keys.
+      #
+      # @param [Hash] input
+      # @param [Array] accepted The accepted parameter names.
+      # @return [Hash]
+      #
+      def whitelist_options(input, accepted)
+        whitelisted = {}
+
+        input.each do |key, value|
+          sym_key = key.to_sym
+
+          if accepted.include?(sym_key)
+            whitelisted[sym_key] = value
+          end
+        end
+
+        return whitelisted
+      end
+    end # InputSanitizer
+  end # Webservice
+end # Opener

data/lib/opener/webservice/option_parser.rb

@@ -0,0 +1,175 @@
+module Opener
+  module Webservice
+    ##
+    # Slop wrapper for parsing webservice options and passing them to Puma.
+    #
+    # @!attribute [r] name
+    #  The name of the component.
+    #  @return [String]
+    #
+    # @!attribute [r] rackup
+    #  Path to the config.ru to use.
+    #  @return [String]
+    #
+    # @!attribute [r] parser
+    #  @return [Slop]
+    #
+    class OptionParser
+      attr_reader :name, :rackup, :parser
+
+      ##
+      # Mapping of environment variables and Slop options.
+      #
+      # @return [Hash]
+      #
+      ENV_OPTIONS = {
+        'OUTPUT_BUCKET'           => :bucket,
+        'AUTHENTICATION_TOKEN'    => :token,
+        'AUTHENTICATION_SECRET'   => :secret,
+        'AUTHENTICATION_ENDPOINT' => :authentication
+      }
+
+      ##
+      # @param [String] name
+      # @param [String] rackup
+      #
+      def initialize(name, rackup)
+        @name   = name
+        @rackup = rackup
+        @parser = configure_slop
+      end
+
+      def parse(*args)
+        parser.parse(*args)
+      end
+
+      ##
+      # Parses the given CLI options and starts Puma.
+      #
+      # @param [Array] argv
+      #
+      def run(argv = ARGV)
+        parser.parse(argv)
+      end
+
+      ##
+      # @return [Slop]
+      #
+      def configure_slop
+        outer       = self
+        server_name = "#{name}-server"
+        cli_name    = server_name.gsub('opener-', '')
+
+        return Slop.new(:strict => false, :indent => 2) do
+          banner "Usage: #{cli_name} [RACKUP] [OPTIONS]"
+
+          separator <<-EOF.chomp
+
+About:
+
+    Runs the OpeNER component as a webservice using Puma. For example:
+
+        language-identifier-server --daemon
+
+    This would start a language identifier server in the background.
+
+Environment Variables:
+
+    These daemons make use of Amazon SQS queues and other Amazon services. In
+    order to use these services you should make sure the following environment
+    variables are set:
+
+    * AWS_ACCESS_KEY_ID
+    * AWS_SECRET_ACCESS_KEY
+    * AWS_REGION
+
+    If you're running this daemon on an EC2 instance then the first two
+    environment variables will be set automatically if the instance has an
+    associated IAM profile. The AWS_REGION variable must _always_ be set.
+
+    Optionally you can also set the following extra variables:
+
+    * NEWRELIC_TOKEN: when set the daemon will send profiling data to New Relic
+      using this token. The application name will be "#{server_name}".
+
+    * ROLLBAR_TOKEN: when set the daemon will report errors to Rollbar using
+      this token. You can freely use this in combination with NEWRELIC_TOKEN.
+
+Puma Options:
+
+    This webserver uses Puma under the hood, but defines its own CLI options.
+    All unrecognized options are passed to the Puma CLI. For more information
+    on the available options for Puma, run `#{cli_name} --puma-help`.
+          EOF
+
+          separator "\nOptions:\n"
+
+          on :h, :help, 'Shows this help message' do
+            abort to_s
+          end
+
+          on :'puma-help', 'Shows the options of Puma' do
+            Puma::CLI.new(['--help']).run
+
+            abort
+          end
+
+          on :b=,
+            :bucket=,
+            'The S3 bucket to store output in',
+            :as => String
+
+          on :authentication=,
+            'An authentication endpoint to use',
+            :as => String
+
+          on :secret=,
+            'Parameter name for the authentication secret',
+            :as => String
+
+          on :token=,
+            'Parameter name for the authentication token',
+            :as => String
+
+          on :'disable-syslog', 'Disables Syslog logging (enabled by default)'
+
+          run do |opts, args|
+            puma_args = [outer.rackup] + args
+
+            ENV['APP_NAME'] = outer.name
+            ENV['APP_ROOT'] = File.expand_path('../../../../', __FILE__)
+            ENV['NRCONFIG'] = File.join(ENV['APP_ROOT'], 'config/newrelic.yml')
+
+            ENV_OPTIONS.each do |key, opt|
+              ENV[key] = opts[opt]
+            end
+
+            unless opts[:'disable-syslog']
+              ENV['ENABLE_SYSLOG'] = '1'
+            end
+
+            if !ENV['RAILS_ENV'] and ENV['RACK_ENV']
+              ENV['RAILS_ENV'] = ENV['RACK_ENV']
+            end
+
+            if ENV['NEWRELIC_TOKEN']
+              NewRelic::Control.instance.init_plugin
+
+              # Enable the GC profiler for New Relic.
+              GC::Profiler.enable
+            end
+
+            Configuration.configure_rollbar
+
+            # Puma on JRuby does some weird stuff with forking/exec. As a result
+            # of this we *have to* update ARGV as otherwise running Puma as a
+            # daemon does not work.
+            ARGV.replace(puma_args)
+
+            Puma::CLI.new(puma_args).run
+          end
+        end
+      end
+    end # OptionParser
+  end # Webservice
+end # Opener