opener-daemons 1.3.0 → 2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: db9deedcb0722c93dce8dbf63951426670ccfc31
-  data.tar.gz: 0c915961e981220c36f5668d89420d12eea6a615
+  metadata.gz: 84d367a150e9b90e3cef7c94b1d90db4791309f7
+  data.tar.gz: 4af1be3dd9992df37774dc7ee6f417c3343d044c
 SHA512:
-  metadata.gz: bd96be27c2fd2c3a312afdb304b903262bbfee59d3a97a46c492aae903d744928e132afbfd91cca2069d2cf68fd3ea587537269a0268ad520146c17f59e4924c
-  data.tar.gz: e460109a333e013dfbfe54228348f030475914406d8cc7faed4af9e1196cf36f04d8b07b55df2902161c6e6a82a18e957dd94a7fe0caf8e0fe70cebf47422392
+  metadata.gz: 7a45c4af290e89c800ee4a702f62f4ffb159cdc86952da55f93769747335cd44351f61d46b4dac558a573e16b1b33b38b208d0ceb3a0fbbc8454fe05cbd37f9e
+  data.tar.gz: 4cb8f013c3349958b89cfdbe2287ec4b275976e5c729df15fcd8302ab63a338ae095fcabc87734c38d4fb36160d2d5cfe7ff57707a6b8488ecac1abc27711cb6

data/README.md

@@ -1,127 +1,169 @@
-# Opener::Daemons
+# OpeNER Daemons
 
-This GEM is part of the OpeNER project, which is the NLP toolchain for the rest
-of us. This particular GEM makes is possible that al OpeNER components can
-actually be launched as deamons reading from and push to Amazon SQS queues.
+This Gem makes it possible for OpeNER components to be used as a daemon using
+Amazon SQS and Amazon S3. SQS is used for job input while S3 is used for storing
+results. Daemons only take URLs as input, they don't allow text to be specified
+directly due to size restrictions of SQS (a maximum of 256 KB).
 
-## Installation
+## Usage
 
-Add this line to your application's Gemfile:
+Create an executable file `bin/<component>-daemon`, for example
+`bin/language-identifier-daemon`, with the following content:
 
-    gem 'opener-daemons'
+    #!/usr/bin/env ruby
+    require 'opener/daemons'
 
-And then execute:
+    controller = Opener::Daemons::Controller.new(
+      :name      => 'opener-<component>',
+      :exec_path => File.expand_path('../../exec/<component>.rb', __FILE__)
+    )
 
-    $ bundle
+    controller.run
 
-Or install it yourself as:
+Replace `<component>` with the name of the component. For example, for the
+language identifier this would result in the following:
 
-    $ gem install opener-daemons
+    #!/usr/bin/env ruby
+    require 'opener/daemons'
 
+    controller = Opener::Daemons::Controller.new(
+      :name      => 'opener-language-identifier',
+      :exec_path => File.expand_path('../../exec/language-identifier.rb', __FILE__)
+    )
 
-## SQS
+    controller.run
 
-The Opener-daemon GEM uses Amazon SQS service as a message service. In order for
-this to work properly you need an Amazon AWS account and you need to set the
-following 3 environment variables in your shell:
+Next, create an executable file `exec/<component>.rb`, for example
+`exec/language-identifier.rb`, with the following content:
 
-```
-export AWS_ACCESS_KEY_ID='...'
-export AWS_SECRET_ACCESS_KEY='...'
-export AWS_REGION='...' #e.g. eu-west-1
-```
+    #!/usr/bin/env ruby
+    require 'opener/daemons'
 
-To see how you specify which queues to use, checkout the usage section below.
+    require_relative '../lib/opener/<component>'
 
-## Implementation
+    daemon = Opener::Daemons::Daemon.new(Opener::<constant>)
 
-This Gem is intended for use with other OpeNER components. In order to turn a
-component in to a Daemon you have to do the following:
+    daemon.start
 
-Add the opener-daemons gem to the gemspec of your component (or the the Gemfile
-if your component is not a gem).
+Replace `<component>` with the component name, replace `<constant>` with the
+corresponding constant. For example, for the language identifier:
 
-```
-  gem.add_dependency 'opener-daemons'
-```
 
-Create a file in the bin/ directory of the component. Following the OpeNER
-naming conventions that will be something like this (e.g. the
-language-identifier). This file provides you with the option to launch a daemon
-from the command line.
+    #!/usr/bin/env ruby
+    require 'opener/daemons'
 
-    touch bin/language-identifier-daemon
+    require_relative '../lib/opener/language_identifier'
 
-Then add the following lines and replace the language-identifier with your own
-component:
+    daemon = Opener::Daemons::Daemon.new(Opener::LanguageIdentifier)
 
-```ruby
-#!/usr/bin/env ruby
-require 'rubygems'
-require 'opener/daemons'
+    daemon.start
 
-exec_path = File.expand_path("exec/language-identifier.rb")
-Opener::Daemons::Controller.new(:name=>"language-identifier",
-                                :exec_path=>exec_path)
-```
+If the component takes extra arguments, such as a resource path, these should be
+set in the `initialize` method of the component using the actual environment
+variables.
 
-After that you have to create a file that does the actual work in an "exec"
-directory. From the root of your component do this:
+## Requirements
 
-```
-mkdir exec
-touch exec/language-identifier.rb
-```
+* A supported Ruby version (see below)
+* Amazon SQS
+* Amazon S3
 
-Then copy paste the following code into that file, replacing the
-"language-identifier" parts with your own component.
+The following Ruby versions are supported:
 
-```ruby
-require 'opener/daemons'
-require 'opener/language_identifier'
+| Ruby     | Required      | Recommended |
+|:---------|:--------------|:------------|
+| MRI      | >= 1.9.3      | >= 2.1.4    |
+| Rubinius | >= 2.2        | >= 2.3.0    |
+| JRuby    | >= 1.7        | >= 1.7.16   |
 
-options = Opener::OptParser.parse!(ARGV)
-daemon = Opener::Daemon.new(Opener::LanguageIdentifier, options)
-daemon.start
-```
+## Installation
 
-Now you should be able to launch yourself a LanguageIdentifier daemon. Check out
-the exact usage of the daemon by typing:
+Install it from RubyGems:
 
-```
-bin/language-identifier-daemon -h
-```
+    gem install opener-daemons
 
-## Usage
+Or using Bundler:
+
+    # add this to your Gemfile
+    gem 'opener-daemons'
+
+    # then run this
+    bundle install
+
+## Job Format
+
+Jobs should be serialized as JSON and should adhere to the JSON schema
+definition [schema/sqs_input.json](schema/sqs_input.json). In short, a job is a
+JSON object with the following fields:
+
+* `input_url`: the input URL
+* `callbacks`: an array of URLs
+* `identifier`: a unique identifier to use for the file stored in S3, if no
+  value is given an identifier will be generated automatically
+* `metadata`: an object containing arbitrary metadata, will be passed to every
+  callback URL
+
+An example:
+
+    {
+        "input_url": "http://example.com/my-kaf.xml",
+        "callbacks": ["http://example.com/my-callback"],
+        "identifier": "foo123",
+        "metadata": {
+            "customer_id": 123
+        }
+    }
+
+For more specific details see the schema.
+
+## Output
+
+Daemon output is stored in an Amazon S3 bucket, output files are named
+`<identifier>.xml` where `<identifier>` is the unique identifier of the
+document. The content type of these documents is set to `application/xml`.
+Metadata associated with the job (as specified in the `metadata` field) is saved
+as metadata of the S3 object.
+
+Callback URLs will receive the URL of an uploaded document, _not_ the actual
+content itself. The S3 URLs are only valid for a limited time (currently 1 hour)
+so callbacks must ensure they can process the input within that time limit.
+
+## Monitoring
+
+Components using this Gem can measure performance using New Relic and report
+errors using Rollbar. To support this the following two environment variables
+must be set:
+
+* `NEWRELIC_TOKEN`
+* `ROLLBAR_TOKEN`
+
+For New Relic the application names will be `opener-<component>` where
+`<component>` is the component name, as defined by a component itself. If one of
+these environment variables is not set the corresponding feature is disabled.
+
+## CLI Options
+
+Each daemon takes a set of options that can be used to configure the input
+queue, the S3 bucket and so forth. For an up to date list of these options and
+their descriptions run a daemon using the `--help` option.
+
+Some of these options set environment variables that can be used by components,
+these are as following:
+
+* `input`: sets the input queue in the `INPUT_QUEUE` variable
+* `threads`: sets the amount of threads to use in the `DAEMON_THREADS` variable
+* `bucket`: sets the S3 bucket to use for output documents in the
+  `OUTPUT_BUCKET` variable
+
+## Amazon Environment Variables
+
+To properly configure the daemons for Amazon you should set the following
+environment variables:
+
+* `AWS_ACCESS_KEY_ID`
+* `AWS_SECRET_ACCESS_KEY`
+* `AWS_REGION`
 
-Once you implemented the daemon you can use the -h option to get usage
-information. It will look like this, with the "language-identifier" strings
-replaced by your own component.
-
-```
-Usage: language-identifier.rb <start|stop|restart> [options]
-
-Specific options:
-    -i, --input INPUT_QUEUE_NAME     Input queue name
-    -o, --output OUTPUT_QUEUE_NAME   Output queue name
-    -b, --batch-size BATCH_SIZE      Request x messages at once where x is between 1 and 10
-    -w, --workers NUMBER             number of worker thread
-    -r, --readers NUMBER             number of reader threads
-    -p, --writers NUMBER             number of writer / pusher threads
-        --log FILENAME               Filename and path of logfile. Defaults to STDOUT
-        --pid FILENAME               Filename and path of pidfile. Defaults to /var/run/{filename}.rb
-        --pidpath DIRNAME            Directory where to put the PID file. Is Overwritten by --pid if that option is present
-        --debug                      Turn on debug log level
-
-Common options:
-    -h, --help                       Show this message
-```
-
-
-## Contributing
-
-1. Fork it ( http://github.com/opener-project/opener-daemons/fork )
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+If you're running this daemon on an EC2 instance then the first two environment
+variables will be set automatically if the instance has an associated IAM
+profile. The `AWS_REGION` variable must _always_ be set.

data/lib/opener/daemons/configuration.rb

@@ -0,0 +1,52 @@
+module Opener
+  module Daemons
+    ##
+    # Configuration object for storing details about a single job.
+    #
+    # @!attribute [r] component
+    #  @return [Class]
+    #
+    # @!attribute [r] input_url
+    #  @return [String]
+    #
+    # @!attribute [r] callbacks
+    #  @return [Array]
+    #
+    # @!attribute [r] metadata
+    #  @return [Hash]
+    #
+    class Configuration
+      attr_reader :component, :input_url, :callbacks, :metadata
+
+      ##
+      # @param [Class] component
+      # @param [Hash] options
+      #
+      # @option options [String] :input_url
+      # @option options [String] :identifier
+      # @option options [Array] :callbacks
+      # @option options [Hash] :metadata
+      #
+      def initialize(component, options = {})
+        @component = component
+
+        options.each do |key, value|
+          instance_variable_set("@#{key}", value) if respond_to?(key)
+        end
+
+        @callbacks ||= []
+        @metadata  ||= {}
+      end
+
+      ##
+      # Returns the identifier of the document. If no identifier was given a
+      # unique one will be generated instead.
+      #
+      # @return [String]
+      #
+      def identifier
+        return @identifier ||= SecureRandom.hex
+      end
+    end # Configuration
+  end # Daemons
+end # Opener

data/lib/opener/daemons/controller.rb

@@ -1,166 +1,160 @@
-#
-# Original Idea by Charles Nutter
-# Copied from: https://gist.github.com/ik5/448884
-# Then adjusted.
-#
-
-require 'rubygems'
-require 'opener/daemons'
-require 'spoon'
-
 module Opener
   module Daemons
+    ##
+    # CLI controller for a component.
+    #
+    # @!attribute [r] name
+    #  The name of the daemon.
+    #  @return [String]
+    #
+    # @!attribute [r] exec_path
+    #  The path to the script to daemonize.
+    #  @return [String]
+    #
     class Controller
       attr_reader :name, :exec_path
 
-      def initialize(options={})
+      ##
+      # @param [Hash] options
+      #
+      # @option options [String] :name
+      # @option options [String] :exec_path
+      #
+      def initialize(options = {})
+        @name      = options.fetch(:name)
         @exec_path = options.fetch(:exec_path)
-        @name = determine_name(options[:name])
-        read_commandline
       end
 
-      def determine_name(name)
-        return identify(name) unless name.nil?
-        get_name_from_exec_path
-      end
+      ##
+      # Runs the CLI
+      #
+      # @param [Array] argv CLI arguments to parse.
+      #
+      def run(argv = ARGV)
+        slop = configure_slop
 
-      def get_name_from_exec_path
-        File.basename(exec_path, ".rb")
+        slop.parse(argv)
       end
 
-      def read_commandline
-        if ARGV[0] == 'start'
-          start
-        elsif ARGV[0] == 'stop'
-          stop
-        elsif ARGV[0] == 'restart'
-          stop
-          start
-        else
-          start_foreground
+      ##
+      # @return [Slop]
+      #
+      def configure_slop
+        parser = OptionParser.new(name)
+
+        parser.parser.run do |opts, args|
+          command  = args.shift
+          new_args = args.reject { |arg| arg == '--' }
+
+          case command
+          when 'start'
+            start_background(opts, new_args)
+          when 'stop'
+            stop(opts)
+          when 'restart'
+            stop(opts)
+            start_background(opts, new_args)
+          else
+            start_foreground(opts, new_args)
+          end
         end
-      end
 
-      def options
-        return @options if @options
-        args = ARGV.dup
-        @options = Opener::Daemons::OptParser.pre_parse!(args)
+        return parser
       end
 
-
-      def pid_path
-        return @pid_path unless @pid_path.nil?
-        @pid_path = if options[:pid]
-                      File.expand_path(@options[:pid])
-                    elsif options[:pidpath]
-                      File.expand_path(File.join(@options[:pidpath], "#{name}.pid"))
-                    else
-                      "/var/run/#{File.basename($0, ".rb")}.pid"
-                    end
+      ##
+      # Runs the daemon in the foreground.
+      #
+      # @param [Slop] options
+      # @param [Array] argv
+      #
+      def start_foreground(options, argv = [])
+        exec(setup_env(options), exec_path, *argv)
       end
 
-      def create_pid(pid)
-        begin
-          open(pid_path, 'w') do |f|
-            f.puts pid
-          end
-        rescue => e
-          STDERR.puts "Error: Unable to open #{pid_path} for writing:\n\t" +
-            "(#{e.class}) #{e.message}"
-          exit!
-        end
-      end
+      ##
+      # Starts the daemon in the background.
+      #
+      # @param [Slop] options
+      # @param [Array] argv
+      #
+      def start_background(options, argv = [])
+        pidfile = Pidfile.new(options[:pidfile])
+        pid     = Process.spawn(
+          setup_env(options),
+          exec_path,
+          *argv,
+          :out => :close,
+          :err => :close,
+          :in  => :close
+        )
+
+        pidfile.write(pid)
 
-      def get_pid
-        pid = false
         begin
-          open(pid_path, 'r') do |f|
-            pid = f.readline
-            pid = pid.to_s.gsub(/[^0-9]/,'')
-          end
-        rescue => e
-          STDOUT.puts "Info: Unable to open #{pid_path} for reading while checking for existing PID:\n\t" +
-            "(#{e.class}) #{e.message}"
-        end
+          # Wait until the process has _actually_ started.
+          Timeout.timeout(options[:wait]) { sleep(0.5) until pidfile.alive? }
 
+          puts "Process with Pidfile #{pidfile.read} started"
+        rescue Timeout::Error
+          pidfile.unlink
 
-        if pid
-          return pid.to_i
-        else
-          return nil
+          abort "Failed to start the process after #{options[:wait]} seconds"
         end
       end
 
-      def remove_pidfile
-        begin
-          File.unlink(pid_path)
-        rescue => e
-          STDERR.puts "ERROR: Unable to unlink #{path}:\n\t" +
-            "(#{e.class}) #{e.message}"
-          exit
-        end
-      end
+      ##
+      # Stops the daemon.
+      #
+      # @param [Slop] options
+      #
+      def stop(options)
+        pidfile = Pidfile.new(options[:pidfile])
 
-      def process_exists?
-        begin
-          pid = get_pid
-          return false unless pid
-          Process.kill(0, pid)
-          true
-        rescue Errno::ESRCH, TypeError # "PID is NOT running or is zombied
-          false
-        rescue Errno::EPERM
-          STDERR.puts "No permission to query #{pid}!";
-          false
-        rescue => e
-          STDERR.puts "Error: Unable to determine status for pid: #{pid}.\n\t" +
-            "(#{e.class}) #{e.message}"
-          false
-        end
-      end
+        if pidfile.alive?
+          id = pidfile.read
 
-      def stop
-        begin
-          pid = get_pid
-          STDOUT.puts "Stopping pid: #{pid}"
-          while true do
-            Process.kill("TERM", pid)
-            Process.wait(pid)
-            sleep(0.1)
-          end
-        rescue Errno::ESRCH # no more process to kill
-          remove_pidfile
-          STDOUT.puts 'Stopped the process'
-        rescue => e
-          STDERR.puts "Unable to terminate process: (#{e.class}) #{e.message}"
+          pidfile.terminate
+          pidfile.unlink
+
+          puts "Process with Pidfile #{id.inspect} terminated"
+        else
+          abort 'Process already terminated or you are not allowed to terminate it'
         end
       end
 
-      def start
-        if process_exists?
-          STDERR.puts "Error: The process #{name} already running. Restarting the process"
-          stop
+      ##
+      # Returns a Hash containing the various environment variables to set for
+      # the daemon (on top of the current environment variables).
+      #
+      # @param [Slop] options
+      # @return [Hash]
+      #
+      def setup_env(options)
+        newrelic_config = File.expand_path(
+          '../../../../config/newrelic.yml',
+          __FILE__
+        )
+
+        env = ENV.to_hash.merge(
+          'INPUT_QUEUE'    => options[:input].to_s,
+          'DAEMON_THREADS' => options[:threads].to_s,
+          'OUTPUT_BUCKET'  => options[:bucket].to_s,
+          'NRCONFIG'       => newrelic_config,
+          'APP_ROOT'       => File.expand_path('../../../../', __FILE__),
+          'APP_NAME'       => name
+        )
+
+        if !env['RAILS_ENV'] and env['RACK_ENV']
+          env['RAILS_ENV'] = env['RACK_ENV']
         end
 
-        STDOUT.puts "Starting DAEMON"
-        pid = Spoon.spawnp exec_path, *ARGV
-        STDOUT.puts "Started DAEMON"
-        create_pid(pid)
-        begin
-          Process.setsid
-        rescue Errno::EPERM => e
-          STDERR.puts "Process.setsid not permitted on this platform, not critical. Continuing normal operations.\n\t (#{e.class}) #{e.message}"
+        unless options[:'disable-syslog']
+          env['ENABLE_SYSLOG'] = 'true'
         end
-        File::umask(0)
-      end
-
-      def start_foreground
-        exec [exec_path, ARGV].flatten.join(" ")
-      end
 
-      def identify(string)
-        string.gsub(/\W/,"-").gsub("--","-")
+        return env
       end
-    end
-  end
-end
+    end # Controller
+  end # Daemons
+end # Opener