sidejob 3.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: d993f3349b71794286776cd15dd1a08052466080
+  data.tar.gz: db8eb058a8329b4be54604762efaa68b7f053bff
+SHA512:
+  metadata.gz: ab477bb2a9ad4ad67977249c081a235c95c74e44cc4157e884d8570c4b6dde177d8f458142eff70323867706216c45329167441be46875d75676e0801c42e6c3
+  data.tar.gz: 94543337bb043a8c35a11ea0823419572d6bdf8c4d48011cc61d2c28e08ff9d2bb891d18d8e8cb10d430eb21d81d183284c5400d3e5288376aa6926fef2ee364

data/.gitignore

@@ -0,0 +1,5 @@
+.bundle/
+*.gem
+coverage/
+.yardoc/
+doc/

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format progress

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/Gemfile.lock

@@ -0,0 +1,61 @@
+PATH
+  remote: .
+  specs:
+    sidejob (3.0.0)
+      sidekiq (~> 3.2.5)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    celluloid (0.15.2)
+      timers (~> 1.1.0)
+    coderay (1.1.0)
+    connection_pool (2.0.0)
+    diff-lcs (1.2.5)
+    docile (1.1.5)
+    json (1.8.1)
+    method_source (0.8.2)
+    multi_json (1.10.1)
+    pry (0.9.12.6)
+      coderay (~> 1.0)
+      method_source (~> 0.8)
+      slop (~> 3.4)
+    redis (3.1.0)
+    redis-namespace (1.5.1)
+      redis (~> 3.0, >= 3.0.4)
+    rspec (3.1.0)
+      rspec-core (~> 3.1.0)
+      rspec-expectations (~> 3.1.0)
+      rspec-mocks (~> 3.1.0)
+    rspec-core (3.1.7)
+      rspec-support (~> 3.1.0)
+    rspec-expectations (3.1.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.1.0)
+    rspec-mocks (3.1.3)
+      rspec-support (~> 3.1.0)
+    rspec-support (3.1.2)
+    sidekiq (3.2.6)
+      celluloid (= 0.15.2)
+      connection_pool (>= 2.0.0)
+      json
+      redis (>= 3.0.6)
+      redis-namespace (>= 1.3.1)
+    simplecov (0.9.0)
+      docile (~> 1.1.0)
+      multi_json
+      simplecov-html (~> 0.8.0)
+    simplecov-html (0.8.0)
+    slop (3.4.7)
+    timers (1.1.0)
+    yard (0.8.7.4)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  pry
+  rspec
+  sidejob!
+  simplecov
+  yard

data/README.md

@@ -0,0 +1,128 @@
+SideJob
+=======
+
+SideJob is built on top of [Sidekiq](https://github.com/mperham/sidekiq) and
+[Redis](http://redis.io/). Sidekiq jobs typically complete relatively quickly.
+Just like a typical side job, a job in SideJob may depend on other jobs or may make slow progress
+and take a long time (such as months). They may be suspended and restarted many times.
+The job should be robust to the crashing or downtime of any portion of the infrastructure.
+
+Requirements
+------------
+
+Ruby 2.0 or greater and Redis 2.8 or greater is recommended.
+
+Jobs
+----
+
+* Jobs have a unique ID assigned using incrementing numbers
+    * This ID is used as Sidekiq's jid
+    * Note: a job can be queued multiple times on Sidekiq's queues
+    * Therefore, Sidekiq's jids are not unique
+* Jobs have a queue and class name
+* Jobs have any number of input and output ports
+* A job can have any number of named child jobs
+* Each job has at most one parent job
+* Jobs can store any JSON encoded object in its internal state
+
+Jobs can have a number of different status. The statuses and possible status transitions:
+
+* -> queued
+* queued -> running | terminating
+* running -> queued | suspended | completed | failed | terminating
+* suspended | completed | failed -> queued | terminating
+* terminating -> terminated
+* terminated -> queued
+
+The difference between suspended, completed, and failed is only in their implications on the
+internal job state. Completed implies that the job has processed all data and can be naturally
+terminated. If additional input arrives, a completed job could continue running. Suspended implies
+that the job is waiting for some input. Failed means that an exception was thrown.
+
+Jobs that have been terminated along with all their children can be deleted entirely.
+
+Ports
+-----
+
+* Ports are named (case sensitive) and must match `/^[a-zA-Z0-9_]+$/`.
+* Any object that can be JSON encoded can be written or read from any input or output port.
+* Ports must be explicitly specified for each job either by the worker configuration or when queuing new jobs unless
+  a port named `*` exists in which case new ports are dynamically created and inherit its options.
+
+Port options:
+
+* mode
+    * Queue - This is the default operation mode. All data written is read in a first in first out manner.
+    * Memory - No data is stored on the port. The most recent value sets the port default value.
+* default - Default value when a read is done on the port with no data
+
+Workers
+-------
+
+* A worker is the implementation of a specific job class
+* Workers are required to register themselves
+* A Sidekiq process should only handle a single queue so all registered workers in the process are for the same queue
+* It should have a perform method that is called on each run
+* It may have a shutdown method that is called before the job is terminated
+* Workers should be idempotent as they may be run more than once for the same state
+* SideJob ensures only one worker thread runs for a given job at a time
+* Workers are responsible for managing state across runs
+* Workers can suspend themselves when waiting for inputs
+
+Data Structure
+--------------
+
+SideJob uses Redis for all job processing and storage. Code using
+SideJob should use API functions instead of accessing redis directly,
+but this is a description of the current data storage format.
+
+The easiest way to set the redis location is via the environment
+variable SIDEJOB_URL, e.g. redis://redis.myhost.com:6379/4
+
+The keys used by Sidekiq:
+
+* queues - Set containing all queue names
+* queue:<queue> - List containing jobs to be run (new jobs pushed on left) on the given queue
+    * A sidekiq job is encoded as json and contains at minimum: queue, retry, class, jid, args
+* schedule - Sorted set by schedule time of jobs to run in the future
+* retry - Sorted set by retry time of jobs to retry
+* dead - Sorted set by addition time of dead jobs
+* processes - Set of sidekiq processes (values are host:pid)
+* <host>:<pid> - Hash representing a connected sidekiq process
+    * beat - heartbeat timestamp (every 5 seconds)
+    * busy - number of busy workers
+    * info - JSON encoded info with keys hostname, started_at, pid, tag, concurrency, queues, labels. Expiry of 60 seconds.
+* <host>:<pid>:workers - Hash containing running workers (thread ID -> { queue: <queue>, payload: <message>, run_at: <timestamp> })
+* <host>:<pid>-signals - List for remotely sending signals to a sidekiq process (USR1 and TERM), 60 second expiry.
+* stat:processed - Cumulative number of jobs processed
+* stat:failed - Cumulative number of jobs failed
+* stat:processed:<date> - Number of jobs processed for given date
+* stat:failed:<date> - Number of jobs failed for given date
+
+Additional keys used by SideJob:
+
+* workers:<queue> - Hash mapping class name to worker configuration. A worker should define
+  the inports and outports hashes that map port names to port options.
+* jobs:last_id - Stores the last job ID (we use incrementing integers from 1)
+* jobs:logs - List with JSON encoded logs.
+    * { timestamp: <date>, job: <id>, read: [{ job: <id>, <in|out>port: <port>, data: [<data>] }, ...], write: [{ job: <id>, <in|out>port: <port>, data: [<data>] }, ...] }
+    * { timestamp: <date>, job: <id>, error: <message>, backtrace: <exception backtrace> }
+* jobs - Hash mapping active job IDs to JSON encoded job state.
+    * queue - queue name
+    * class - name of class
+    * args - array of arguments passed to worker's perform method
+    * created_at - timestamp that the job was first queued
+    * created_by - string indicating the entity that created the job. SideJob uses job:<id> for jobs created by another job.
+    * ran_at - timestamp of the start of the last run
+    * Any additional keys used by the worker to track internal state
+* job:<id>:status - Job status as string.
+* job:<id>:in:<inport> and job:<id>:out:<outport> - List with unread port data. New data is pushed on the right.
+* job:<id>:inports:mode and job:<id>:outports:mode - Hash mapping port name to port mode. All existing ports must be here.
+* job:<id>:inports:default and job:<id>:outports:default - Hash mapping port name to JSON encoded default value for port.
+* job:<id>:ancestors - List with parent job IDs up to the root job that has no parent.
+    Newer jobs are pushed on the left so the immediate parent is on the left and the root job is on the right.
+* job:<id>:children - Hash mapping child job name to child job ID
+* job:<id>:rate:<timestamp> - Rate limiter used to prevent run away executing of a job.
+    Keys are automatically expired.
+* job:<id>:lock - Used to prevent multiple worker threads from running a job.
+    Auto expired to prevent stale locks.

data/bin/build

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+echo "Building..."
+
+# clean source tree
+git clean -fxd
+
+bundle check --no-color || bundle install --no-color || exit 1
+bundle exec rspec spec || exit 1
+
+bundle exec yard

data/bin/console

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'sidejob'
+
+require 'pry'
+Pry.start

data/lib/sidejob/job.rb

@@ -0,0 +1,425 @@
+module SideJob
+  # Methods shared between {SideJob::Job} and {SideJob::Worker}.
+  module JobMethods
+    attr_reader :id
+    attr_accessor :logger
+
+    # @return [Boolean] True if two jobs or workers have the same id
+    def ==(other)
+      other.respond_to?(:id) && id == other.id
+    end
+
+    # @see #==
+    def eql?(other)
+      self == other
+    end
+
+    # @return [Fixnum] Hash value based on the id
+    def hash
+      id.hash
+    end
+
+    # @return [String] Prefix for all redis keys related to this job
+    def redis_key
+      "job:#{id}"
+    end
+    alias :to_s :redis_key
+
+    # Returns if the job still exists.
+    # @return [Boolean] Returns true if this job exists and has not been deleted
+    def exists?
+      SideJob.redis.hexists 'jobs', id
+    end
+
+    # If a job logger is defined, call the log method on it with the log entry. Otherwise, call {SideJob.log}.
+    # @param entry [Hash] Log entry
+    def log(entry)
+      entry[:job] = id unless entry[:job]
+      (@logger || SideJob).log(entry)
+    end
+
+    # Groups all port reads and writes within the block into a single logged event.
+    # @param metadata [Hash] If provided, the metadata is merged into the final log entry
+    def group_port_logs(metadata={}, &block)
+      new_group = @logger.nil?
+      @logger ||= GroupPortLogs.new(self)
+      @logger.add_metadata metadata
+      yield
+    ensure
+      if new_group
+        @logger.done
+        @logger = nil
+      end
+    end
+
+    # Retrieve the job's status.
+    # @return [String] Job status
+    def status
+      SideJob.redis.get "#{redis_key}:status"
+    end
+
+    # Set the job status.
+    # @param status [String] The new job status
+    def status=(status)
+      SideJob.redis.set "#{redis_key}:status", status
+    end
+
+    # Prepare to terminate the job. Sets status to 'terminating'.
+    # Then queues the job so that its shutdown method if it exists can be run.
+    # After shutdown, the status will be 'terminated'.
+    # If the job is currently running, it will finish running first.
+    # If the job is already terminated, it does nothing.
+    # To start the job after termination, call {#run} with force: true.
+    # @param recursive [Boolean] If true, recursively terminate all children (default false)
+    # @return [SideJob::Job] self
+    def terminate(recursive: false)
+      if status != 'terminated'
+        self.status = 'terminating'
+        sidekiq_queue
+      end
+      if recursive
+        children.each_value do |child|
+          child.terminate(recursive: true)
+        end
+      end
+      self
+    end
+
+    # Run the job.
+    # This method ensures that the job runs at least once from the beginning.
+    # If the job is currently running, it will run again.
+    # Just like sidekiq, we make no guarantees that the job will not be run more than once.
+    # Unless force is set, if the status is terminating or terminated, the job will not be run.
+    # @param force [Boolean] Whether to run if job is terminated (default false)
+    # @param at [Time, Float] Time to schedule the job, otherwise queue immediately
+    # @param wait [Float] Run in the specified number of seconds
+    # @return [SideJob::Job] self
+    def run(force: false, at: nil, wait: nil)
+      check_exists
+
+      case status
+        when 'terminating', 'terminated'
+          return unless force
+      end
+
+      self.status = 'queued'
+
+      time = nil
+      if at
+        time = at
+        time = time.to_f if time.is_a?(Time)
+      elsif wait
+        time = Time.now.to_f + wait
+      end
+      sidekiq_queue(time)
+
+      self
+    end
+
+    # Returns a child job by name.
+    # @param name [Symbol, String] Child job name to look up
+    # @return [SideJob::Job, nil] Child job or nil if not found
+    def child(name)
+      SideJob.find(SideJob.redis.hget("#{redis_key}:children", name))
+    end
+
+    # Returns all children jobs.
+    # @return [Hash<String => SideJob::Job>] Children jobs by name
+    def children
+      SideJob.redis.hgetall("#{redis_key}:children").each_with_object({}) {|child, hash| hash[child[0]] = SideJob.find(child[1])}
+    end
+
+    # Returns all ancestor jobs.
+    # @return [Array<SideJob::Job>] Ancestors (parent will be first and root job will be last)
+    def ancestors
+      SideJob.redis.lrange("#{redis_key}:ancestors", 0, -1).map { |id| SideJob.find(id) }
+    end
+
+    # Returns the parent job.
+    # @return [SideJob::Job, nil] Parent job or nil if none
+    def parent
+      parent = SideJob.redis.lindex("#{redis_key}:ancestors", 0)
+      parent = SideJob.find(parent) if parent
+      parent
+    end
+
+    # Returns if job and all children are terminated.
+    # @return [Boolean] True if this job and all children recursively are terminated
+    def terminated?
+      return false if status != 'terminated'
+      children.each_value do |child|
+        return false unless child.terminated?
+      end
+      return true
+    end
+
+    # Deletes the job and all children jobs (recursively) if all are terminated.
+    # @return [Boolean] Whether the job was deleted
+    def delete
+      return false unless terminated?
+
+      # recursively delete all children first
+      children.each_value do |child|
+        child.delete
+      end
+
+      # delete all SideJob keys
+      ports = inports.map(&:redis_key) + outports.map(&:redis_key)
+      SideJob.redis.multi do |multi|
+        multi.hdel 'jobs', id
+        multi.del ports + %w{status children ancestors inports:mode outports:mode inports:default outports:default}.map {|x| "#{redis_key}:#{x}" }
+      end
+      reload
+      return true
+    end
+
+    # Returns an input port.
+    # @param name [Symbol,String] Name of the port
+    # @return [SideJob::Port]
+    # @raise [RuntimeError] Error raised if port does not exist
+    def input(name)
+      get_port :in, name
+    end
+
+    # Returns an output port
+    # @param name [Symbol,String] Name of the port
+    # @return [SideJob::Port]
+    # @raise [RuntimeError] Error raised if port does not exist
+    def output(name)
+      get_port :out, name
+    end
+
+    # Gets all input ports.
+    # @return [Array<SideJob::Port>] Input ports
+    def inports
+      load_ports if ! @ports
+      @ports[:in].values
+    end
+
+    # Sets the input ports for the job.
+    # The ports are merged with the worker configuration.
+    # Any current ports that are not in the new port set are deleted (including any data on those ports).
+    # @param ports [Hash{Symbol,String => Hash}] Input port configuration. Port name to options.
+    def inports=(ports)
+      set_ports :in, ports
+    end
+
+    # Gets all output ports.
+    # @return [Array<SideJob::Port>] Output ports
+    def outports
+      load_ports if ! @ports
+      @ports[:out].values
+    end
+
+    # Sets the input ports for the job.
+    # The ports are merged with the worker configuration.
+    # Any current ports that are not in the new port set are deleted (including any data on those ports).
+    # @param ports [Hash{Symbol,String => Hash}] Output port configuration. Port name to options.
+    def outports=(ports)
+      set_ports :out, ports
+    end
+
+    # Returns some data from the job's state.
+    # The job state is cached for the lifetime of the job object. Call {#reload} if the state may have changed.
+    # @param key [Symbol,String] Retrieve value for the given key
+    # @return [Object,nil] Value from the job state or nil if key does not exist
+    # @raise [RuntimeError] Error raised if job no longer exists
+    def get(key)
+      load_state
+      @state[key.to_s]
+    end
+
+    # Clears the state and ports cache.
+    def reload
+      @state = nil
+      @ports = nil
+      @config = nil
+    end
+
+    # Returns the worker configuration for the job.
+    # @see SideJob::Worker.config
+    def config
+      @config ||= SideJob::Worker.config(get(:queue), get(:class))
+    end
+
+    private
+
+    # Queue or schedule this job using sidekiq.
+    # @param time [Time, Float, nil] Time to schedule the job if specified
+    def sidekiq_queue(time=nil)
+      queue = get(:queue)
+      klass = get(:class)
+      args = get(:args)
+
+      if ! SideJob.redis.hexists("workers:#{queue}", klass)
+        self.status = 'terminated'
+        raise "Worker no longer registered for #{klass} in queue #{queue}"
+      end
+      item = {'jid' => id, 'queue' => queue, 'class' => klass, 'args' => args || [], 'retry' => false}
+      item['at'] = time if time && time > Time.now.to_f
+      Sidekiq::Client.push(item)
+    end
+
+    # Caches all inports and outports.
+    def load_ports
+      @ports = {}
+      %i{in out}.each do |type|
+        @ports[type] = {}
+        SideJob.redis.hkeys("#{redis_key}:#{type}ports:mode").each do |name|
+          if name == '*'
+            @ports["#{type}*"] = SideJob::Port.new(self, type, name)
+          else
+            @ports[type][name] = SideJob::Port.new(self, type, name)
+          end
+        end
+      end
+    end
+
+    # Returns an input or output port.
+    # @param type [:in, :out] Input or output port
+    # @param name [Symbol,String] Name of the port
+    # @return [SideJob::Port]
+    def get_port(type, name)
+      load_ports if ! @ports
+      name = name.to_s
+      return @ports[type][name] if @ports[type][name]
+
+      if @ports["#{type}*"]
+        # create port with default port options for dynamic ports
+        port = SideJob::Port.new(self, type, name)
+        port.options = @ports["#{type}*"].options
+        @ports[type][name] = port
+        return port
+      else
+        raise "Unknown #{type}put port: #{name}"
+      end
+    end
+
+    # Sets the input/outputs ports for the job and overwrites all current options.
+    # The ports are merged with the worker configuration.
+    # Any current ports that are not in the new port set are deleted (including any data on those ports).
+    # @param type [:in, :out] Input or output ports
+    # @param ports [Hash{Symbol,String => Hash}] Port configuration. Port name to options.
+    def set_ports(type, ports)
+      current = SideJob.redis.hkeys("#{redis_key}:#{type}ports:mode") || []
+
+      replace_port_data = []
+      ports = (ports || {}).stringify_keys
+      ports = (config["#{type}ports"] || {}).merge(ports)
+      ports.each_key do |port|
+        ports[port] = ports[port].stringify_keys
+        replace_port_data << port if ports[port]['data']
+      end
+
+      SideJob.redis.multi do |multi|
+        # remove data from old ports
+        ((current - ports.keys) | replace_port_data).each do |port|
+          multi.del "#{redis_key}:#{type}:#{port}"
+        end
+
+        # completely replace the mode and default keys
+
+        multi.del "#{redis_key}:#{type}ports:mode"
+        modes = ports.map do |port, options|
+          [port, options['mode'] || 'queue']
+        end.flatten(1)
+        multi.hmset "#{redis_key}:#{type}ports:mode", *modes if modes.length > 0
+
+        defaults = ports.map do |port, options|
+          if options.has_key?('default')
+            [port, options['default'].to_json]
+          else
+            nil
+          end
+        end.compact.flatten(1)
+        multi.del "#{redis_key}:#{type}ports:default"
+        multi.hmset "#{redis_key}:#{type}ports:default", *defaults if defaults.length > 0
+      end
+
+      @ports = nil
+
+      group_port_logs do
+        ports.each_pair do |port, options|
+          if options['data']
+            port = get_port(type, port)
+            options['data'].each do |x|
+              port.write x
+            end
+          end
+        end
+      end
+    end
+
+    # @raise [RuntimeError] Error raised if job no longer exists
+    def check_exists
+      raise "Job #{id} no longer exists!" unless exists?
+    end
+
+    def load_state
+      if ! @state
+        state = SideJob.redis.hget('jobs', id)
+        raise "Job #{id} no longer exists!" if ! state
+        @state = JSON.parse(state)
+      end
+      @state
+    end
+  end
+
+  # Wrapper for a job which may not be in progress unlike SideJob::Worker.
+  # @see SideJob::JobMethods
+  class Job
+    include JobMethods
+
+    # @param id [String] Job id
+    def initialize(id)
+      @id = id
+    end
+  end
+
+  # Logger that groups all port read/writes together.
+  # @see {JobMethods#group_port_logs}
+  class GroupPortLogs
+    def initialize(job)
+      @metadata = {job: job.id}
+    end
+
+    # If entry is not a port log, send it on to {SideJob.log}. Otherwise, collect the log until {#done} is called.
+    # @param entry [Hash] Log entry
+    def log(entry)
+      if entry[:read] && entry[:write]
+        # collect reads and writes by port and group data together
+        @port_events ||= {read: {}, write: {}} # {job: id, <in|out>port: port} -> data array
+        %i{read write}.each do |type|
+          entry[type].each do |event|
+            data = event.delete(:data)
+            @port_events[type][event] ||= []
+            @port_events[type][event].concat data
+          end
+        end
+      else
+        SideJob.log(entry)
+      end
+    end
+
+    # Merges the collected port read and writes and send logs to {SideJob.log}.
+    def done
+      return unless @port_events && (@port_events[:read].length > 0 || @port_events[:write].length > 0)
+
+      entry = {}
+      %i{read write}.each do |type|
+        entry[type] = @port_events[type].map do |port, data|
+          port.merge({data: data})
+        end
+      end
+
+      SideJob.log @metadata.merge(entry)
+      @port_events = nil
+    end
+
+    # Add metadata fields to the final log entry.
+    # @param metadata [Hash] Data to be merged with the existing metadata and final log entry
+    def add_metadata(metadata)
+      @metadata.merge!(metadata.symbolize_keys)
+    end
+  end
+end