sqewer 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5d7500a5cfddbad5db77d066880dc9c4db85b505
+  data.tar.gz: ed15e8c5303a3dd9067cfeb52754af09a1b4b3f5
+SHA512:
+  metadata.gz: 391b04bba9100e0360b44407ab8cb81790d550c4d4ebb14921b9ef46498038b5cadb245a1fe5c4d910193c9e6a1c69bbd9184ea5631451f17c0d065ba9081bd9
+  data.tar.gz: cb9d4cde98e3b157d5527f106efce4deb370e0ab7712e113e9189fab2db6b41a94b13ac2c2dce7f2fc411e0de2b539cdb74864bcde462f2f52832ea85ce0d6de

data/.gitlab-ci.yml

@@ -0,0 +1,12 @@
+rake:
+  script:
+  - git submodule update --init
+  - ls -la
+  - gem install bundler
+  - bundle config --global jobs 4
+  - bundle config --global path /cache/gems
+  - bundle config build.nokogiri "--use-system-libraries --with-xml2-include=/usr/include/libxml2"
+  - bundle check || bundle install
+  - bundle exec rake
+  tags:
+  - ruby

data/.yardopts

@@ -0,0 +1 @@
+--markup markdown

data/DETAILS.md

@@ -0,0 +1,180 @@
+A more in-depth explanation of the systems below.
+
+## Job storage
+
+Jobs are (by default) stored in SQS as JSON blobs. A very simple job ticket looks like this:
+
+    {"_job_class": "MyJob", "_job_params": null}
+
+When this ticket is being picked up by the worker, the worker will do the following:
+
+    job = MyJob.new
+    job.run
+
+So the smallest job class has to be instantiatable, and has to respond to the `run` message.
+
+## Jobs with arguments and parameters
+
+Job parameters can be passed as keyword arguments. Properties in the job ticket (encoded as JSON) are
+directly translated to keyword arguments of the job constructor. With a job ticket like this:
+
+    {
+      "_job_class": "MyJob",
+      "_job_params": {"ids": [1,2,3]}
+    }
+
+the worker will instantiate your `MyJob` class with the `ids:` keyword argument:
+
+    job = MyJob.new(ids: [1,2,3])
+    job.run
+
+Note that at this point only arguments that are raw JSON types are supported:
+
+* Hash
+* Array
+* Numeric
+* String
+* nil/false/true
+
+If you need marshalable Ruby types there instead, you might need to implement a custom `Serializer.`
+
+## Jobs spawning dependent jobs
+
+If your `run` method on the job object accepts arguments (has non-zero `arity` ) the `ExecutionContext` will
+be passed to the `run` method.
+
+    job = MyJob.new(ids: [1,2,3])
+    job.run(execution_context)
+
+The execution context has some useful methods:
+
+ * `logger`, for logging the state of the current job. The logger messages will be prefixed with the job's `inspect`.
+ * `submit!` for submitting more jobs to the same queue
+
+A job submitting a subsequent job could look like this:
+
+    class MyJob
+      def run(ctx)
+        ...
+        ctx.submit!(DeferredCleanupJob.new)
+      end
+    end
+
+## Job submission
+
+In general, a job object that needs some arguments for instantiation must return a Hash from it's `to_h` method. The hash must
+include all the keyword arguments needed to instantiate the job when executing. For example:
+
+    class SendMail
+      def initialize(to:, body:)
+        ...
+      end
+      
+      def run()
+        ...
+      end
+      
+      def to_h
+        {to: @to, body: @body}
+      end
+    end
+
+Or if you are using `ks` gem (https://rubygems.org/gems/ks) you could inherit your Job from it:
+
+    class SendMail < Ks.strict(:to, :body)
+      def run
+        ...
+      end
+    end
+
+## Job marshaling
+
+By default, the jobs are converted to JSON and back from JSON using the Sqewer::Serializer object. You can
+override that object if you need to handle job tickets that come from external sources and do not necessarily
+conform to the job serialization format used internally. For example, you can handle S3 bucket notifications:
+
+    class CustomSerializer < Sqewer::Serializer
+      # Overridden so that we can instantiate a custom job
+      # from the AWS notification payload.
+      # Return "nil" and the job will be simply deleted from the queue
+      def unserialize(message_blob)
+        message = JSON.load(message_blob)
+        return if message['Service'] # AWS test
+        return HandleS3Notification.new(message) if message['Records']
+        
+        super # as default
+      end
+    end
+
+Or you can override the serialization method to add some metadata to the job ticket on job submission:
+
+    class CustomSerializer < Sqewer::Serializer
+      def serialize(job_object)
+        json_blob = super
+        parsed = JSON.load(json_blob)
+        parsed['_submitter_host'] = Socket.gethostname
+        JSON.dump(parsed)
+      end
+    end
+
+If you return `nil` from your `unserialize` method the job will not be executed,
+but will just be deleted from the SQS queue.
+
+## Starting and running the worker
+
+The very minimal executable for running jobs would be this:
+
+    #!/usr/bin/env ruby
+    require 'my_applicaion'
+    Sqewer::CLI.run
+
+This will connect to the queue at the URL set in the `SQS_QUEUE_URL` environment variable.
+
+You can also run a worker without signal handling, for example in test
+environments. Note that the worker is asynchronous, it has worker threads
+which do all the operations by themselves.
+
+    worker = Sqewer::Worker.new
+    worker.start
+    # ...and once you are done testing
+    worker.stop
+
+## Configuring the worker
+
+One of the reasons this library exists is that sometimes you need to set up some more
+things than usually assumed to be possible. For example, you might want to have a special
+logging library:
+
+    worker = Sqewer::Worker.new(logger: MyCustomLogger.new)
+
+Or you might want a different job serializer/deserializer (for instance, if you want to handle
+S3 bucket notifications coming into the same queue):
+
+    worker = Sqewer::Worker.new(serializer: CustomSerializer.new)
+
+The `Sqewer::CLI` module that you run from the commandline handler application accepts the
+same options as the `Worker` constructor, so everything stays configurable.
+
+## Execution and serialization wrappers (middleware)
+
+You can wrap job processing in middleware. A full-featured middleware class looks like this:
+
+    class MyWrapper
+      # Surrounds the job instantiation from the string coming from SQS.
+      def around_deserialization(serializer, msg_id, msg_payload)
+        # msg_id is the receipt handle, msg_payload is the message body string
+        yield
+      end
+      
+      # Surrounds the actual job execution
+      def around_execution(job, context)
+        # job is the actual job you will be running, context is the ExecutionContext.
+        yield
+      end
+    end
+
+You need to set up a `MiddlewareStack` and supply it to the `Worker` when instantiating:
+
+    stack = Sqewer::MiddlewareStack.new
+    stack << MyWrapper.new
+    w = Sqewer::Worker.new(middleware_stack: stack)

data/FAQ.md

@@ -0,0 +1,54 @@
+# FAQ
+
+This document tries to answer some questions that may arise when reading or using the library. Hopefully
+this can provide some answers with regards to how things are put together.
+
+## Why no ActiveJob? 
+
+An adapter will be added in the future
+
+## Why separate `new` and `run` methods instead of just `perform`?
+
+Because the job needs access to the execution context of the worker. It turned out that keeping the context
+in global/thread/class variables was somewhat nasty, and jobs needed access to the current execution context
+to enqueue the subsequent jobs, and to get access to loggers (and other context-sensitive objects). Therefore
+it makes more sense to offer Jobs access to the execution context, and to make a Job a command object.
+
+Also, Jobs usually use their parameters in multiple smaller methods down the line. It therefore makes sense
+to save those parameters in instance variables or in struct members.
+
+## Why keyword constructors for jobs?
+
+Because keyword constructors map very nicely to JSON objects and provide some (at least rudimentary) arity safety,
+by checking for missing keywords and by allowing default keyword argument values. Also, we already have some
+products that use those job formats. Some have dozens of classes of jobs, all with those signatures and tests.
+
+## Why no weighted queues?
+
+Because very often when you want to split queues servicing one application it means that you do not have enough
+capacity to serve all of the job _types_ in a timely manner. Then you try to assign priority to separate jobs,
+whereas in fact what you need are jobs that execute _roughly_ at the same speed - so that your workers do not
+stall when clogged with mostly-long jobs. Also, multiple queues introduce more configuration, which, for most
+products using this library, was a very bad idea (more workload for deployment).
+
+## Why so many configurable components?
+
+Because sometimes your requirements differ just-a-little-bit from what is provided, and you have to swap your 
+implementation in instead. One product needs foreign-submitted SQS jobs (S3 notifications). Another product
+needs a custom Logger subclass. Yet another product needs process-based concurrency on top of threads.
+Yet another process needs to manage database connections when running the jobs. Have 3-4 of those, and a
+pretty substantial union of required features will start to emerge. Do not fear - most classes of the library
+have a magic `.default` method which will liberate you from most complexities.
+
+## Why multithreading for workers?
+
+Because it is fast and relatively memory-efficient. Most of the workload we encountered was IO-bound or even
+network-IO bound. In that situation it makes more sense to use threads that switch quickly, instead of burdening
+the operating system with too many processes. An optional feature for one-process-per-job is going to be added
+soon, for tasks that really warrant it (like image manipulation). For now, however, threads are working quite OK.
+
+## Why no Celluloid?
+
+Because I found that a producer-consumer model with a thread pool works quite well, and can be created based on
+the Ruby standard library alone.
+

data/Gemfile

@@ -0,0 +1,18 @@
+source "http://rubygems.org"
+
+gem 'aws-sdk', '~> 2'
+gem 'very_tiny_state_machine', '~> 1'
+gem 'hash_tools'
+gem 'exceptional_fork'
+
+group :development do
+  gem 'ks'
+  gem 'dotenv'
+  gem 'rake'
+  gem "rspec", "~> 3.2.0"
+  gem 'simplecov', :require => false
+#  gem "autospec" -> I would love to have this, but it wants capybara-webkit too :-(
+  gem "rdoc", "~> 3.12"
+  gem "bundler", "~> 1.0"
+  gem "jeweler", "~> 2.0.1"
+end

data/README.md

@@ -0,0 +1,69 @@
+An AWS SQS-based queue processor, for highly distributed job engines.
+
+## The shortest introduction possible
+
+In your environment, set `SQS_QUEUE_URL`. Then, define a job class:
+
+    class MyJob
+      def run
+       File.open('output', 'a') { ... }
+      end
+    end
+
+Then submit the job:
+
+    Sqewer.submit!(MyJob.new)
+
+and to start processing, in your commandline handler:
+
+    #!/usr/bin/env ruby
+    require 'my_applicaion'
+    Sqewer::CLI.run
+
+To add arguments to the job
+
+    class JobWithArgs
+      include Sqewer::SimpleJob
+      attr_accessor :times
+      
+      def run
+        ...
+      end
+    end
+    ...
+    Sqewer.submit!(JobWithArgs.new(times: 20))
+
+Submitting jobs from other jobs (the job will go to the same queue the parent job came from):
+
+    class MyJob
+      def run(worker_context)
+        ...
+        worker_context.submit!(CleanupJob.new)
+      end
+    end
+
+The messages will only be deleted from SQS once the job execution completes without raising an exception.
+
+## Detailed usage instructions
+
+For more detailed usage information, see [DETAILS.md](./DETAILS.md)
+
+## Frequently asked questions (A.K.A. _why is it done this way_)
+
+Please see [FAQ.md](./FAQ.md). This might explain some decisions behind the library in greater detail.
+
+## Contributing to the library
+ 
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
+* Fork the project.
+* Start a feature/bugfix branch.
+* Commit and push until you are happy with your contribution.
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Run your tests against a _real_ SQS queue. You will need your tests to have permissions to create and delete SQS queues.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2016 WeTransfer. See LICENSE.txt for further details.
+

data/Rakefile

@@ -0,0 +1,41 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+require_relative 'lib/sqewer/version'
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://guides.rubygems.org/specification-reference/ for more options
+  gem.version = Sqewer::VERSION
+  gem.name = "sqewer"
+  gem.homepage = "https://gitlab.wetransfer.net/julik/sqewer"
+  gem.license = "MIT"
+  gem.description = %Q{Process jobs from SQS}
+  gem.summary = %Q{A full-featured library for all them worker needs}
+  gem.email = "me@julik.nl"
+  gem.authors = ["Julik Tarkhanov"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+# desc "Code coverage detail"
+# task :simplecov do
+#   ENV['COVERAGE'] = "true"
+#   Rake::Task['spec'].execute
+# end
+
+task :default => :spec

data/example.env

@@ -0,0 +1,6 @@
+# Dotenv is used when running tests for the library. You may choose to use Dotenv
+# for your main production application as well. When you use Sqewer as a library
+# Dotenv is not required, and .env is not going to be force-loaded for you.
+AWS_ACCESS_KEY_ID=secret
+AWS_SECRET_ACCESS_KEY=secret
+AWS_REGION=eu-west-1

data/lib/sqewer.rb

@@ -0,0 +1,11 @@
+# The enclosing module for the library
+module Sqewer
+  Dir.glob(__dir__ + '/**/*.rb').each {|p| require p unless p == __FILE__ }
+  
+  # Shortcut access to Submitter#submit.
+  #
+  # @see {Sqewer::Submitter#submit!}
+  def self.submit!(*jobs, **options)
+    Sqewer::Submitter.default.submit!(*jobs, **options)
+  end
+end

data/lib/sqewer/atomic_counter.rb

@@ -0,0 +1,22 @@
+require 'thread'
+
+# Maintains a thread-safe counter wrapped in a Mutex.
+class Sqewer::AtomicCounter
+  def initialize
+    @m, @v = Mutex.new, 0
+  end
+  
+  # Returns the current value of the counter
+  #
+  # @return [Fixnum] the current value of the counter
+  def to_i
+    @m.synchronize { @v + 0 }
+  end
+  
+  # Increments the counter
+  #
+  # @return [Fixnum] the current value of the counter
+  def increment!
+    @m.synchronize { @v += 1 }
+  end
+end

data/lib/sqewer/cli.rb

@@ -0,0 +1,44 @@
+module Sqewer::CLI
+  # Start the commandline handler, and set up a centralized signal handler that reacts
+  # to USR1 and TERM to do a soft-terminate on the worker.
+  #
+  # @param worker[Sqewer::Worker] the worker to start. Must respond to `#start` and `#stop`
+  # @return [void]
+  def start(worker = Sqewer::Worker.default)
+    # Use a self-pipe to accumulate signals in a central location
+    self_read, self_write = IO.pipe
+    %w(INT TERM USR1 USR2 TTIN).each do |sig|
+      begin
+        trap(sig) { self_write.puts(sig) }
+      rescue ArgumentError
+        # Signal not supported
+      end
+    end
+    
+    begin
+      worker.start
+      # The worker is non-blocking, so in the main CLI process we select() on the signal
+      # pipe and handle the signal in a centralized fashion
+      while (readable_io = IO.select([self_read]))
+        signal = readable_io.first[0].gets.strip
+        handle_signal(worker, signal)
+      end
+    rescue Interrupt
+      worker.stop
+      exit 1
+    end
+  end
+  
+  def handle_signal(worker, sig)
+    case sig
+    when 'USR1', 'TERM'
+      worker.stop
+      exit 0
+    #when 'TTIN' # a good place to print the worker status
+    else
+      raise Interrupt
+    end
+  end
+  
+  extend self
+end