workerholic 0.0.21 → 0.0.22

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2c83d9577835bf37fe458f1eec211b3b3290beef
-  data.tar.gz: 2cb91be6cad32404d1652a0912075c472c420815
+  metadata.gz: fdbe7b74f8270c226d2601c566f6a0417a2bc1b5
+  data.tar.gz: b9e159ae9b745c367f79ac92b11711e8650946a6
 SHA512:
-  metadata.gz: a6876d6abb09a6fc454233dcadcac636f8153f451300c9645be508018defa9b28384cff95abd2b6127c9d5a48d7da777832eedd9bb8e85da4fc9a38e59c830b0
-  data.tar.gz: ddf1db604ee58382c66a6221c56b49d853feac73aff35214e911e7526b2d1d5a72176ca2a2992acd80056fb730639aaa2eafeef53d846c5bee20a74915135fd3
+  metadata.gz: 2f500485b1718548a16660432deb738ccd9eaa3d8ce29dc407c839d3c4201f8ac13babbf3f7c5d2c7158fbd6880d60a50ab0e19cbd29b058e3e8f210dcc07a8a
+  data.tar.gz: 14360fab31e56ee4f37d4a5db775f7ad8bdfdb47f26e55ff09aa6549618df5207506021878afcf7a9b68254114879fd1e82193db488f055fc727ec07dd2785be

data/README.md

@@ -1,2 +1,226 @@
-# workerholic
-A Background Job Processing Manager
+# Workerholic: background job processor
+
+## Summary
+
+Workerholic is a multi-threaded, multi-process background job processing manager. It is an experimental project and as such is not meant to replace other stable engines like Sidekiq or Resque. In fact, Workerholic is inspired in large part by these projects.
+
+In general, background job managers are great for asynchronous processing of time-consuming tasks like calling third party APIs and performing long calculations. What we are building aims to cover all those use cases.
+
+## Features
+### Overview
+#### Here's a brief overview of Workerholic's data flow:
+- enqueue a job
+- job is serialized and added to queue in Redis
+- workers pick jobs from Redis queues, deserialize and process them
+- on completion (or failure), relevant statistics are added to Redis
+- web-ui monitors application metrics and outputs the result
+
+### Job retry
+  Workerholic will retry every unsuccessfully performed job up to 5 times before placing it into a failed jobs queue
+
+### Multiple queues
+  It is possible to specify your own queues for each job your application needs to perform. This way every job has its own namespace and can be easily distinguished between multiple jobs.
+
+### Job scheduler
+  You can schedule a job to be performed at certain time
+
+### Job persistence in Redis
+  Every job (both active and completed) is stored in a Redis database. This way you will not lose any jobs even if your application crashes
+
+### Graceful shutdown
+  Workerholic will finish all currently processing jobs before shutting down
+
+### Web UI with statistics
+  Detailed statistics for processed jobs, queues sizes, overall performance and memory usage
+
+### Workers provisioning
+  Each worker is treated as a separate entity with its state changing dynamically based on the current number of jobs to perform
+
+### Auto-balancing
+  By default, each job queue will get a fair number of workers assigned to it. With provided optional argument, Workerholic can dynamically reassign workers to different queues depending on the number of jobs in each queue
+
+### Multi-process execution
+  Workerholic can be executed in multiple processes to utilize different CPU cores (MRI)
+
+## Installation
+### Installing the gem
+Install the gem with the following command:
+
+    $ gem install workerholic
+
+Or, add the following line to your application's Gemfile:
+
+    gem 'workerholic', '~> 0.1'
+
+And then make sure to execute the following command:
+
+    $ bundle install
+
+### Starting Redis
+In order to start Redis, execute the following:
+
+    redis-server
+
+By default, Workerholic will try to connect to Redis via `localhost:6379`.
+
+For a production environment, make sure to set a `REDIS_URL` environment variable to the address of your redis server.
+
+## Usage
+### Including Workerholic
+
+In order to perform your jobs asynchronously you will need to include Workerholic in your job classes:
+
+```ruby
+class MyJob
+  include Workerholic::Job
+
+  def perform(args)
+    # job logic goes here
+  end
+end
+```
+
+It is important to follow this pattern:
+- include `Workerholic::Job` in your job class
+- define a `perform` method holding your job's logic
+
+### Job Options
+#### Specifying a Queue
+
+Workerholic allows you to specify a name for the queue that your job will be enqueued in:
+
+```ruby
+class MyJob
+  include Workerholic::Job
+
+  job_options queue_name: 'my_queue'
+
+  def perform(args)
+    # job logic goes here
+  end
+end
+```
+
+### Performing a Job Asynchronously
+
+You can perform a job asynchronously:
+
+```ruby
+my_job = MyJob.new
+my_job.perform_async(arg1, arg2)
+```
+
+This will ensure that your job is performed in the background, asynchronously, as soon as possible.
+
+### Scheduling the Job
+
+You can schedule a job to be executed at a later time:
+
+```ruby
+my_job = MyJob.new
+my_job.peform_delayed(100, arg1, arg2)
+```
+
+The above example ensures that `my_job` will be performed in 100 seconds.
+
+## Configuration
+### Loading Application's Files
+#### For a Rails Application
+
+When using Workerholic with a Rails application, as long as you make sure to start Workerholic from the root directory of your Rails application, it will automatically detect your Rails application and eager load your application's files.
+
+#### For Another Application
+
+For Workerholic to execute the jobs you enqueue properly it needs to have access to the job classes.
+
+Make sure to require all classes/dependencies needed with a single file:
+
+    workerholic -r my_app/all_dependencies_needed.rb
+
+### Setting Number of Workers
+
+When starting Workerholic you can specify the number of workers you want running and performing jobs:
+
+    workerholic -w 25
+
+### Setting Number of Redis Connections
+
+Internally, Workerholic uses a connection pool. By default, Workerholic will create a connection pool with a number of `workers_count + 5` Redis connections.
+
+In a production environment, you might be limited by the number of concurrent connections to Redis you are allowed to have. Make sure to check what the limit is and you can then start Workerholic by specifying a number of connections:
+
+    workerholic -c 10
+
+### Setting Number of Processes to Boot
+
+Workerholic allows you to start multiple processes in parallel by forking children processes from the main process. This can be achieved with the following option:
+
+    workerholic -p 3
+
+This will allow you to run 3 processes in parallel, with each process having its own workers and connection pool to Redis.
+
+### Enabling Workers Auto-Balancing Option
+
+By default, Workerholic will evenly provision workers between job queues.
+Use the following option if you want to have workers provisioned based on the load for each queue:
+
+    workerholic -a
+
+This will ensure that each queue will be provisioned with a number of workers based on its relative load compared to the aggregated load for all job queues.
+
+## Integration
+### ActiveJob
+
+Workerholic integrates with ActiveJob. Add the following line to your `application.rb` file:
+
+```ruby
+class Application < Rails::Application
+  # ...
+
+  config.active_job.queue_adapter = :workerholic
+
+  # ...
+end
+```
+
+After that line is added, you can use `ActiveJob`'s API to execute your jobs asynchronously:
+
+```ruby
+class MyJob < ApplicationJob
+  queue_as: 'my_queue'
+
+  def perform(args)
+    # job logic goes here
+  end
+end
+
+MyJob.perform_later(args)
+```
+
+### Web UI
+
+Workerholic comes with a Web UI tracking various statistics about the jobs that are being performed.
+
+#### For a Rails Application
+
+For a Rails application you will need to mount the Web UI on a specific route. In your `routes.rb` file make sure to add the following:
+
+```ruby
+require 'workerholic/web/application'
+
+Rails.application.routes.draw do
+  # ...
+
+  mount WorkerholicWeb => '/workerholic'
+
+  # ...
+end
+```
+
+#### For Another Application
+
+If you are using another kind of application, you can start the Web UI using the following command:
+
+    workerholic-web
+
+You can then view the the app at: `localhost:4567`

data/app_test/job_test.rb

@@ -1,4 +1,4 @@
-$LOAD_PATH.unshift(__dir__ + '/../lib')
+# $LOAD_PATH.unshift(__dir__ + '/../lib')
 require 'workerholic'
 require 'prime'
 

data/app_test/run.rb

@@ -92,4 +92,4 @@ end
 
 TestRunner.non_blocking(10000)
 TestRunner.blocking(10000)
-TestRunner.fibonacci_cruncher(10000)
+TestRunner.fibonacci_cruncher(10000)

data/bin/workerholic

@@ -1,6 +1,6 @@
 #!/usr/bin/env ruby
 
-#require 'workerholic/cli'
+# require 'workerholic/cli'
 require_relative '../lib/workerholic/cli'
 
 Workerholic::CLI.instance.run

data/demo_app/.gitignore

@@ -0,0 +1,19 @@
+# See https://help.github.com/articles/ignoring-files for more about ignoring files.
+#
+# If you find yourself ignoring temporary files generated by your text editor
+# or operating system, you probably want to add a global ignore instead:
+#   git config --global core.excludesfile '~/.gitignore_global'
+
+# Ignore bundler config.
+/.bundle
+
+# Ignore all logfiles and tempfiles.
+/log/*
+/tmp/*
+!/log/.keep
+!/tmp/.keep
+
+/node_modules
+/yarn-error.log
+
+.byebug_history

data/demo_app/Gemfile

@@ -0,0 +1,40 @@
+source 'https://rubygems.org'
+
+git_source(:github) do |repo_name|
+  repo_name = "#{repo_name}/#{repo_name}" unless repo_name.include?("/")
+  "https://github.com/#{repo_name}.git"
+end
+
+
+gem 'rails', '~> 5.1.3'
+gem 'pg', '~> 0.18'
+gem 'puma', '~> 3.7'
+gem 'sass-rails', '~> 5.0'
+gem 'uglifier', '>= 1.3.0'
+gem 'coffee-rails', '~> 4.2'
+#gem 'turbolinks', '~> 5'
+gem 'jbuilder', '~> 2.5'
+gem 'slim-rails'
+gem 'http'
+gem 'workerholic'
+
+group :development, :test do
+  # Call 'byebug' anywhere in the code to stop execution and get a debugger console
+  gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
+  # Adds support for Capybara system testing and selenium driver
+  gem 'capybara', '~> 2.13'
+  gem 'selenium-webdriver'
+end
+
+group :development do
+  # Access an IRB console on exception pages or by using <%= console %> anywhere in the code.
+  gem 'web-console', '>= 3.3.0'
+  gem 'pry-byebug'
+  gem 'listen', '>= 3.0.5', '< 3.2'
+  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
+#  gem 'spring'
+#  gem 'spring-watcher-listen', '~> 2.0.0'
+end
+
+# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
+gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

data/demo_app/README.md

@@ -0,0 +1,24 @@
+# README
+
+This README would normally document whatever steps are necessary to get the
+application up and running.
+
+Things you may want to cover:
+
+* Ruby version
+
+* System dependencies
+
+* Configuration
+
+* Database creation
+
+* Database initialization
+
+* How to run the test suite
+
+* Services (job queues, cache servers, search engines, etc.)
+
+* Deployment instructions
+
+* ...

data/demo_app/Rakefile

@@ -0,0 +1,6 @@
+# Add your own tasks in files placed in lib/tasks ending in .rake,
+# for example lib/tasks/capistrano.rake, and they will automatically be available to Rake.
+
+require_relative 'config/application'
+
+Rails.application.load_tasks

data/demo_app/app/assets/config/manifest.js

@@ -0,0 +1,3 @@
+//= link_tree ../images
+//= link_directory ../javascripts .js
+//= link_directory ../stylesheets .css

data/demo_app/app/assets/javascripts/application.js

@@ -0,0 +1,14 @@
+// This is a manifest file that'll be compiled into application.js, which will include all the files
+// listed below.
+//
+// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, or any plugin's
+// vendor/assets/javascripts directory can be referenced here using a relative path.
+//
+// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the
+// compiled file. JavaScript code in this file should be added after the last require_* statement.
+//
+// Read Sprockets README (https://github.com/rails/sprockets#sprockets-directives) for details
+// about supported directives.
+//
+//= require rails-ujs
+//= require_tree .

data/demo_app/app/assets/javascripts/cable.js

@@ -0,0 +1,13 @@
+// Action Cable provides the framework to deal with WebSockets in Rails.
+// You can generate new channels where WebSocket features live using the `rails generate channel` command.
+//
+//= require action_cable
+//= require_self
+//= require_tree ./channels
+
+(function() {
+  this.App || (this.App = {});
+
+  App.cable = ActionCable.createConsumer();
+
+}).call(this);

data/demo_app/app/assets/stylesheets/application.css

@@ -0,0 +1,15 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS and SCSS file within this directory, lib/assets/stylesheets, or any plugin's
+ * vendor/assets/stylesheets directory can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the bottom of the
+ * compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
+ * files in this directory. Styles in this file should be added after the last require_* statement.
+ * It is generally better to create a new file per style scope.
+ *
+ *= require_tree .
+ *= require_self
+ */

data/demo_app/app/assets/stylesheets/main.scss

@@ -0,0 +1,140 @@
+html {
+  font-family: Helvetica, sans-serif;
+  box-sizing: border-box;
+  height: 100%;
+  width: 80%;
+  margin: 0 auto;
+}
+
+header {
+  width: 70%;
+  margin: 0 auto;
+  text-align: center;
+
+  #app-header {
+    background-color: rgba(243, 243, 243, 0.62);
+    border-radius: 4px;
+    padding: 10px;
+    box-shadow: 1px 1px 1px 0px rgba(0, 0, 0, 0.15);
+  }
+
+  h1 {
+    color: rgba(122, 135, 173, 0.84);
+    cursor: default;
+  }
+}
+
+main {
+  form.work-form {
+    margin: 50px 0;
+    text-align: left;
+    position: absolute;
+    left: 50%;
+    transform: translateX(-30%);
+    width: 50%;
+
+    fieldset {
+      padding: 0;
+      border: none;
+    }
+
+    fieldset.actions {
+      margin: 50px 0;
+      text-align: center;
+      width: 60%;
+
+      input[type='submit'], button[type='reset'] {
+        width: 100px;
+        height: 28px;
+        -webkit-appearance: none;
+        font-size: 14px;
+        background: #e06e51;
+        border: none;
+        border-radius: 4px;
+        box-shadow: 1px 1px 0.5px 0.5px rgba(0, 0, 0, 0.36);
+        color: rgba(235, 235, 235, 0.90);
+        opacity: .85;
+        cursor: pointer;
+        transition: opacity 200ms ease-in-out, color 200ms ease-in-out;
+
+        &:hover {
+          opacity: 1;
+          color: rgba(235, 235, 235, 1);
+        }
+      }
+
+      button[type='reset'] {
+        margin-left: 20px;
+        background: #353535;
+      }
+    }
+
+    label {
+      display: inline-block;
+      margin: 10px 20px 10px 0;
+      width: 30%;
+      color: rgba(0, 0, 0, 0.67);
+    }
+
+    input[type='number'] {
+      display: inline-block;
+      height: 18px;
+      width: 150px;
+      padding: 4px;
+      outline: none;
+      border: 1px solid #000;
+      transition: border-color 200ms ease-in-out;
+      font-size: 14px;
+      border-radius: 2px;
+
+      &:focus {
+        border-color: #e06e51;
+      }
+    }
+  }
+}
+
+footer {
+  text-align: center;
+
+  #app-footer {
+    border-top: 1px solid rgba(0, 0, 0, 0.09);
+    position: absolute;
+    transform: translateX(-50%);
+    bottom: 10px;
+    left: 50%;
+
+    p {
+      color: #b3b3b3;
+
+      a {
+        color: rgba(0, 0, 0, 0.55);
+        transition: color 200ms ease-in-out;
+
+        &:hover {
+          color: #000;
+        }
+      }
+    }
+  }
+}
+
+.message {
+  text-align: center;
+  font-size: 14px;
+  margin: 4px auto;
+  color: rgba(0, 0, 0, 0.65);
+  width: 70%;
+
+  .success {
+    border-radius: 4px;
+    background: #c3e8c3;
+    padding: 6px 0;
+  }
+
+  .error {
+    border-radius: 4px;
+    background: #e8aeae;
+    padding: 6px 0;
+  }
+}