oni 0.0.1 → 1.0.0

checksums.yaml

@@ -1,15 +1,15 @@
 ---
 !binary "U0hBMQ==":
   metadata.gz: !binary |-
-    NjhlYmExNmMzYzg4ZmY0YWQwMjcwNzE4YTA2OGYyYTQwOTg2ZmI3Nw==
+    Yzg0ZTUxZTdiN2E0YmVkM2M2NGFkYTNiMWY0ZDZiNDVhZDAyMmM1Ng==
   data.tar.gz: !binary |-
-    NGQ3ODEzZmNhYmQ1NTVlMTQzY2IwN2ZjYTZhYWExOWRmZWMxNjZhOQ==
+    MzJiNjliMjhiZjcwZTk4NTkyZjFkZjU4YTI1YjJkMmEyNjQ0Y2EwYw==
 SHA512:
   metadata.gz: !binary |-
-    YTM4YWZiNzA4NzJlOWU4MGRhZjVkZjUxZjhkZmNmMDU5NDM4MGZmNTg0YmJh
-    ZDg5NWE2NTI3MzNjMjQ2NDcwY2UyYzBjOGMzOGY1ZDg5YWM3ZDMzYmVjMzIz
-    NTJlNjdjMDdjNGM0NmNkZDdhZWVhM2YzZjRhMjI0MGU3ZmZiZWQ=
+    OTE4MzgzNjMxZDI5YzZhYjVmYWQ2OTUzZWE1MTI5NzY5YzUzNWY0ZjY4MjNl
+    MzhhN2JkOGFjNTEzMzk2ODU0YjlhNTE5YmY2NGQ3MzMxZTI5MDk3YmFjZWU5
+    YjYwZmI5MmU0NGFmNjkwY2RhMGIxMzk1NDUwYzg4OTUwZGFiNTQ=
   data.tar.gz: !binary |-
-    YzQwNzAzMDRlMDgyNGQ4Mjg2NWY0NmU3YjNhOTQxODYwNWI0NGU1M2YxZjA0
-    ZTJkYTU0MmU4MTA3YmNkMTcxZWY5MmMzZDFhNmYyMDhhYmFkMDcwMWQyYjQ4
-    Y2EzMjQyMTFiNmEyZGMwNmZmNTFmNWRhMDQ0YTc3MTQ1M2U2NjU=
+    NjAzNWExYjhmNWI0ZWQzMjY2NWY0NTMwYjQ1MjZlNjExNDg2NjJlNWY5MmNl
+    ZGY2Y2JiMTExMDgwNGE3ODVlMDJiNjk5NWRmMWM1ZjYwODg4YmQzNWMxYTUz
+    ZDFiNjcxZGM0OTdiYTBhYWZmMjE3MjcxOWVlZmY2MDZmMmMzODc=

data/.gitignore

@@ -0,0 +1,4 @@
+coverage
+yardoc
+pkg
+Gemfile.lock

data/.travis.yml

@@ -0,0 +1,23 @@
+---
+script: rake
+
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1.0-preview1
+  - jruby
+  - jruby-head
+  - jruby-1.7.4
+  - rbx-2.1.1
+
+notifications:
+  email:
+    recipients:
+      - development@olery.com
+    email:
+      on_success: change
+      on_failure: change
+
+branches:
+  only:
+    - master

data/.yardopts

@@ -0,0 +1,12 @@
+./lib/oni/**/*.rb ./lib/oni.rb
+-m markdown
+-M kramdown
+-o yardoc
+-r ./README.md
+--private
+--protected
+--asset ./doc/css/common.css:css/common.css
+--verbose
+-
+./doc/*.md
+LICENSE

data/Gemfile

@@ -0,0 +1,7 @@
+source 'https://rubygems.org/'
+
+gemspec
+
+group :testing do
+  gem 'rubysl', :platform => :rbx
+end

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2013, Olery <http://olery.com/>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,170 @@
+# Oni
+
+* [Design](#design)
+  * [The Daemon](#the-daemon)
+  * [The Mapper](#the-mapper)
+  * [The Worker](#the-worker)
+* [Requirements](#requirements)
+* [Installation & Basic Usage](#installation--basic-usage)
+* [License](#license)
+
+Oni is a Ruby framework that aims to make it easier to write concurrent daemons
+using a common code structure. Oni itself does not actually daemonize your
+code, manage PID files, resources, etc. Instead you should use Oni in
+combination with other Gems such as [daemon-kit][daemon-kit].
+
+Oni was built to standardize the structure amongst the different daemon-kit
+projects used at [Olery][olery]. As time progressed new structures were used
+for new daemons and the old ones were often left as is.
+
+Another problem we faced was concurrency. Most daemons were built in a
+single threaded, single processed manner without an easy place to hook some
+kind of concurrency model in.
+
+Oni takes care of these problems by providing the following:
+
+* A concurrency model in the form of separate worker threads (5 by default).
+* Clear separation of logic into 3 distinctive parts.
+* A common structure for your daemon projects.
+
+Oni assumes developers are somewhat familiar with threading and the potential
+issues that may arise, it also assumes that your code doesn't leak large
+amounts of memory over time. Currently there are no plans to include some kind
+of internal resource management system in Oni, this may change in the future.
+
+## Design
+
+To understand the design of Oni we'll first look at the typical work flow of a
+daemon:
+
+1. A job gets put in a queue (Amazon SQS).
+2. Daemon polls queue, takes message.
+3. Optional message format validation (this was rarely enforced since we had
+   control over the input and assumed it to be correct).
+4. Work gets offloaded to some extra class, in older designs of our daemons it
+   would happen in the daemon layer directly.
+5. Optionally the input data would be modified and re-queued into a separate
+   queue. For example, we often pass along data through multiple queues from
+   the start until the very end (e.g. batch IDs).
+
+Oni tries to make these kind of workflows by breaking them up into 3 different
+layers:
+
+1. A daemon layer tasked with receiving and scheduling work.
+2. A "mapper" tasked with transforming (and validating) input/output for/from
+   the worker. It sits between the daemon and the worker.
+3. A worker that actually performs a task (asynchronously)
+
+Each layer would only do the specific thing it should be doing and would
+offload other work to the next step in the process. The 3 parts are described
+in detail below.
+
+### The Daemon
+
+The daemon layer spawns a number of threads that will each receive and perform
+work separately. Typically these workers are long running tasks that poll some
+kind of message queue for jobs to process.
+
+In initial iterations Oni used a main job dispatcher (running in the main
+thread) and a separate thread pool for the workers. This proved problematic
+with message queue setups as it would result in the main thread pulling in all
+available jobs and then internally queing them again given there weren't enough
+workers available. This would mean that if the process would crash the messages
+were lost. As a result of this each worker is started in it's own separate
+thread.
+
+Comparing Oni with other framework structures one could see the daemon layer as
+a controller (in MVC frameworks), it merely dispatches work to the mapper and
+worker instead of doing everything itself.
+
+### The Mapper
+
+The mapper is tasked with two things:
+
+1. Take the input from the daemon, validate it and transform it into a
+   structure that the worker can understand.
+2. Take the resulting output, optionally modify it and pass it back to the
+   daemon.
+
+The input transformation is put in place to ensure that workers only get data
+that they actually need instead of just receiving the raw message (which may
+include all kinds of meta data completely useless to a worker). It also ensures
+that the input is actually correct before ever passing it to a worker.
+
+A typical thing at Olery is that a job gets scheduled and has to pass through
+multiple steps (= daemons) before being completed. Some of these daemons would
+add extra meta-data to the message (e.g. batch IDs, timings, etc) but these
+aren't strictly required to perform an actual job, thus there's no need to pass
+it around several layers deep into your codebase.
+
+In the above case the mapper would take care of validating/scrubbing the input
+and adding extra meta-data to the output.
+
+### The Worker
+
+The worker would perform the actual work and return some kind of output to the
+daemon. Oni assumes that workers behave reasonably well, currently there's no
+mechanism in place to deal with memory leaks and the likes. Oni also assumes
+that developers are somewhat capable of dealing with asynchronous code since
+all work is performed asynchronously outside of the daemon layer.
+
+## Requirements
+
+* Ruby 1.9.3 or newer, preferably an implementation without a GIL such as
+  Rubinius or Jruby.
+* Basic understanding of threading/concurrent programming
+
+## Installation & Basic Usage
+
+Install the Gem:
+
+    gem install oni
+
+Basic usage of Oni is as following:
+
+    require 'oni'
+
+    class MyWorker < Oni::Worker
+      def initialize(number)
+        @number = number
+      end
+
+      def process
+        return @number * 2
+      end
+    end
+
+    class MyMapper < Oni::Mapper
+      def map_input(input)
+        return input[:number]
+      end
+
+      def map_output(output)
+        return {:number => output, :completed => Time.now}
+      end
+    end
+
+    class MyDaemon < Oni::Daemon
+      set :mapper, MyMapper
+      set :worker, MyWorker
+
+      # Here you'd receive your message, e.g. from a queue. We'll use static
+      # data as an example.
+      def receive
+        yield({:number => 10})
+      end
+
+      # This would get executed upon completion of a job.
+      def complete(message, result, timings)
+        puts result
+      end
+    end
+
+## License
+
+The source code of this repository and Oni itself are licensed under the MIT
+license unless specified otherwise. A copy of this license can be found in the
+file "LICENSE" in the root directory of this repository.
+
+[olery]: http://www.olery.com/
+[daemon-kit]: https://github.com/kennethkalmer/daemon-kit

data/Rakefile

@@ -0,0 +1,13 @@
+require_relative 'lib/oni/version'
+
+require 'rake/clean'
+require 'bundler/gem_tasks'
+require 'ci/reporter/rake/rspec'
+
+CLEAN.include('coverage', 'yardoc')
+
+Dir['./task/*.rake'].each do |task|
+  import(task)
+end
+
+task :default => :test

data/doc/changelog.md

@@ -0,0 +1,6 @@
+# @title Changelog
+# Changelog
+
+## 1.0.0 - November 18th, 2013
+
+The first public release of Oni.

data/doc/css/common.css

@@ -0,0 +1,69 @@
+
+body
+{
+    font-size:   14px;
+    line-height: 1.6;
+    margin:      0 auto;
+    max-width:   960px;
+}
+
+p code
+{
+    background:    #f2f2f2;
+    padding-left:  3px;
+    padding-right: 3px;
+}
+
+pre.code
+{
+    font-size:   13px;
+    line-height: 1.4;
+}
+
+/**
+ * YARD uses generic table styles, using a special class means those tables
+ * don't get messed up.
+ */
+.table
+{
+    border:          1px solid #ccc;
+    border-right:    none;
+    border-collapse: separate;
+    border-spacing:  0;
+    text-align:      left;
+}
+
+.table.full
+{
+    width: 100%;
+}
+
+    .table .field_name
+    {
+        min-width: 160px;
+    }
+
+    .table thead tr th.no_sort:first-child
+    {
+        width: 25px;
+    }
+
+    .table thead tr th, .table tbody tr td
+    {
+        border-bottom:  1px solid #ccc;
+        border-right:   1px solid #ccc;
+        min-width:      20px;
+        padding:        8px 5px;
+        text-align:     left;
+        vertical-align: top;
+    }
+
+    .table tbody tr:last-child td
+    {
+        border-bottom: none;
+    }
+
+    .table tr:nth-child(odd) td
+    {
+        background: #f9f9f9;
+    }

data/examples/github_status.rb

@@ -0,0 +1,75 @@
+require_relative '../lib/oni'
+require 'net/https'
+require 'json'
+
+module GithubStatus
+  class Mapper < Oni::Mapper
+    def map_input(input)
+      return input['url']
+    end
+
+    def map_output(output)
+      return output['status']
+    end
+  end
+
+  class Worker < Oni::Worker
+    attr_reader :url
+
+    def initialize(url)
+      @url = url
+    end
+
+    def process
+      uri_object   = URI.parse(url)
+      http         = Net::HTTP.new(uri_object.host, uri_object.port)
+      http.use_ssl = true
+      request      = Net::HTTP::Get.new(uri_object.request_uri)
+      response     = http.request(request)
+
+      return JSON(response.body)
+    end
+  end
+
+  class Daemon < Oni::Daemon
+    # Check GitHub every 10 minutes.
+    set :interval, 600
+
+    # Since this daemon does the same thing over and over we'll only run 1
+    # thread instead of multiple ones.
+    set :threads, 1
+
+    # The URL to check.
+    set :status_url, 'https://status.github.com/api/status.json'
+
+    set :mapper, Mapper
+    set :worker, Worker
+
+    def receive
+      loop do
+        # This is to mimic some kind of job coming from a queue.
+        yield({'url' => option(:status_url)})
+
+        sleep(option(:interval))
+      end
+    end
+
+    def complete(message, output, timings)
+      sec = timings.real.round(3)
+
+      puts "GitHub status: #{output}, retrieved in #{sec} seconds"
+    end
+  end # Daemon
+end # GithubStatus
+
+daemon = GithubStatus::Daemon.new
+
+%w{INT TERM}.each do |signal|
+  trap(signal) do
+    puts 'Shutting down...'
+
+    daemon.stop
+  end
+end
+
+daemon.start

data/jenkins.sh

@@ -0,0 +1,16 @@
+# This configuration file is used to test/build the project on Olery's private
+# Jenkins instance. Patches containing changes to this file made by people
+# outside of Olery will most likely be rejected.
+
+# The name of the project, used for other settings such as the MySQL database
+# and the package name.
+PROJECT_NAME="oni"
+
+# Whether or not to use a MySQL database, set to a non empty value to enable
+# this. Enabling this will tell Jenkins to create/drop the database and to
+# import any migrations if needed.
+unset USE_MYSQL
+
+# The command to run for the test suite. Junction itself doesn't have a test
+# suite so we'll use a noop.
+TEST_COMMAND="rake jenkins --trace"