threaded_in_memory_queue 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: caa7adac3732ae22522e0e9c76514d76b807c4c4
4
+ data.tar.gz: 61eb8f9b6e4848a16c36dbb6c630df00c4ec5c97
5
+ SHA512:
6
+ metadata.gz: 12cff123c6b70a73e98f2d468db8a9161d063ecc4ad93ce483bd3664828b86b19abe87efb113e831f0754a6e854cccf7d085c57134b688fcec03cf71bcdd36a7
7
+ data.tar.gz: 75309317bc57b8f362a19fe574794e97a1e51eb41b1f257ca023a92827f23b076e89e32493a133f2ce6d69bc7c0f9e44c433d387cbd400213194a8fd9b68e295
@@ -0,0 +1,11 @@
1
+ language: ruby
2
+ rvm:
3
+ - 1.9.3
4
+ - 2.0.0
5
+ - ruby-head
6
+ - jruby-19mode
7
+ - rbx-19mode
8
+
9
+ matrix:
10
+ allow_failures:
11
+ - rvm: ruby-head
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source "https://rubygems.org"
2
+
3
+ gemspec
@@ -0,0 +1,20 @@
1
+ PATH
2
+ remote: .
3
+ specs:
4
+ threaded_in_memory_queue (0.0.1)
5
+
6
+ GEM
7
+ remote: https://rubygems.org/
8
+ specs:
9
+ metaclass (0.0.1)
10
+ mocha (0.14.0)
11
+ metaclass (~> 0.0.1)
12
+ rake (10.1.0)
13
+
14
+ PLATFORMS
15
+ ruby
16
+
17
+ DEPENDENCIES
18
+ mocha
19
+ rake
20
+ threaded_in_memory_queue!
@@ -0,0 +1,97 @@
1
+ ## Threaded In Memory Queue
2
+
3
+ [![Build Status](https://travis-ci.org/schneems/threaded_in_memory_queue.png?branch=master)](https://travis-ci.org/schneems/threaded_in_memory_queue)
4
+
5
+
6
+ A simple non-durable in memory queue for running background tasks using threads.
7
+
8
+ ## Why
9
+
10
+ Projects like [Resque](https://github.com/resque/resque), [delayed job](https://github.com/collectiveidea/delayed_job), [queue classic](https://github.com/ryandotsmith/queue_classic), and [sidekiq](https://github.com/mperham/sidekiq) are great. They use data stores like postgres and redis to store information to be processed later. If you're prototyping a system or don't have access to a data store, you might still want to push off some work to a background process. If that's the case an in-memory threaded queue might be a good fit.
11
+
12
+ ## Install
13
+
14
+ In your `Gemfile`:
15
+
16
+ ```ruby
17
+ gem 'threaded_in_memory_queue'
18
+ ```
19
+
20
+ Then run `$ bundle install`
21
+
22
+ ## Use it
23
+
24
+ Add this code in an initializer to start the in memory queue worker (configuration options are below):
25
+
26
+ ```ruby
27
+ ThreadedInMemoryQueue.start
28
+ ```
29
+
30
+ Define your task to be processed:
31
+
32
+ ```ruby
33
+ class Archive
34
+ def self.call(repo_id, branch = 'master')
35
+ repo = Repository.find(repo_id)
36
+ repo.create_archive(branch)
37
+ end
38
+ end
39
+ ```
40
+
41
+ It can be any object that responds to `call` but we recommend a class or module which makes switching to a durable queue later easier.
42
+
43
+ Then to enqueue a task to be run in the background use `ThreadedInMemoryQueue.enqueue`:
44
+
45
+ ```ruby
46
+ repo = Repo.last
47
+ ThreadedInMemoryQueue.enqueue(Archive, repo.last, 'staging')
48
+ ```
49
+
50
+ The first argument is a class that defines the task to be processed and the rest of the arguments are passed to the task when it is run.
51
+
52
+ # Configure
53
+
54
+ The default number of worker threads is 16, you can configure that when you start your queue:
55
+
56
+ ```ruby
57
+ ThreadedInMemoryQueue.start(size: 5)
58
+ ```
59
+
60
+ By default jobs have a timeout value of 60 seconds. Since this is an in-memory queue (goes away when your process terminates) it is in your best interests to keep jobs small and quick, and not overload the queue. You can configure a different timeout on start:
61
+
62
+ ```ruby
63
+ ThreadedInMemoryQueue.start(timeout: 90) # timeout is in seconds
64
+ ```
65
+
66
+ Want a different logger? Specify a different Logger:
67
+
68
+ ```ruby
69
+ ThreadedInMemoryQueue.start(logger: MyCustomLogger.new)
70
+ ```
71
+
72
+ For testing or guaranteed code execution use the Inline option:
73
+
74
+ ```ruby
75
+ ThreadedInMemoryQueue.inline = true
76
+ ```
77
+
78
+ This option bypasses the queue and executes code as it comes.
79
+
80
+ ## Thread Considerations
81
+
82
+ This worker operates in the same process as your app, that means if your app is CPU bound, it will not be very useful. This worker uses threads which means that to be useful your app needs to either use IO (database calls, file writes/reads, shelling out, etc.) or run on JRuby or Rubinius.
83
+
84
+ To make sure all items in your queue are processed you can add a condition `at_exit` to your program:
85
+
86
+ ```ruby
87
+ at_exit do
88
+ ThreadedInMemoryQueue.stop
89
+ end
90
+ ```
91
+
92
+ This call takes an optional timeout value (in seconds), the default is 10.
93
+
94
+ ## License
95
+
96
+ MIT
97
+
@@ -0,0 +1,14 @@
1
+ # encoding: UTF-8
2
+ require 'bundler/gem_tasks'
3
+
4
+ require 'rake'
5
+ require 'rake/testtask'
6
+
7
+ task :default => [:test]
8
+
9
+ test_task = Rake::TestTask.new(:test) do |t|
10
+ t.libs << 'lib'
11
+ t.libs << 'test'
12
+ t.pattern = 'test/**/*_test.rb'
13
+ t.verbose = false
14
+ end
@@ -0,0 +1,49 @@
1
+ require 'thread'
2
+ require 'timeout'
3
+ require 'logger'
4
+
5
+ require 'threaded_in_memory_queue/version'
6
+ require 'threaded_in_memory_queue/inline'
7
+ require 'threaded_in_memory_queue/timeout'
8
+
9
+ module ThreadedInMemoryQueue
10
+ extend Inline
11
+
12
+ class << self
13
+ attr_accessor :logger
14
+ end
15
+
16
+ def self.start(options = {})
17
+ self.logger = options[:logger] if options[:logger]
18
+ self.master = Master.new(options).start
19
+ return self
20
+ end
21
+
22
+ def self.master
23
+ Thread.current[:threaded_in_memory_queue_master]
24
+ end
25
+
26
+ def self.master=(master)
27
+ Thread.current[:threaded_in_memory_queue_master] = master
28
+ end
29
+
30
+ def self.enqueue(klass, *args)
31
+ raise NoWorkersError, "must start worker before enqueueing jobs" unless master
32
+ master.enqueue(klass, *args)
33
+ return true
34
+ end
35
+
36
+ def self.stop(timeout = 10)
37
+ return true unless master
38
+ master.stop(timeout)
39
+ return true
40
+ end
41
+ end
42
+
43
+ ThreadedInMemoryQueue.logger = Logger.new(STDOUT)
44
+ ThreadedInMemoryQueue.logger.level = Logger::INFO
45
+
46
+
47
+ require 'threaded_in_memory_queue/errors'
48
+ require 'threaded_in_memory_queue/worker'
49
+ require 'threaded_in_memory_queue/master'
@@ -0,0 +1,5 @@
1
+ module ThreadedInMemoryQueue
2
+ class NoWorkersError < RuntimeError; end
3
+
4
+ class WorkerNotStarted < RuntimeError; end
5
+ end
@@ -0,0 +1,15 @@
1
+ module ThreadedInMemoryQueue
2
+ module Inline
3
+ def inline
4
+ Thread.current[:threaded_worker_inline]
5
+ end
6
+
7
+ def inline?
8
+ inline
9
+ end
10
+
11
+ def inline=(inline)
12
+ Thread.current[:threaded_worker_inline] = inline
13
+ end
14
+ end
15
+ end
@@ -0,0 +1,63 @@
1
+ module ThreadedInMemoryQueue
2
+ class Master
3
+ include ThreadedInMemoryQueue::Timeout
4
+ attr_reader :workers, :logger
5
+ extend Inline
6
+
7
+ DEFAULT_TIMEOUT = 60 # seconds, 1 minute
8
+ DEFAULT_SIZE = 16
9
+
10
+ def initialize(options = {})
11
+ @queue = Queue.new
12
+ @size = options[:size] || DEFAULT_SIZE
13
+ @timeout = options[:timeout] || DEFAULT_TIMEOUT
14
+ @logger = options[:logger] || ThreadedInMemoryQueue.logger
15
+ @workers = []
16
+ end
17
+
18
+ def start
19
+ return self if alive?
20
+ @size.times { @workers << Worker.new(@queue, timeout: @timeout).start }
21
+ return self
22
+ end
23
+
24
+ def join
25
+ workers.each {|w| w.join }
26
+ return self
27
+ end
28
+
29
+ def poison
30
+ workers.each {|w| w.poison }
31
+ return self
32
+ end
33
+
34
+ def stop(timeout = 10)
35
+ poison
36
+ timeout(timeout, "waiting for workers to stop") do
37
+ while self.alive?
38
+ sleep 0.1
39
+ end
40
+ self.join
41
+ end
42
+ return self
43
+ end
44
+
45
+ def enqueue(klass, *json)
46
+ if self.class.inline?
47
+ klass.call(*json)
48
+ else
49
+ raise NoWorkersError, "No workers" unless alive?
50
+ @queue.enq([klass, json])
51
+ end
52
+ return true
53
+ end
54
+
55
+ def alive?
56
+ return false if workers.empty?
57
+ workers.detect {|w| w.alive? }
58
+ end
59
+ end
60
+ end
61
+
62
+
63
+
@@ -0,0 +1,11 @@
1
+ module ThreadedInMemoryQueue
2
+ module Timeout
3
+ def timeout(timeout, message = "", &block)
4
+ ::Timeout.timeout(timeout) do
5
+ yield
6
+ end
7
+ rescue ::Timeout::Error
8
+ logger.error("Took longer than #{timeout} to #{message.inspect}")
9
+ end
10
+ end
11
+ end
@@ -0,0 +1,3 @@
1
+ module ThreadedInMemoryQueue
2
+ VERSION = "0.0.1"
3
+ end
@@ -0,0 +1,55 @@
1
+ module ThreadedInMemoryQueue
2
+ class Worker
3
+ DEFAULT_TIMEOUT = 60 # seconds, 1 minute
4
+ POISON = "poison"
5
+ include ThreadedInMemoryQueue::Timeout
6
+ attr_reader :queue, :logger
7
+
8
+ def initialize(queue, options = {})
9
+ @queue = queue
10
+ @timeout = options[:timeout] || DEFAULT_TIMEOUT
11
+ @logger = options[:logger] || ThreadedInMemoryQueue.logger
12
+ end
13
+
14
+ def thread
15
+ raise WorkerNotStarted, "Must start worker before using" unless @thread
16
+ @thread
17
+ end
18
+
19
+ def start
20
+ @thread ||= create_thread
21
+ return self
22
+ end
23
+
24
+ def poison(times = 1)
25
+ @queue.enq(POISON)
26
+ end
27
+
28
+ def alive?
29
+ return false unless @thread
30
+ thread.alive?
31
+ end
32
+
33
+ def join
34
+ return false unless @thread
35
+ thread.join
36
+ end
37
+
38
+ private
39
+ def create_thread
40
+ Thread.new {
41
+ logger.info("Worker #{object_id} ready")
42
+ loop do
43
+ payload = queue.pop
44
+ job, json = *payload
45
+ break if payload == POISON
46
+
47
+ self.timeout(@timeout, "job: #{job.to_s}") do
48
+ job.call(*json)
49
+ end
50
+ end
51
+ logger.info("Worker #{object_id} stopped")
52
+ }
53
+ end
54
+ end
55
+ end
@@ -0,0 +1,11 @@
1
+ Bundler.require
2
+
3
+ require 'threaded_in_memory_queue'
4
+ require 'test/unit'
5
+ require "mocha/setup"
6
+
7
+
8
+ module Dummy
9
+ end
10
+
11
+ ThreadedInMemoryQueue.logger.level = Logger::WARN
@@ -0,0 +1,61 @@
1
+ require 'test_helper'
2
+
3
+ class MasterTest < Test::Unit::TestCase
4
+
5
+ def test_thread_worker_creation
6
+ size = 1
7
+ master = ThreadedInMemoryQueue::Master.new(size: size)
8
+ master.start
9
+ assert_equal size, master.workers.size
10
+ master.stop
11
+
12
+ size = 3
13
+ master = ThreadedInMemoryQueue::Master.new(size: size)
14
+ master.start
15
+ assert_equal size, master.workers.size
16
+ master.stop
17
+
18
+ size = 6
19
+ master = ThreadedInMemoryQueue::Master.new(size: size)
20
+ master.start
21
+ assert_equal size, master.workers.size
22
+ master.stop
23
+
24
+ size = 16
25
+ master = ThreadedInMemoryQueue::Master.new(size: size)
26
+ master.start
27
+ assert_equal size, master.workers.size
28
+ master.stop
29
+ end
30
+
31
+ def test_calls_contents_of_blocks
32
+ Dummy.expects(:process).with(1).once
33
+ Dummy.expects(:process).with(2).once
34
+
35
+ master = ThreadedInMemoryQueue::Master.new(size: 1)
36
+ master.start
37
+
38
+ job = Proc.new {|x| Dummy.process(x) }
39
+
40
+ master.enqueue(job, 1)
41
+ master.enqueue(job, 2)
42
+ master.stop
43
+ end
44
+
45
+ def test_calls_context_of_klass
46
+ Dummy.expects(:process).with(1).once
47
+ Dummy.expects(:process).with(2).once
48
+
49
+ job = Class.new do
50
+ def self.call(num)
51
+ Dummy.process(num)
52
+ end
53
+ end
54
+
55
+ master = ThreadedInMemoryQueue::Master.new(size: 1)
56
+ master.start
57
+ master.enqueue(job, 1)
58
+ master.enqueue(job, 2)
59
+ master.stop
60
+ end
61
+ end
@@ -0,0 +1,40 @@
1
+ require 'test_helper'
2
+
3
+ class WorkerTest < Test::Unit::TestCase
4
+ def setup
5
+ @worker = ThreadedInMemoryQueue::Worker.new(Queue.new, timeout: 1)
6
+ @worker.start
7
+ end
8
+
9
+ def teardown
10
+
11
+ end
12
+
13
+ def test_calls_contents_of_blocks
14
+ Dummy.expects(:process).with(1).once
15
+ Dummy.expects(:process).with(2).once
16
+
17
+ job = Proc.new {|x| Dummy.process(x) }
18
+
19
+ @worker.queue << [job, 1]
20
+ @worker.queue << [job, 2]
21
+ @worker.poison
22
+ @worker.join
23
+ end
24
+
25
+ def test_calls_context_of_klass
26
+ Dummy.expects(:process).with(1).once
27
+ Dummy.expects(:process).with(2).once
28
+
29
+ job = Class.new do
30
+ def self.call(num)
31
+ Dummy.process(num)
32
+ end
33
+ end
34
+
35
+ @worker.queue << [job, 1]
36
+ @worker.queue << [job, 2]
37
+ @worker.poison
38
+ @worker.join
39
+ end
40
+ end
@@ -0,0 +1,56 @@
1
+ require 'test_helper'
2
+
3
+ class ThreadedInMemoryQueueTest < Test::Unit::TestCase
4
+
5
+ def test_calls_contents_of_blocks
6
+ Dummy.expects(:process).with(1).once
7
+ Dummy.expects(:process).with(2).once
8
+
9
+ ThreadedInMemoryQueue.start
10
+
11
+ job = Proc.new {|x| Dummy.process(x) }
12
+
13
+ ThreadedInMemoryQueue.enqueue(job, 1)
14
+ ThreadedInMemoryQueue.enqueue(job, 2)
15
+ ThreadedInMemoryQueue.stop
16
+ end
17
+
18
+ def test_calls_contents_of_klasses
19
+ Dummy.expects(:process).with(1).once
20
+ Dummy.expects(:process).with(2).once
21
+
22
+ ThreadedInMemoryQueue.start
23
+
24
+ job = Class.new do
25
+ def self.call(num)
26
+ Dummy.process(num)
27
+ end
28
+ end
29
+
30
+ ThreadedInMemoryQueue.enqueue(job, 1)
31
+ ThreadedInMemoryQueue.enqueue(job, 2)
32
+ ThreadedInMemoryQueue.stop
33
+ end
34
+
35
+ def test_configure_size
36
+ size = 1
37
+ ThreadedInMemoryQueue.start(size: size)
38
+ assert_equal size, ThreadedInMemoryQueue.master.workers.size
39
+ ThreadedInMemoryQueue.stop
40
+
41
+ size = 3
42
+ ThreadedInMemoryQueue.start(size: size)
43
+ assert_equal size, ThreadedInMemoryQueue.master.workers.size
44
+ ThreadedInMemoryQueue.stop
45
+
46
+ size = 6
47
+ ThreadedInMemoryQueue.start(size: size)
48
+ assert_equal size, ThreadedInMemoryQueue.master.workers.size
49
+ ThreadedInMemoryQueue.stop
50
+
51
+ size = 16
52
+ ThreadedInMemoryQueue.start(size: size)
53
+ assert_equal size, ThreadedInMemoryQueue.master.workers.size
54
+ ThreadedInMemoryQueue.stop
55
+ end
56
+ end
@@ -0,0 +1,23 @@
1
+ # -*- encoding: utf-8 -*-
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'threaded_in_memory_queue/version'
5
+
6
+ Gem::Specification.new do |gem|
7
+ gem.name = "threaded_in_memory_queue"
8
+ gem.version = ThreadedInMemoryQueue::VERSION
9
+ gem.authors = ["Richard Schneeman"]
10
+ gem.email = ["richard.schneeman+rubygems@gmail.com"]
11
+ gem.description = %q{ Queue stuff in memory }
12
+ gem.summary = %q{ Memory, Enqueue stuff you will }
13
+ gem.homepage = "https://github.com/schneems/threaded_in_memory_queue"
14
+ gem.license = "MIT"
15
+
16
+ gem.files = `git ls-files`.split($/)
17
+ gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
18
+ gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
19
+ gem.require_paths = ["lib"]
20
+
21
+ gem.add_development_dependency "rake"
22
+ gem.add_development_dependency "mocha"
23
+ end
metadata ADDED
@@ -0,0 +1,93 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: threaded_in_memory_queue
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Richard Schneeman
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2013-07-15 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: rake
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - '>='
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - '>='
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: mocha
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - '>='
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - '>='
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ description: ' Queue stuff in memory '
42
+ email:
43
+ - richard.schneeman+rubygems@gmail.com
44
+ executables: []
45
+ extensions: []
46
+ extra_rdoc_files: []
47
+ files:
48
+ - .travis.yml
49
+ - Gemfile
50
+ - Gemfile.lock
51
+ - README.md
52
+ - Rakefile
53
+ - lib/threaded_in_memory_queue.rb
54
+ - lib/threaded_in_memory_queue/errors.rb
55
+ - lib/threaded_in_memory_queue/inline.rb
56
+ - lib/threaded_in_memory_queue/master.rb
57
+ - lib/threaded_in_memory_queue/timeout.rb
58
+ - lib/threaded_in_memory_queue/version.rb
59
+ - lib/threaded_in_memory_queue/worker.rb
60
+ - test/test_helper.rb
61
+ - test/threaded_in_memory_queue/master_test.rb
62
+ - test/threaded_in_memory_queue/worker_test.rb
63
+ - test/threaded_in_memory_queue_test.rb
64
+ - threaded_in_memory_queue.gemspec
65
+ homepage: https://github.com/schneems/threaded_in_memory_queue
66
+ licenses:
67
+ - MIT
68
+ metadata: {}
69
+ post_install_message:
70
+ rdoc_options: []
71
+ require_paths:
72
+ - lib
73
+ required_ruby_version: !ruby/object:Gem::Requirement
74
+ requirements:
75
+ - - '>='
76
+ - !ruby/object:Gem::Version
77
+ version: '0'
78
+ required_rubygems_version: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - '>='
81
+ - !ruby/object:Gem::Version
82
+ version: '0'
83
+ requirements: []
84
+ rubyforge_project:
85
+ rubygems_version: 2.0.3
86
+ signing_key:
87
+ specification_version: 4
88
+ summary: Memory, Enqueue stuff you will
89
+ test_files:
90
+ - test/test_helper.rb
91
+ - test/threaded_in_memory_queue/master_test.rb
92
+ - test/threaded_in_memory_queue/worker_test.rb
93
+ - test/threaded_in_memory_queue_test.rb