piscina 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/LICENSE +21 -0
- data/README.md +104 -0
- data/lib/piscina.rb +3 -0
- data/lib/piscina/http_job.rb +73 -0
- data/lib/piscina/piscina.rb +99 -0
- data/lib/piscina/piscina_logger.rb +75 -0
- metadata +80 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 7be7bd0b99e9978bb6758a13a6a8b8f7e775f871
|
4
|
+
data.tar.gz: 42aeb2c118c89db8abb5d76bc1f0287dda408b6b
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 9b195cea45cb009df944c2898a7bec584e8c655b69ad0409db9339142e04a52a8767ee538aab0b17833f51607a9254cea8ff6c961dad9f79bffc2bfe3099ce75
|
7
|
+
data.tar.gz: 893c4025b9046ac206ec0c7419da930188cbaf64b8094cc1a5814865c336855eceddc14bf0318f09ab891ed298b054738db1d71ad19a223a17967b397a2f88b1
|
data/LICENSE
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2014 Joseph Feeney
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,104 @@
|
|
1
|
+
Piscina
|
2
|
+
=======
|
3
|
+
|
4
|
+
Spanish for "Pool"
|
5
|
+
|
6
|
+
## Blurb
|
7
|
+
JRuby SQS job processor/dispatcher and multi-threaded logger.
|
8
|
+
|
9
|
+
## Dependencies
|
10
|
+
- Some flavor of JRuby
|
11
|
+
- Ruby Standard Library
|
12
|
+
- AWS SDK
|
13
|
+
- FakeSQS for tests
|
14
|
+
|
15
|
+
## Piscina
|
16
|
+
Each instance of Piscina creates N + 2 OS threads. One thread to pull SQS messages off the queue
|
17
|
+
and place them into a buffer, 1 thread for logging, and N threads in a pool that process those jobs.
|
18
|
+
If the number of jobs that are placed into the pool are greater than N, the pool will just buffer
|
19
|
+
those jobs until there is a thread ready to process them. This is handled internally by the Java
|
20
|
+
class Excecutors.
|
21
|
+
|
22
|
+
Piscina instances handle SIGTERM's and other shutdowns gracefully by allowing the buffered jobs
|
23
|
+
to finish before the thread pool is terminated.
|
24
|
+
|
25
|
+
Piscina makes the assumption that each SQS queue can be handled by one class. The class itself
|
26
|
+
can have more complicated logic to process different types of messages based on the content of the
|
27
|
+
message itself but that is left up to you (the developer).
|
28
|
+
|
29
|
+
Piscina takes two arguments
|
30
|
+
SQS_URL : the specific URL for your SQS queue
|
31
|
+
Klass : Klass must know Klass#perform(msg) and must handle the deletion or the immediate
|
32
|
+
requeue-ing of the message if perform is unsuccessful. Otherwise default SQS
|
33
|
+
visibility rules will apply. Exceptions are caught without exception (**ahem**). Dead Letter
|
34
|
+
Queue (DLQ) policies are handled by the SQS queue itself. E.g. three attempts are made
|
35
|
+
to deliver the message but all three attempts result in an exception the SQS queue will
|
36
|
+
place the message in the DLQ for further inspection.
|
37
|
+
|
38
|
+
options[:pool_threads] : How many active threads there should be in each Piscina pool.
|
39
|
+
|
40
|
+
### In use
|
41
|
+
If, for example, you'd like to use a local SQS API in development and in test, but hit the real
|
42
|
+
end points in production you could
|
43
|
+
|
44
|
+
```ruby
|
45
|
+
# Development.rb
|
46
|
+
sqs_url = "https://localhost:4658/QUEUE_NAME"
|
47
|
+
Piscina.new(sqs_url, HttpJob)
|
48
|
+
|
49
|
+
# Production.rb
|
50
|
+
sqs_url = "https://sqs.AWS_REGION.amazonaws.com/YOUR_ACCOUNT_NUM/QUEUE_NAME"
|
51
|
+
Piscina.new(sqs_url, HttpJob)
|
52
|
+
```
|
53
|
+
|
54
|
+
Otherwise you can initialize all of your queues at startup
|
55
|
+
|
56
|
+
## Piscina Logger
|
57
|
+
PiscinaLogger is created to handle logging events for Piscina. However, it is suitable for any
|
58
|
+
logging where many threads need to write to the same file at once, and the order in which they
|
59
|
+
are logged is unimportant.
|
60
|
+
|
61
|
+
PiscinaLogger uses a fixed thread pool of size one to write to the log file. Writes are queued
|
62
|
+
up until the thread is ready to process again.
|
63
|
+
|
64
|
+
### Config Options
|
65
|
+
Configuration Defaults:
|
66
|
+
- log_directory = RailsRoot/log
|
67
|
+
- logging_level = Logger:INFO
|
68
|
+
|
69
|
+
```ruby
|
70
|
+
PiscinaLogger.configure do |config|
|
71
|
+
config.log_directory = "/User/home/derp_kid/derp_code/log"
|
72
|
+
config.logging_level = Logger::DEBUG
|
73
|
+
end
|
74
|
+
```
|
75
|
+
|
76
|
+
The default options are pretty sane. Check them out in configuration.rb.
|
77
|
+
|
78
|
+
## HTTPJob
|
79
|
+
An example job (that works!). This is just an arbitrary job that can be performed with any
|
80
|
+
queue that implements the format below in the message body.
|
81
|
+
|
82
|
+
REMEMBER: SQS does NOT guarantee that messages will be delivered once. Make sure any calls
|
83
|
+
to your endpoint can handle this (idempotency).
|
84
|
+
|
85
|
+
Format:
|
86
|
+
```json
|
87
|
+
{
|
88
|
+
"headers": {
|
89
|
+
"user_token":"That's a nice token" // Any number of Key:Value pairs
|
90
|
+
},
|
91
|
+
"http_method": "GET", // or POST PUT DELETE
|
92
|
+
"uri":"https://SOMEHOST.WHATEVER/your/endpoint", // can be HTTP or HTTPS.
|
93
|
+
// we'll dynamically catch that
|
94
|
+
|
95
|
+
"body":"THIS IS SOME TEXT OR A BODY" // Body will only be processed in Post's or
|
96
|
+
// Put's
|
97
|
+
}
|
98
|
+
```
|
99
|
+
|
100
|
+
# Build and Install without gem
|
101
|
+
`gem build piscina.gemspec && gem install piscina-*.gem`
|
102
|
+
|
103
|
+
# Test
|
104
|
+
`jruby test/simple_test.rb`
|
data/lib/piscina.rb
ADDED
@@ -0,0 +1,73 @@
|
|
1
|
+
require 'net/http'
|
2
|
+
require 'json'
|
3
|
+
|
4
|
+
class HttpJob
|
5
|
+
HTTP_DELETE = "DELETE"
|
6
|
+
HTTP_GET = "GET"
|
7
|
+
HTTP_POST = "POST"
|
8
|
+
HTTP_PUT = "PUT"
|
9
|
+
|
10
|
+
HTTP_METHODS = [
|
11
|
+
HTTP_DELETE,
|
12
|
+
HTTP_GET,
|
13
|
+
HTTP_POST,
|
14
|
+
HTTP_PUT
|
15
|
+
]
|
16
|
+
|
17
|
+
def self.perform(msg)
|
18
|
+
params = JSON.parse(msg.body)
|
19
|
+
|
20
|
+
method = params["http_method"]
|
21
|
+
uri = params["uri"]
|
22
|
+
headers = params["headers"]
|
23
|
+
body = params["body"]
|
24
|
+
|
25
|
+
req = HttpJob.create_request(method, uri, headers, body)
|
26
|
+
res = HttpJob.get_response(req, uri)
|
27
|
+
|
28
|
+
# Net::HTTPSuccess covers all 2XX responses
|
29
|
+
unless res.is_a?(Net::HTTPSuccess)
|
30
|
+
# Messages not explicitly deleted will be placed back into the queue. DLQ policies will take
|
31
|
+
# effect if a message cannot be processed a number of times.
|
32
|
+
raise "Could not perform request to:#{uri.to_s}"
|
33
|
+
end
|
34
|
+
|
35
|
+
msg.delete
|
36
|
+
end
|
37
|
+
|
38
|
+
def self.create_request(method, uri, headers, body)
|
39
|
+
raise "http_method or uri not defined" unless method && uri
|
40
|
+
raise "#{method} is not a valid HTTP method" unless HTTP_METHODS.include?(method)
|
41
|
+
|
42
|
+
begin
|
43
|
+
uri_obj = URI.parse(uri)
|
44
|
+
rescue => e
|
45
|
+
raise "There was an error parsing #{uri}. ErrorMessage: #{e.message}"
|
46
|
+
end
|
47
|
+
|
48
|
+
req = "Net::HTTP::#{method.capitalize}".constantize.new(uri_obj.request_uri)
|
49
|
+
|
50
|
+
if headers
|
51
|
+
headers.each do |header, val|
|
52
|
+
req[header] = val
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
if [HTTP_PUT, HTTP_POST].include?(method) && body
|
57
|
+
req.body = body
|
58
|
+
end
|
59
|
+
|
60
|
+
req
|
61
|
+
end
|
62
|
+
|
63
|
+
def self.get_response(req, uri)
|
64
|
+
uri = URI.parse(uri)
|
65
|
+
|
66
|
+
Net::HTTP.start(uri.hostname,
|
67
|
+
uri.port,
|
68
|
+
:use_ssl => uri.scheme == 'https') do |http|
|
69
|
+
|
70
|
+
http.request(req)
|
71
|
+
end
|
72
|
+
end
|
73
|
+
end
|
@@ -0,0 +1,99 @@
|
|
1
|
+
require_relative 'piscina_logger'
|
2
|
+
|
3
|
+
java_import java.util.concurrent.Executors
|
4
|
+
|
5
|
+
class Piscina
|
6
|
+
DEFAULT_THREADS_IN_POOL = 10 # threads
|
7
|
+
DEFAULT_MESSAGE_TIMEOUT = 15 # seconds
|
8
|
+
MESSAGE_VISIBILITY_TIMEOUT = 12*60 # seconds
|
9
|
+
MESSAGE_RETRY_VISIBILITY = 5 # seconds
|
10
|
+
|
11
|
+
AWS_REGION = 'us-east-1'
|
12
|
+
AWS_ACCOUNT_NUM = '682315851866'
|
13
|
+
|
14
|
+
def initialize(sqs_url, klass, options={})
|
15
|
+
@queue = create_or_initialize_queue(sqs_url)
|
16
|
+
@klass = klass
|
17
|
+
|
18
|
+
pool_threads = options[:pool_threads] || DEFAULT_THREADS_IN_POOL
|
19
|
+
@thread_pool = Executors.newFixedThreadPool(pool_threads)
|
20
|
+
|
21
|
+
# Use SQS queue name for log name.
|
22
|
+
queue_name = sqs_url.split("/")[-1]
|
23
|
+
@logger = PiscinaLogger.new(queue_name)
|
24
|
+
|
25
|
+
# Listen for SIGTERM's and other shutdown messages
|
26
|
+
at_exit do
|
27
|
+
shutdown_instance
|
28
|
+
end
|
29
|
+
|
30
|
+
self.poll
|
31
|
+
end
|
32
|
+
|
33
|
+
# Unfortunately, we can't use the AWS SDK's Queue#poll with a block as it uses
|
34
|
+
# Queue#call_message_block which will ALWAYS delete a received message.
|
35
|
+
def poll
|
36
|
+
# Creates a real OS thread through JRuby
|
37
|
+
Thread.new do
|
38
|
+
poll_sqs
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def poll_sqs
|
43
|
+
loop do
|
44
|
+
msg = @queue.receive_messages(visibility_timeout: MESSAGE_VISIBILITY_TIMEOUT,
|
45
|
+
wait_time_seconds: DEFAULT_MESSAGE_TIMEOUT)
|
46
|
+
|
47
|
+
# receive_message can time out and return nil
|
48
|
+
next if msg.nil?
|
49
|
+
break if @thread_pool.nil? || @thread_pool.isShutdown
|
50
|
+
|
51
|
+
process_message(msg)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
def process_message(msg)
|
56
|
+
@thread_pool.execute do
|
57
|
+
begin
|
58
|
+
@klass.perform(msg)
|
59
|
+
|
60
|
+
body = msg.body.delete("\n").delete(" ")
|
61
|
+
|
62
|
+
@logger.info("Successfully processed message:#{msg.id};body:#{body}")
|
63
|
+
rescue => e
|
64
|
+
@logger.error("Could not process message:#{msg.id};body:#{body};error:#{e.message}")
|
65
|
+
|
66
|
+
# DLQ policy -> Messages are attempted N times and then benched. Policy is defined by the
|
67
|
+
# queue itself.
|
68
|
+
msg.visibility_timeout = MESSAGE_RETRY_VISIBILITY
|
69
|
+
end
|
70
|
+
end
|
71
|
+
end
|
72
|
+
|
73
|
+
def shutdown_instance
|
74
|
+
# Shutting down thread pools does not happen immediately;
|
75
|
+
# all jobs are allowed to finished before thread is closed.
|
76
|
+
#TODO make sure to wait for shutdown
|
77
|
+
@thread_pool.shutdown
|
78
|
+
@logger.shutdown
|
79
|
+
end
|
80
|
+
|
81
|
+
def send_message(text)
|
82
|
+
@queue.send_message(text)
|
83
|
+
end
|
84
|
+
|
85
|
+
private
|
86
|
+
def create_or_initialize_queue(sqs_url)
|
87
|
+
queue = AWS::SQS::Queue.new(sqs_url)
|
88
|
+
|
89
|
+
# TODO should probably not create queues dynamically in prod
|
90
|
+
# make this configurable
|
91
|
+
unless queue.exists?
|
92
|
+
# raise "Queue does not exist" unless queue.exists?
|
93
|
+
queue_name = sqs_url.split("/")[-1]
|
94
|
+
AWS::SQS.new.queues.create(queue_name)
|
95
|
+
end
|
96
|
+
|
97
|
+
queue
|
98
|
+
end
|
99
|
+
end
|
@@ -0,0 +1,75 @@
|
|
1
|
+
java_import java.util.concurrent.Executors
|
2
|
+
|
3
|
+
class PiscinaLogger
|
4
|
+
class << self
|
5
|
+
attr_accessor :configuration
|
6
|
+
end
|
7
|
+
|
8
|
+
def initialize(log_name)
|
9
|
+
# Create a buffered pool of size one that will handle writing to the logs
|
10
|
+
@thread_pool = Executors.newFixedThreadPool(1)
|
11
|
+
|
12
|
+
# Creates a standard Ruby Logger
|
13
|
+
@logger = PiscinaLogger.create_logger(log_name)
|
14
|
+
end
|
15
|
+
|
16
|
+
def self.create_logger(log_name)
|
17
|
+
log_path = construct_path_to_log(log_name)
|
18
|
+
logger = Logger.new(log_path, 'daily')
|
19
|
+
|
20
|
+
logger.formatter = proc do |severity, datetime, progname, msg|
|
21
|
+
"#{severity} [#{datetime.strftime('%Y-%m-%d %H:%M:%S.%L')}]: #{msg}\n"
|
22
|
+
end
|
23
|
+
|
24
|
+
logger.level = PiscinaLogger.configuration.logging_level
|
25
|
+
|
26
|
+
logger
|
27
|
+
end
|
28
|
+
|
29
|
+
def self.construct_path_to_log(log_name)
|
30
|
+
raise "No logging directory defined" unless PiscinaLogger.configuration.log_directory
|
31
|
+
|
32
|
+
log_dir = PiscinaLogger.configuration.log_directory
|
33
|
+
File.join(log_dir, "#{log_name}_piscina.log")
|
34
|
+
end
|
35
|
+
|
36
|
+
def write(message, level)
|
37
|
+
@thread_pool.execute do
|
38
|
+
@logger.send(level, message)
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
[:debug, :info, :warn, :error, :fatal, :unknown].each do |level|
|
43
|
+
define_method("#{level.to_s}") do |message|
|
44
|
+
write(message, level)
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def shutdown
|
49
|
+
# TODO wait for pool to close down before closing logger
|
50
|
+
@thread_pool.shutdown
|
51
|
+
@logger.close
|
52
|
+
end
|
53
|
+
|
54
|
+
# Configure!
|
55
|
+
def self.configure
|
56
|
+
self.configuration ||= Configuration.new
|
57
|
+
yield(configuration)
|
58
|
+
end
|
59
|
+
|
60
|
+
class Configuration
|
61
|
+
attr_accessor :log_directory
|
62
|
+
attr_accessor :logging_level
|
63
|
+
|
64
|
+
def initialize
|
65
|
+
@logging_level = Logger::INFO
|
66
|
+
|
67
|
+
# Assume a rails app
|
68
|
+
@log_directory = if defined?(Rails)
|
69
|
+
File.join(Rails.root.to_s, "log")
|
70
|
+
else
|
71
|
+
nil
|
72
|
+
end
|
73
|
+
end
|
74
|
+
end
|
75
|
+
end
|
metadata
ADDED
@@ -0,0 +1,80 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: piscina
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Joe Feeney
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2014-06-14 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: aws-sdk
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - "~>"
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '1.44'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '1.44'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: fake_sqs
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0.1'
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0.1'
|
41
|
+
description: 'Piscina is a threaded SQS consumer for JRuby. Each Piscina instance
|
42
|
+
is associated with a Klass that knows how to #perform(msg). Piscina can also handle
|
43
|
+
multi threaded logging to a single file through PiscinaLogger.'
|
44
|
+
email:
|
45
|
+
- joe@glasswaves.co
|
46
|
+
executables: []
|
47
|
+
extensions: []
|
48
|
+
extra_rdoc_files: []
|
49
|
+
files:
|
50
|
+
- LICENSE
|
51
|
+
- README.md
|
52
|
+
- lib/piscina.rb
|
53
|
+
- lib/piscina/http_job.rb
|
54
|
+
- lib/piscina/piscina.rb
|
55
|
+
- lib/piscina/piscina_logger.rb
|
56
|
+
homepage: https://github.com/iamatypeofwalrus/Piscina
|
57
|
+
licenses:
|
58
|
+
- MIT
|
59
|
+
metadata: {}
|
60
|
+
post_install_message:
|
61
|
+
rdoc_options: []
|
62
|
+
require_paths:
|
63
|
+
- lib
|
64
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
69
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
70
|
+
requirements:
|
71
|
+
- - ">="
|
72
|
+
- !ruby/object:Gem::Version
|
73
|
+
version: '0'
|
74
|
+
requirements: []
|
75
|
+
rubyforge_project:
|
76
|
+
rubygems_version: 2.3.0
|
77
|
+
signing_key:
|
78
|
+
specification_version: 4
|
79
|
+
summary: JRuby SQS job processor/dispatcher and multi-threaded logger
|
80
|
+
test_files: []
|