pauldowman-sqs_accelerator 0.0.1

data/README.markdown

@@ -0,0 +1,80 @@
+SQS Accelerator
+===============
+
+[http://github.com/pauldowman/sqs_accelerator](http://github.com/pauldowman/sqs_accelerator)
+
+_WARNING: this is totally experimental right now! It's just an idea I'm playing with and I'm trying to see if it works. (But please try it out and tell me what you think!)_
+
+This is a simple and scalable event-driven server (using [Event Machine](http://eventmachine.rubyforge.org)) that proxies requests to Amazon's [Simple Queue Service](http://aws.amazon.com/sqs/) hosted queue. It's purpose is to queue messages very quickly, because otherwise SQS is too slow to use from within a web app (to be precise, the time it takes to queue a message is often too long).
+
+It provides a simple RESTful interface for interacting with SQS that's also convenient to use from a browser.
+
+One instance of the SQS Accelerator can send messages to any number of SQS queues, the queue name is given in the URL and the AWS credentials are given in HTTP headers (as HTTP Basic Authentication headers actually so it's convenient to use from a browser)
+
+
+How it works
+------------
+
+The SQS Accelerator accepts messages from your web app and returns a response instantly without waiting for a response from SQS. Your web app then happily continues and returns it's response to the user. The SQS Accelerator finishes queueing the message to SQS in the background.
+
+It runs as a daemon, if it's accepting queued messages from a web app one instance should be run on each app server. It listens on port 9292.
+
+
+FAQ
+------
+
+__Q: Won't lots of messages build up inside SQS Accelerator since it receives them faster than it sends them to SQS?__
+
+A: No, but you might get errors if you hit the maximum number of open connections (not sure what this limit is yet, but it should be very high with EventMachine). SQS has high latency, but it can accept a virtually unlimited number of incoming connections, so if you can hold a large number of open connections to it then your throughput can be very high. Because an evented client can hold a large number of open connections it can maintain a very high throughput.
+
+__Q: Isn't this like queueing messages locally before queueing them in SQS?__
+
+A: No, it's more like a proxy than a queue, the messages don't build up inside SQS Accelerator, they are all being sent to SQS concurrently.
+
+__Q: Why not just use a local queue?__
+
+A: You could do that instead of using SQS. But SQS is simple and scalable, and it's also simple and scalable to run an instance of SQS Accelerator on each app server.
+
+__Q: Are you sure about all this?__
+
+A: Not yet, it's just a theory so far, my next priority is to benchmark this and make sure it really works as it should in theory.
+
+
+Usage instructions
+------------------
+
+* Install this gem
+* Run sqs_accelerator.ru
+* Hit [http://localhost:9292/](http://localhost:9292/) in a browser
+  * Make an HTTP GET request to /queues to list all queues
+  * Make an HTTP POST request to /queues with queue_name=newqueue to create a queue named newqueue
+  * Make an HTTP GET request to /queues/queuename to show info about a queue named "queuename" and to give a form to send a message
+  * Make an HTTP POST request to /queues/queuename to send a message
+  * Use your SQS credentials for HTTP auth  
+* TODO improve these instructions
+
+
+Thanks to the following great projects
+--------------------------------------
+
+* [Event Machine](http://eventmachine.rubyforge.org), the thing that makes this even possible.
+* [async_sinatra](http://github.com/raggi/async_sinatra), allows writing evented HTTP servers using the nice Sinatra framework/DSL.
+* [em-http-request](http://github.com/igrigorik/em-http-request), an asynchronous HTTP Client.
+* [RightAWS](http://rightscale.rubyforge.org/right_aws_gem_doc) (I stole a few lines of the SQS code).
+* [Amazon SQS](http://aws.amazon.com/sqs/) for having a decent [Query API](http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/), even if it's not actually RESTful. :-)
+
+
+Still to do
+-----
+
+* Find all the TODO comments in the code
+* Benchmarking
+* A command to start and stop the daemon, maybe a [god](http://god.rubyforge.org/) config file
+* Use SSL when connecting to SQS to protect message content (AWS credentials are never sent, they're just used to sign the message)
+* Unit tests (I'm just trying to figure out if this idea even works first)
+* Create a Ruby client library
+* Switch all actions to use evented HTTP client instead of EM.defer. Right now some actions are using EM.defer to use the RightAWS client in a thread. These actions will be less scalable. Sending messages, the most important action, _is_ using the evented client. This means making direct HTTP requests to the [SQS Query API](http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/)
+* Fix bugs and make it nicer.
+* Refactor all the SQS stuff out of the actions
+* Some configuration options
+

data/lib/server.rb

@@ -0,0 +1,166 @@
+require "em-http"
+require "haml"
+require "logger"
+require "pp"
+require "sinatra/async"
+require "xml"
+
+require "#{File.dirname(__FILE__)}/sqs_accelerator"
+require "#{File.dirname(__FILE__)}/sqs_helper"
+require "#{File.dirname(__FILE__)}/sqs_proxy"
+
+class SqsAccelerator::Server < Sinatra::Base  
+  register Sinatra::Async
+  
+  configure do 
+    # TODO fix logging.
+    LOGGER = Logger.new(STDOUT)
+    LOGGER.level = Logger::DEBUG
+    use Rack::CommonLogger, LOGGER
+  end
+  
+  # use http basic auth to get aws_access_key_id and aws_secret_access_key
+  helpers do
+    include SqsAccelerator::SqsHelper
+
+    def deny
+      response['WWW-Authenticate'] = %(Basic realm="SQS Accelerator - use SQS credentials")
+      response.status = 401 # "Unauthorized": http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
+      body "Please provide the SQS access key id and secret access key as the userid and password.\n"
+    end
+
+    def authorized?
+      @auth ||=  Rack::Auth::Basic::Request.new(request.env)
+      @auth.provided? && @auth.basic? && @auth.credentials
+      # We don't care what the credentials are, we'll just pass them on to SQS
+    end
+    
+    def aws_access_key_id
+      @auth.credentials[0]
+    end
+    
+    def aws_secret_access_key
+      @auth.credentials[1]
+    end
+    
+    def logger
+      LOGGER
+    end
+  end
+  
+  aget '/' do
+    body haml :home
+  end
+  
+  aget '/queues' do
+    deny and return unless authorized?
+    
+    request_hash = generate_request_hash('ListQueues')
+    http = EventMachine::HttpRequest.new("http://queue.amazonaws.com/").get :query => request_hash, :timeout => 120
+    
+    http.callback {
+      queue_urls = []
+      doc = parse_response(http.response)
+      doc.find('//sqs:QueueUrl').each do |u|
+        queue_urls << u.content.strip
+      end
+      
+      body haml :all_queues, :locals => { :queue_urls => queue_urls }
+    }
+    
+    http.errback {
+      # TODO a decent log message and error page here
+      logger.debug "fail"
+      body "fail"
+    }
+    
+  end
+  
+  # Create a queue
+  apost '/queues' do
+    deny and return unless authorized?
+
+    queue_name = params[:queue_name]
+    visibility_timeout = params[:visibility_timeout]
+    
+    # TODO switch to using evented client
+    operation = proc do
+      sqs = SqsAccelerator::SqsProxy.new(aws_access_key_id, aws_secret_access_key, :logger => logger)
+      sqs.create_queue(queue_name, visibility_timeout)
+      # TODO check the result and return an error unless success
+      # TODO set Location header field with URI
+      response.status = 201 # "Created": http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
+      body haml :queue_created, :locals => { :queue_name => queue_name }
+    end
+    EM.defer(operation)    
+  end
+  
+  # Get info on a queue. Doesn't create the queue if it doesn't exist
+  aget '/queues/:queue_name' do
+    deny and return unless authorized?
+  
+    queue_name = params[:queue_name]
+  
+    # TODO switch to using evented client
+    operation = proc do          
+      sqs = SqsAccelerator::SqsProxy.new(aws_access_key_id, aws_secret_access_key, :logger => logger)
+      queue_info = sqs.get_queue_info(queue_name)
+      body haml :queue_info, :locals => { :queue_name => queue_name, :queue_info => queue_info }
+    end
+    EM.defer(operation)
+  end
+  
+  # Send a new message on a queue. Returns immediately and sends message asynchronously
+  apost '/queues/:queue_name' do
+    deny and return unless authorized?
+
+    queue_name = params[:queue_name]
+    message_body = params[:message_body]
+    
+    logger.info "Received message for queue #{queue_name}"
+    logger.debug "message_body: #{message_body}"
+    
+    # TODO check that chars are allowed by SQS: #x9 | #xA | #xD | [#x20 to #xD7FF] | [#xE000 to #xFFFD] | [#x10000 to #x10FFFF]
+    # TODO deal with unicode, where chars != bytes
+    if message_body.size > SqsAccelerator::SqsHelper::MAX_MESSAGE_SIZE
+      logger.error "Message is too large, rejecting."
+      response.status = 413 # "Request Entity Too Large": http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
+      body "Message must be less than SqsAccelerator::SqsHelper::MAX_MESSAGE_SIZE bytes"
+      return
+    end
+    
+    # Messaage seems to be OK, return a response immediately then send message to SQS
+    response.status = 202 # "Accepted": http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
+    body "Message will be queued, check log for errors.\n"
+    # TODO generate and return a message id then log that with the SQS message id?
+    
+    request_hash = generate_request_hash("SendMessage", :message => message_body)
+    # build the body string ourselves for now until Ilya's gem gets updated because of the content-length bug
+    request_body = request_hash.to_a.collect{|key,val| CGI.escape(key.to_s) + "=" + CGI.escape(val.to_s) }.join("&")
+    http = EventMachine::HttpRequest.new("http://queue.amazonaws.com/#{queue_name}").post({:body => request_body, :head => {"Content-Type" => "application/x-www-form-urlencoded"}})
+    
+    http.callback {
+      doc = parse_response(http.response)
+      id_el = doc.find_first('//sqs:MessageId')
+      md5_el = doc.find_first('//sqs:MD5OfMessageBody')
+      if id_el && md5_el
+        message_id = id_el.content.strip
+        checksum = md5_el.content.strip
+        # TODO check md5
+        # TODO a decent log message here
+        logger.info "Queued message, SQS message id is: #{message_id}"
+      else
+        logger.error "SQS returned an error response"
+        # TODO parse the response and print something useful
+        # TODO retry a few times with exponentially increasing delay
+      end
+    }
+    
+    http.errback {
+      # TODO a decent log message here
+      logger.error "fail"
+      # TODO dump the message to a temp file and write a utility to re-send dumped messages
+    }
+    
+  end
+end

data/lib/sqs_accelerator.rb

@@ -0,0 +1,2 @@
+module SqsAccelerator  
+end

data/lib/sqs_helper.rb

@@ -0,0 +1,63 @@
+require "cgi"
+require "base64"
+require "rexml/document"
+require "openssl"
+require "digest/sha1"
+require 'md5'
+
+module SqsAccelerator::SqsHelper
+  # Thanks to RightScale for the great RightAWS library.
+  # Parts of this code have been inspired by/stolen from RightAws.
+
+  SIGNATURE_VERSION = "1"
+  API_VERSION       = "2008-01-01"
+  DEFAULT_HOST      = "queue.amazonaws.com"
+  DEFAULT_PORT      = 80
+  DEFAULT_PROTOCOL  = 'http'
+  REQUEST_TTL       = 30
+  DEFAULT_VISIBILITY_TIMEOUT = 30
+  MAX_MESSAGE_SIZE  = (8 * 1024)
+
+  @@digest = OpenSSL::Digest::Digest.new("sha1")
+
+  def sign(aws_secret_access_key, auth_string)
+    Base64.encode64(OpenSSL::HMAC.digest(@@digest, aws_secret_access_key, auth_string)).strip
+  end
+  
+  # From Amazon's SQS Dev Guide, a brief description of how to escape:
+  # "URL encode the computed signature and other query parameters as specified in 
+  # RFC1738, section 2.2. In addition, because the + character is interpreted as a blank space 
+  # by Sun Java classes that perform URL decoding, make sure to encode the + character 
+  # although it is not required by RFC1738."
+  # Avoid using CGI::escape to escape URIs. 
+  # CGI::escape will escape characters in the protocol, host, and port
+  # sections of the URI.  Only target chars in the query
+  # string should be escaped.
+  def URLencode(raw)
+    e = URI.escape(raw)
+    e.gsub(/\+/, "%2b")
+  end
+  
+  def generate_request_hash(action, params={})
+    message = params[:message]
+    params.each{ |key, value| params.delete(key) if (value.nil? || key.is_a?(Symbol)) }
+    request_hash = { "Action"           => action,
+                     "Expires"          => (Time.now + REQUEST_TTL).utc.strftime("%Y-%m-%dT%H:%M:%SZ"),
+                     "AWSAccessKeyId"   => aws_access_key_id,
+                     "Version"          => API_VERSION,
+                     "SignatureVersion" => SIGNATURE_VERSION }
+    request_hash["MessageBody"] = message if message
+    request_hash.merge(params)
+    request_data = request_hash.sort{|a,b| (a[0].to_s.downcase)<=>(b[0].to_s.downcase)}.to_s
+    request_hash['Signature'] = sign(aws_secret_access_key, request_data)
+    logger.debug "request_hash:\n#{request_hash.pretty_inspect}"
+    return request_hash
+  end
+
+  def parse_response(string)
+    parser = XML::Parser.string(string)
+    doc = parser.parse
+    doc.root.namespaces.default_prefix = "sqs"
+    return doc
+  end
+end

data/lib/sqs_proxy.rb

@@ -0,0 +1,27 @@
+require "right_aws"
+
+# This class will be removed. It's just for the operations that haven't been 
+# switched to use the evented http client yet.
+
+class SqsAccelerator::SqsProxy
+
+  def initialize(aws_access_key_id, aws_secret_access_key, options = {})
+    options = {:multi_thread => true}.merge(options)
+    @logger = options[:logger]
+    @sqs = RightAws::SqsGen2.new(aws_access_key_id, aws_secret_access_key, options)
+  end
+  
+  def create_queue(queue_name, visibility_timeout)
+    RightAws::SqsGen2::Queue.create(@sqs, queue_name, true, visibility_timeout)
+  end
+  
+  def get_queue_info(queue_name)
+    # TODO do something smart if the queue doesn't exist
+    queue = @sqs.queue(queue_name, false)
+    queue_info = {
+      :num_messages => queue.size,
+      :visibility_timeout => queue.visibility
+    }
+    return queue_info
+  end
+end

data/sqs_accelerator.gemspec

@@ -0,0 +1,30 @@
+spec = Gem::Specification.new do |s|
+  s.name = 'sqs_accelerator'
+  s.version = '0.0.1'
+  s.date = '2009-06-22'
+  s.summary = 'SQS Accelerator'
+  s.description = "A simple and scalable event-drive server that proxies requests to Amazon's Simple Queue Service to queue messages very quickly."
+  s.email = 'paul@pauldowman.com'
+  s.homepage = "http://github.com/pauldowman/sqs-accelerator"
+  s.has_rdoc = false
+  s.authors = ["Paul Dowman"]
+  s.add_dependency('eventmachine', '>= 0.12.2')
+  s.add_dependency('igrigorik-em-http-request', '>= 0.1.6')
+  s.add_dependency('sinatra', '>= 0.9.2')
+  s.add_dependency('async_sinatra', '>= 0.1.4')
+  s.add_dependency('libxml-ruby', '>= 1.1.3')
+  s.rubyforge_project = "sqs-accelerator"
+
+  # ruby -rpp -e' pp `git ls-files`.split("\n") '
+  s.files = ["README.markdown",
+   "lib/server.rb",
+   "lib/sqs_accelerator.rb",
+   "lib/sqs_helper.rb",
+   "lib/sqs_proxy.rb",
+   "sqs_accelerator.gemspec",
+   "sqs_accelerator.ru",
+   "views/all_queues.haml",
+   "views/home.haml",
+   "views/queue_info.haml"]
+  
+end

data/sqs_accelerator.ru

@@ -0,0 +1,5 @@
+#!/usr/bin/env rackup -Ilib:../lib -s thin
+
+require 'lib/server'
+
+run SqsAccelerator::Server.new

data/views/all_queues.haml

@@ -0,0 +1,13 @@
+!!! XML
+!!!
+%html{:xmlns => "http://www.w3.org/1999/xhtml", 'xml:lang' => "en", :lang => "en"}
+  %head
+    %title
+      SQS Queues
+  %body
+    %h3
+      SQS Queues:
+
+    - queue_urls.each do |url|
+      = url
+      %br/

data/views/home.haml

@@ -0,0 +1,26 @@
+!!! XML
+!!!
+%html{:xmlns => "http://www.w3.org/1999/xhtml", 'xml:lang' => "en", :lang => "en"}
+  %head
+    %title
+      SQS Accelerator
+  %body
+    %h3
+      SQS Accelerator
+      
+  %p
+    TODO more detail and links here
+    %ul
+      %li
+        Make an HTTP GET request to 
+        %a{:href => "/queues"}
+          \/queues
+        to list all queues
+      %li
+        Make an HTTP POST request to /queues with queue_name=newqueue to create a queue named newqueue
+      %li
+        Make an HTTP GET request to /queues/queuename to show info about a queue named "queuename" and to give a form to send a message
+      %li
+        Make an HTTP POST request to /queues/queuename to send a message
+      %li
+        Use your SQS credentials for HTTP auth

data/views/queue_info.haml

@@ -0,0 +1,31 @@
+!!! XML
+!!!
+%html{:xmlns => "http://www.w3.org/1999/xhtml", 'xml:lang' => "en", :lang => "en"}
+  %head
+    %title
+      SQS Queue name: 
+      = queue_name
+  %body
+    %h3
+      SQS Queue name: 
+      = queue_name
+
+    %p
+      Approximate number of messages in queue: 
+      = queue_info[:num_messages]
+
+    %p
+      Visibility timeout:
+      = queue_info[:visibility_timeout]
+
+    %p
+      Queue a new message:
+    %form{:method => "POST"}
+      %p
+        Send message:
+        %br/
+        %textarea{:name => "message_body", :cols => "80", :rows => "20"}
+
+      %p
+        %input{:type => "submit"}
+

metadata

@@ -0,0 +1,111 @@
+--- !ruby/object:Gem::Specification 
+name: pauldowman-sqs_accelerator
+version: !ruby/object:Gem::Version 
+  version: 0.0.1
+platform: ruby
+authors: 
+- Paul Dowman
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-06-22 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: eventmachine
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.12.2
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: igrigorik-em-http-request
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.1.6
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: sinatra
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.9.2
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: async_sinatra
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.1.4
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: libxml-ruby
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.1.3
+    version: 
+description: A simple and scalable event-drive server that proxies requests to Amazon's Simple Queue Service to queue messages very quickly.
+email: paul@pauldowman.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- README.markdown
+- lib/server.rb
+- lib/sqs_accelerator.rb
+- lib/sqs_helper.rb
+- lib/sqs_proxy.rb
+- sqs_accelerator.gemspec
+- sqs_accelerator.ru
+- views/all_queues.haml
+- views/home.haml
+- views/queue_info.haml
+has_rdoc: false
+homepage: http://github.com/pauldowman/sqs-accelerator
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: sqs-accelerator
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: SQS Accelerator
+test_files: []
+