RubyGems - tap-http - Versions diffs - 0.2.1 → 0.3.0 - Mend

tap-http 0.2.1 → 0.3.0

Files changed (14) hide show

data/History +12 -0
data/README +9 -70
data/controllers/capture_controller.rb +55 -0
data/lib/tap/http/get.rb +19 -0
data/lib/tap/http/request.rb +263 -0
data/lib/tap/http/submit.rb +31 -0
data/lib/tap/test/http_test.rb +36 -121
data/views/capture_controller/index.erb +30 -0
metadata +17 -7
data/cgi/echo.rb +0 -24
data/cgi/http_to_yaml.rb +0 -108
data/cgi/parse_http.rb +0 -129
data/lib/tap/http/dispatch.rb +0 -411
data/lib/tap/test/http_test/requests.rb +0 -194

data/History CHANGED Viewed

@@ -1,3 +1,15 @@
+== 0.3.0 / 2009-02-19
+Rework of cgi scripts a tap controllers.  Nearly complete
+internal refactoring (this is not a backwards compatible
+release).
+* Reworked CGI scripts as Tap Controllers
+* Reworked Dispatch as the Request module
+* Added Get and Submit tasks
+* Reworked HttpTest to allow specification of a
+  Rack application
 == 0.2.1 / 2009-02-17
 Minor updates to utilize Tap 0.12.0

data/README CHANGED Viewed

@@ -16,85 +16,24 @@ they allow the capture and resubmission of web forms.
 === Usage
-TapHttp submits http requests using the Tap::Http::Dispatch module.  Headers,
-parameters, and other configurations may be specified, but the only required
-field is :url.
+TapHttp submits http requests using the Tap::Http::Request module.  Headers,
+parameters, and other configurations may be specified, but only a request
+method and uri are required.
   include Tap::Http
-  res = Dispatch.submit(
-    :params => {'q' => 'tap-http'},
-    :url => 'http://www.google.com/search')
+  res = Request.get('http://www.google.com/search')
   res.body[0,80]  # => "<!doctype html><head><title>tap-http - Google Search</title><style>body{backgrou"
-=== Getting Http Configurations
+=== Submitting Web Forms
-More complicated http requests may be captured and resubmitted using a
-combination of tools that redirects web forms to a tap server and reformats
-the request as YAML. To do so:
-* Install {Firefox}[http://www.mozilla.com/en-US/firefox/]
-* Install {Ubiquity}[http://labs.mozilla.com/2008/08/introducing-ubiquity/]
-* Install {redirect-http}[http://gist.github.com/25932]
-Start a tap server from the command line (of course tap-http must be installed):
+Http requests from web forms may be captured and resubmitted using a combination
+of tools.  To do so start a tap server from the command line (of course tap-http
+must be installed):
   % tap server
-Now in the browser, go to a web form like {google}[http://www.google.com/] and
-invoke the redirection.
-* Bring up Ubiquity in Firefox by pressing 'option+space'
-* Enter the command: 'redirect-http http://localhost:8080/http_to_yaml'
-You should see a notice that the form is being redirected.  Fill out the form
-and submit as normal; the redirect command will send the form to the tap server
-instead of performing the original action.  The tap server returns a yaml file
-with the http configuration.
-  # Copy and paste into a configuration file. Multiple configs
-  # can be added to a single file to perform batch submission.
-  - headers:
-      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
-      Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
-      Accept-Encoding: gzip,deflate
-      Accept-Language: en-us,en;q=0.5
-      Connection: keep-alive
-      Host: www.google.com
-      Keep-Alive: "300"
-      Referer: http://www.google.com/
-      User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4
-    params:
-      aq: f
-      btnG: Google Search
-      hl: en
-      oq: ""
-      q: tap-http
-    request_method: GET
-    url: http://www.google.com/search
-    version: 1.1
-Save the file as 'request.yml' and resubmit the form using the Tap::Http::Dispatch
-task.
-  % rap load requests.yml --:i dispatch --+ dump --no-audit
-    I[10:51:40]               load request.yml
-    I[10:51:40]                GET http://www.google.com/search
-    I[10:51:41]                    OK
-  # date: 2008-11-25 10:51:41
-  ---
-  tap/http/request (2772040):
-  - - !ruby/object:Net::HTTPOK
-      body: !binary |
-        H4sIAAAAAAAC/6xabXPbNhL+3l/B0BeN1FAUJfktoihf07pppmkm06TXu0lz
-        HZAEScQgQZOQZVfhf79dgBRJS4ndmRvPSAC42F3sPvsCyssnoQjkXU6NRKZ8
-        tUwoCVdLySSnK0nycSJlboyNl0LEnBrvKCmCZDnRz5elvIMvX4R3W58EV3Eh
-        1lm4OIqiyA0EF8XiyHEc...
-Note the result is encoded as gzip, as per the parameters. As with all tasks,
-the request results could be passed into a workflow. Alternatively, the
-configuration could be used to submit the request using Dispatch.
+Now open a browser and work through the {tutorial}[http://localhost:8080/capture].
 === Bugs/Known Issues

data/controllers/capture_controller.rb ADDED Viewed

@@ -0,0 +1,55 @@
+require 'tap/controller'
+require 'tap/http/utils'
+class CaptureController < Tap::Controller
+  def index
+    render 'index.erb'
+  end
+  def say
+    "<pre>#{request.params['words']}</pre>"
+  end
+  # Echos back redirected HTTP requests  as YAML, suitable for use with the
+  # Tap::Http::Submit task.  All HTTP parameters and headers are echoed back
+  # directly, except for the '__original_action'  parameter which is used in
+  # conjuction with the 'Referer' header to reconstruct the original url of
+  # the request.  The '__original_action' parameter is not echoed.
+  def http_to_yaml
+    headers = {}
+    request.env.each_pair do |key, value|
+      headers[key] = value unless key =~ /^(rack|tap)/
+    end
+    params = request.params
+    original_action = params.delete("__original_action").to_s
+    referer = headers['Referer'].to_s
+    uri = Tap::Http::Utils.determine_url(original_action, referer)
+    headers['Host'] = URI.parse(uri).host
+    config = {
+      'headers' => headers,
+      'params' => params,
+      'uri' => uri,
+      'request_method' => request.request_method
+    }
+    response['Content-Type'] = "text/plain"
+  %Q{# Save as a configuration file.  Resubmit using:
+#
+#  % rap load <config_file> --: submit --: dump --no-audit
+#
+#{YAML.dump(config)}
+}
+  end
+  def echo
+    headers = {}
+    request.env.each_pair do |key, value|
+      headers[key] = value unless key =~ /^(rack|tap)/
+    end
+    "<pre>#{headers.to_yaml}#{request.params.to_yaml}</pre>"
+  end
+end

data/lib/tap/http/get.rb ADDED Viewed

@@ -0,0 +1,19 @@
+require 'tap/http/request'
+module Tap
+  module Http
+    # Tap::Http::Get::manifest gets the uri
+    # Submits an Http request to the specified uri and returns the message body.
+    class Get < Tap::Task
+      include Request
+      def process(uri)
+        log :get, uri
+        res = Request.get(uri)
+        log(nil, res.message)
+        res.body
+      end
+    end
+  end
+end

data/lib/tap/http/request.rb ADDED Viewed

@@ -0,0 +1,263 @@
+require 'tap/http/utils'
+require 'net/http'
+require 'rack/utils'
+module Tap
+  module Http
+    # Request is a module for submitting HTTP requests.  Request take a request
+    # method, a uri, and an options hash allowing these parameters:
+    #
+    #   headers:: a hash of headers
+    #   params:: a hash of parameters, values may be arrays when multiple
+    #            values are assigned to a single key
+    #   version: the HTTP version, by default 1.1
+    #   redirection_limit:: the number of redirections allowed, by default 10
+    #
+    module Request
+      module_function
+      # Constructs and submits an http request to the uri using the specified
+      # request method and options (see above).  Currently only get and post
+      # are supported as request methods. A block may be given to receive the
+      # Net::HTTP and request just prior to submission.
+      #
+      # Returns the Net::HTTP response.
+      def request(request_method, uri, opts={})
+        uri = "http://#{uri}" unless uri.to_s =~ /^http/
+        uri = URI.parse(uri) unless uri.kind_of?(URI)
+        uri.path = "/" if uri.path.empty?
+        headers = opts[:headers] || {}
+        params = opts[:params] || {}
+        # construct the request
+        request = case request_method.to_s
+        when /^get$/i then construct_get(uri, headers, params)
+        when /^post$/i then construct_post(uri, headers, params)
+        else
+          raise ArgumentError, "unsupported request method: #{request_method}"
+        end
+        # set the http version
+        version = opts[:version] || 1.1
+        version_method = "version_#{version.to_s.gsub(".", "_")}".to_sym
+        if ::Net::HTTP.respond_to?(version_method)
+          ::Net::HTTP.send(version_method)
+        else
+          raise ArgumentError, "unsupported HTTP version: #{version}"
+        end
+        # submit the request
+        res = ::Net::HTTP.new(uri.host, uri.port).start do |http|
+          yield(http, request) if block_given?
+          http.request(request)
+        end
+        # fetch redirections
+        redirection_limit = opts[:redirection_limit] || 10
+        redirection_limit ? fetch_redirection(res, redirection_limit) : res
+      end
+      # Shortcut to submit a get request.
+      def get(uri, opts={})
+        request(:get, uri, opts)
+      end
+      # Shortcut to submit a post request.
+      def post(uri, opts={})
+        request(:post, uri, opts)
+      end
+      def escape(str)
+        Rack::Utils.escape(str)
+      end
+      # Constructs a Net::HTTP::Post query, setting headers and parameters.
+      #
+      # ==== Supported Content Types:
+      #
+      # - application/x-www-form-urlencoded (the default)
+      # - multipart/form-data
+      #
+      # The multipart/form-data content type may specify a boundary.  If no
+      # boundary is specified, a randomly generated boundary will be used
+      # to delimit the parameters.
+      #
+      #   post = construct_post(
+      #     URI.parse('http://some.url/'),
+      #     {:content_type => 'multipart/form-data; boundary=1234'},
+      #     {:key => 'value'})
+      #
+      #   post.body
+      #   # => %Q{--1234\r
+      #   # Content-Disposition: form-data; name="key"\r
+      #   # \r
+      #   # value\r
+      #   # --1234--\r
+      #   # }
+      #
+      # (Note the carriage returns are required in multipart content)
+      #
+      # The content-length header is determined automatically from the
+      # formatted request body; manually specified content-length headers
+      # will be overridden.
+      #
+      def construct_post(uri, headers, params)
+        req = ::Net::HTTP::Post.new( "#{uri.path}#{format_query(uri)}" )
+        headers = headerize_keys(headers)
+        content_type = headers['Content-Type']
+        case content_type
+        when nil, /^application\/x-www-form-urlencoded$/i
+          req.body = format_www_form_urlencoded(params)
+          headers['Content-Type'] ||= "application/x-www-form-urlencoded"
+          headers['Content-Length'] = req.body.length
+        when /^multipart\/form-data(;\s*boundary=(.*))?$/i
+          # extract the boundary if it exists
+          boundary = $2 || rand.to_s[2..20]
+          req.body = format_multipart_form_data(params, boundary)
+          headers['Content-Type'] = "multipart/form-data; boundary=#{boundary}"
+          headers['Content-Length'] = req.body.length
+        else
+          raise ArgumentError, "unsupported Content-Type for POST: #{content_type}"
+        end
+        headers.each_pair {|key, value| req[key] = value }
+        req
+      end
+      # Constructs a Net::HTTP::Get query.  All parameters in uri and params are
+      # encoded and added to the request URI.
+      #
+      #   get = construct_get(URI.parse('http://some.url/path'), {}, {:key => 'value'})
+      #   get.path            # => "/path?key=value"
+      #
+      def construct_get(uri, headers, params)
+        req = ::Net::HTTP::Get.new( "#{uri.path}#{format_query(uri, params)}" )
+        headerize_keys(headers).each_pair {|key, value| req[key] = value }
+        req
+      end
+      # Checks the type of the response; if it is a redirection, fetches the
+      # redirection.  Otherwise return the response.
+      #
+      # Notes:
+      # - Fetch will recurse up to the input redirection limit (default 10)
+      # - Responses that are not Net::HTTPRedirection or Net::HTTPSuccess
+      #   raise an error.
+      def fetch_redirection(res, limit=10)
+        raise 'exceeded the redirection limit' if limit < 1
+        case res
+        when ::Net::HTTPRedirection
+          redirect = ::Net::HTTP.get_response( URI.parse(res['location']) )
+          fetch_redirection(redirect, limit - 1)
+        when ::Net::HTTPSuccess
+          res
+        else
+          raise StandardError, res.error!
+        end
+      end
+      # Constructs a URI query string from the uri and the input parameters.
+      # Multiple values for a parameter may be specified using an array.
+      #
+      #   format_query(URI.parse('http://some.url/path'), {:key => 'value'})
+      #   # => "?key=value"
+      #
+      #   format_query(URI.parse('http://some.url/path?one=1'), {:two => '2'})
+      #   # => "?one=1&two=2"
+      #
+      def format_query(uri, params={})
+        query = []
+        query << uri.query if uri.query
+        params.each_pair do |key, values|
+          values = [values] unless values.kind_of?(Array)
+          values.each { |value| query << "#{escape(key)}=#{escape(value)}" }
+        end
+        "#{query.empty? ? '' : '?'}#{query.join('&')}"
+      end
+      # Formats params as 'application/x-www-form-urlencoded' for use as the
+      # body of a post request.  Multiple values for a parameter may be
+      # specified using an array.  The result is obviously URI encoded.
+      #
+      #   format_www_form_urlencoded(:key => 'value with spaces')
+      #   # => "key=value+with+spaces"
+      #
+      def format_www_form_urlencoded(params={})
+        query = []
+        params.each_pair do |key, values|
+          values = [values] unless values.kind_of?(Array)
+          values.each { |value| query << "#{escape(key)}=#{escape(value)}" }
+        end
+        query.join('&')
+      end
+      # Formats params as 'multipart/form-data' using the specified boundary,
+      # for use as the body of a post request.  Multiple values for a parameter
+      # may be specified using an array.  All newlines include a carriage
+      # return for proper formatting.
+      #
+      #   format_multipart_form_data(:key => 'value')
+      #   # => %Q{--1234567890\r
+      #   # Content-Disposition: form-data; name="key"\r
+      #   # \r
+      #   # value\r
+      #   # --1234567890--\r
+      #   # }
+      #
+      # To specify a file, use a hash of file-related headers.
+      #
+      #   format_multipart_form_data(:key => {
+      #     'Content-Type' => 'text/plain',
+      #     'Filename' => "path/to/file.txt"}
+      #   )
+      #   # => %Q{--1234567890\r
+      #   # Content-Disposition: form-data; name="key"; filename="path/to/file.txt"\r
+      #   # Content-Type: text/plain\r
+      #   # \r
+      #   # \r
+      #   # --1234567890--\r
+      #   # }
+      #
+      def format_multipart_form_data(params, boundary="1234567890")
+        body = []
+        params.each_pair do |key, values|
+          values = [values] unless values.kind_of?(Array)
+          values.each do |value|
+            body << case value
+            when Hash
+              hash = headerize_keys(value)
+              filename = hash.delete('Filename') || ""
+              content = File.exists?(filename) ? File.read(filename) : ""
+              header = "Content-Disposition: form-data; name=\"#{key.to_s}\"; filename=\"#{filename}\"\r\n"
+              hash.each_pair { |key, value| header << "#{key}: #{value}\r\n" }
+              "#{header}\r\n#{content}\r\n"
+            else
+              %Q{Content-Disposition: form-data; name="#{key.to_s}"\r\n\r\n#{value.to_s}\r\n}
+            end
+          end
+        end
+        body.collect {|p| "--#{boundary}\r\n#{p}" }.join('') + "--#{boundary}--\r\n"
+      end
+      # Helper to headerize the keys of a hash to headers.
+      # See Utils#headerize.
+      def headerize_keys(hash)
+        result = {}
+        hash.each_pair do |key, value|
+          result[Utils.headerize(key)] = value
+        end
+        result
+      end
+    end
+  end
+end

data/lib/tap/http/submit.rb ADDED Viewed

@@ -0,0 +1,31 @@
+require 'tap/http/request'
+module Tap
+  module Http
+    # Tap::Http::Submit::manifest submits a captured http request
+    class Submit < Tap::Task
+      def process(opts)
+        opts = symbolize(opts)
+        request_method = opts.delete(:request_method) || 'GET'
+        uri = opts.delete(:uri)
+        log request_method, uri
+        res = Request.request(request_method, uri, opts)
+        log(nil, res.message)
+        res.body
+      end
+      protected
+      # taken from ActiveSupport
+      def symbolize(hash) # :nodoc:
+        hash.inject({}) do |options, (key, value)|
+          options[(key.to_sym rescue key) || key] = value
+          options
+        end
+      end
+    end
+  end
+end

data/lib/tap/test/http_test.rb CHANGED Viewed

@@ -1,7 +1,8 @@
+require 'rack'
 require 'webrick'
-require 'singleton'
+require 'thread'
 require 'tap/test/subset_test'
-require 'pp'
+require 'stringio'
 module Tap
   module Test
@@ -11,42 +12,24 @@ module Tap
     # validate echoed requests.
     #
     module HttpTest
-      # Server echos back all requests.
-      class Server
-        include Singleton
-        include WEBrick
-        attr_accessor :server_thread, :server
+      class MockServer
+        def initialize(body, status=200, headers={})
+          @response = [status, headers, [body]]
+        end
-        def initialize
-          # set the default log level to warn to prevent general access logs, unless otherwise specified
-          log_level = ENV["WEB_LOG_LEVEL"] ? ENV["WEB_LOG_LEVEL"].upcase  : "WARN"
-          logger = Log.new($stderr, Log.const_get( log_level ) )
-          self.server = HTTPServer.new(:Port => 2000,
-            :Logger => logger,
-            :AccessLog => [
-              [ logger, AccessLog::COMMON_LOG_FORMAT ],
-              [ logger, AccessLog::REFERER_LOG_FORMAT ]])
-          server.mount_proc("/") do |req, res|
-            res.body << req.request_line
-            res.body << req.raw_header.join('')
-            # an extra line must be added to delimit the headers from the body.
-            if req.body
-              res.body << "\r\n"
-              res.body << req.body
-            end
-            res['Content-Type'] = "text/html"
-          end
+        def call(env)
+          @response
         end
-        # Starts the server on a new thread.
-        def start_web_server
-          self.server_thread ||= Thread.new { server.start }
+      end
+      class EchoServer
+        def self.call(env)
+          body = env['rack.input'].read
+          headers = {}
+          env.each_pair {|key, value| headers[key] = [value] unless key =~ /^rack/ }
+          [200, headers, [body]]
         end
       end
@@ -54,96 +37,28 @@ module Tap
         base.send(:include, Tap::Test::SubsetTest)
       end
-      # WEB subset of tests.  Starts HTTPTest::Server if necessary
-      # and yields to the block.
-      def web_test
-        subset_test("WEB", "w") do
-          Server.instance.start_web_server
-          yield
-        end
+      def default_config(log_dev=StringIO.new(''))
+        common_logger = WEBrick::Log.new(log_dev, WEBrick::Log.const_get(:WARN) )
+        {
+          :Port => 2000,
+          :Logger => common_logger,
+          :AccessLog => common_logger
+        }
       end
-      REQUEST_ATTRIBUTES = %w{
-        request_method  http_version
-        host port path
-        script_name path_info
-        header cookies query
-        accept accept_charset
-        accept_encoding accept_language
-        user
-        addr peeraddr
-        attributes
-        keep_alive}
-      UNCHECKED_REQUEST_ATTRIBUTES = %w{
-        request_line
-        unparsed_uri
-        request_uri
-        request_time
-        raw_header
-        query_string}
-      # Parses expected and actual as an http request (using Tap::Net.parse_http_request)
-      # and asserts that all of the REQUEST_ATTRIBUTES are equal.  See the parse_http_request
-      # documentation for some important notes, particularly involving "\r\n" vs "\n" and
-      # post requests.
-      def assert_request_equal(expected, actual)
-        e = WEBrick::HTTPRequest.new(WEBrick::Config::HTTP)
-        e.parse( StringIO.new(expected) )
-        a = WEBrick::HTTPRequest.new(WEBrick::Config::HTTP)
-        a.parse( StringIO.new(actual) )
-        errors = []
-        REQUEST_ATTRIBUTES.each do |attribute|
-          exp = e.send(attribute)
-          act = a.send(attribute)
-          next if exp == act
-          errors << "<#{PP.singleline_pp(exp, '')}> expected for #{attribute} but was:\n<#{PP.singleline_pp(act, '')}>."
-        end
-        if errors.empty?
-          # this rather unecessary assertion is used simply to
-          # make assert_request_equal cause an assertion.
-          assert errors.empty?
-        else
-          flunk errors.join("\n")
+      def web_test(app=EchoServer, config=default_config)
+        subset_test("WEB", "w") do
+          begin
+            server = ::WEBrick::HTTPServer.new(config);
+            server.mount("/", Rack::Handler::WEBrick, app);
+            Thread.new { server.start }
+            yield
+          ensure
+            server.shutdown
+          end
         end
       end
-      # Convenience method that strips str, strips each line of str, and
-      # then rejoins them using "\r\n" as is typical of HTTP messages.
-      #
-      #   strip_align %Q{
-      #     GET /echo HTTP/1.1
-      #     Accept: */*
-      #     Host: localhost:2000}
-      #
-      #   # => "GET /echo HTTP/1.1\r\nAccept: */*\r\nHost: localhost:2000\r\n"
-      def strip_align(str)
-        str.strip.split(/\r?\n/).collect do |line|
-          "#{line.strip}\r\n"
-        end.compact.join('')
-      end
-      # Turns a hash of parameters into an encoded HTTP query string.
-      # Multiple values for a given key can be specified by an array.
-      #
-      #   to_query('key' => 'value', 'array' => ['one', 'two']) # => "array=one&array=two&key=value"
-      #
-      # Note: the order of the parameters in the result is determined
-      # by hash.each_pair and is thus fairly unpredicatable.
-      def to_query(hash)
-        query = []
-        hash.each_pair do |key,values|
-          values = values.kind_of?(Array) ? values : [values]
-          values.each { |value| query << "#{key}=#{value}" }
-        end
-        URI.encode(query.join('&'))
-      end
     end
   end
 end