RubyGems - tap-http - Versions diffs - 0.2.1 → 0.3.0 - Mend

tap-http 0.2.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (14) hide show

data/History +12 -0
data/README +9 -70
data/controllers/capture_controller.rb +55 -0
data/lib/tap/http/get.rb +19 -0
data/lib/tap/http/request.rb +263 -0
data/lib/tap/http/submit.rb +31 -0
data/lib/tap/test/http_test.rb +36 -121
data/views/capture_controller/index.erb +30 -0
metadata +17 -7
data/cgi/echo.rb +0 -24
data/cgi/http_to_yaml.rb +0 -108
data/cgi/parse_http.rb +0 -129
data/lib/tap/http/dispatch.rb +0 -411
data/lib/tap/test/http_test/requests.rb +0 -194

data/History CHANGED Viewed

@@ -1,3 +1,15 @@
+== 0.3.0 / 2009-02-19
+Rework of cgi scripts a tap controllers.  Nearly complete
+internal refactoring (this is not a backwards compatible
+release).
+* Reworked CGI scripts as Tap Controllers
+* Reworked Dispatch as the Request module
+* Added Get and Submit tasks
+* Reworked HttpTest to allow specification of a
+  Rack application
 == 0.2.1 / 2009-02-17
 Minor updates to utilize Tap 0.12.0

data/README CHANGED Viewed

@@ -16,85 +16,24 @@ they allow the capture and resubmission of web forms.
 === Usage
-TapHttp submits http requests using the Tap::Http::Dispatch module.  Headers,
-parameters, and other configurations may be specified, but the only required
-field is :url.
+TapHttp submits http requests using the Tap::Http::Request module.  Headers,
+parameters, and other configurations may be specified, but only a request
+method and uri are required.
   include Tap::Http
-  res = Dispatch.submit(
-    :params => {'q' => 'tap-http'},
-    :url => 'http://www.google.com/search')
+  res = Request.get('http://www.google.com/search')
   res.body[0,80]  # => "<!doctype html><head><title>tap-http - Google Search</title><style>body{backgrou"
-=== Getting Http Configurations
+=== Submitting Web Forms
-More complicated http requests may be captured and resubmitted using a
-combination of tools that redirects web forms to a tap server and reformats
-the request as YAML. To do so:
-* Install {Firefox}[http://www.mozilla.com/en-US/firefox/]
-* Install {Ubiquity}[http://labs.mozilla.com/2008/08/introducing-ubiquity/]
-* Install {redirect-http}[http://gist.github.com/25932]
-Start a tap server from the command line (of course tap-http must be installed):
+Http requests from web forms may be captured and resubmitted using a combination
+of tools.  To do so start a tap server from the command line (of course tap-http
+must be installed):
   % tap server
-Now in the browser, go to a web form like {google}[http://www.google.com/] and
-invoke the redirection.
-* Bring up Ubiquity in Firefox by pressing 'option+space'
-* Enter the command: 'redirect-http http://localhost:8080/http_to_yaml'
-You should see a notice that the form is being redirected.  Fill out the form
-and submit as normal; the redirect command will send the form to the tap server
-instead of performing the original action.  The tap server returns a yaml file
-with the http configuration.
-  # Copy and paste into a configuration file. Multiple configs
-  # can be added to a single file to perform batch submission.
-  - headers:
-      Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
-      Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
-      Accept-Encoding: gzip,deflate
-      Accept-Language: en-us,en;q=0.5
-      Connection: keep-alive
-      Host: www.google.com
-      Keep-Alive: "300"
-      Referer: http://www.google.com/
-      User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4
-    params:
-      aq: f
-      btnG: Google Search
-      hl: en
-      oq: ""
-      q: tap-http
-    request_method: GET
-    url: http://www.google.com/search
-    version: 1.1
-Save the file as 'request.yml' and resubmit the form using the Tap::Http::Dispatch
-task.
-  % rap load requests.yml --:i dispatch --+ dump --no-audit
-    I[10:51:40]               load request.yml
-    I[10:51:40]                GET http://www.google.com/search
-    I[10:51:41]                    OK
-  # date: 2008-11-25 10:51:41
-  ---
-  tap/http/request (2772040):
-  - - !ruby/object:Net::HTTPOK
-      body: !binary |
-        H4sIAAAAAAAC/6xabXPbNhL+3l/B0BeN1FAUJfktoihf07pppmkm06TXu0lz
-        HZAEScQgQZOQZVfhf79dgBRJS4ndmRvPSAC42F3sPvsCyssnoQjkXU6NRKZ8
-        tUwoCVdLySSnK0nycSJlboyNl0LEnBrvKCmCZDnRz5elvIMvX4R3W58EV3Eh
-        1lm4OIqiyA0EF8XiyHEc...
-Note the result is encoded as gzip, as per the parameters. As with all tasks,
-the request results could be passed into a workflow. Alternatively, the
-configuration could be used to submit the request using Dispatch.
+Now open a browser and work through the {tutorial}[http://localhost:8080/capture].
 === Bugs/Known Issues

data/controllers/capture_controller.rb ADDED Viewed

@@ -0,0 +1,55 @@
+require 'tap/controller'
+require 'tap/http/utils'
+class CaptureController < Tap::Controller
+  def index
+    render 'index.erb'
+  end
+  def say
+    "<pre>#{request.params['words']}</pre>"
+  end
+  # Echos back redirected HTTP requests  as YAML, suitable for use with the
+  # Tap::Http::Submit task.  All HTTP parameters and headers are echoed back
+  # directly, except for the '__original_action'  parameter which is used in
+  # conjuction with the 'Referer' header to reconstruct the original url of
+  # the request.  The '__original_action' parameter is not echoed.
+  def http_to_yaml
+    headers = {}
+    request.env.each_pair do |key, value|
+      headers[key] = value unless key =~ /^(rack|tap)/
+    end
+    params = request.params
+    original_action = params.delete("__original_action").to_s
+    referer = headers['Referer'].to_s
+    uri = Tap::Http::Utils.determine_url(original_action, referer)
+    headers['Host'] = URI.parse(uri).host
+    config = {
+      'headers' => headers,
+      'params' => params,
+      'uri' => uri,
+      'request_method' => request.request_method
+    }
+    response['Content-Type'] = "text/plain"
+  %Q{# Save as a configuration file.  Resubmit using:
+#
+#  % rap load <config_file> --: submit --: dump --no-audit
+#
+#{YAML.dump(config)}
+}
+  end
+  def echo
+    headers = {}
+    request.env.each_pair do |key, value|
+      headers[key] = value unless key =~ /^(rack|tap)/
+    end
+    "<pre>#{headers.to_yaml}#{request.params.to_yaml}</pre>"
+  end
+end

data/lib/tap/http/get.rb ADDED Viewed

@@ -0,0 +1,19 @@
+require 'tap/http/request'
+module Tap
+  module Http
+    # Tap::Http::Get::manifest gets the uri
+    # Submits an Http request to the specified uri and returns the message body.
+    class Get < Tap::Task
+      include Request
+      def process(uri)
+        log :get, uri
+        res = Request.get(uri)
+        log(nil, res.message)
+        res.body
+      end
+    end
+  end
+end

data/lib/tap/http/request.rb ADDED Viewed

@@ -0,0 +1,263 @@
+require 'tap/http/utils'
+require 'net/http'
+require 'rack/utils'
+module Tap
+  module Http
+    # Request is a module for submitting HTTP requests.  Request take a request
+    # method, a uri, and an options hash allowing these parameters:
+    #
+    #   headers:: a hash of headers
+    #   params:: a hash of parameters, values may be arrays when multiple
+    #            values are assigned to a single key
+    #   version: the HTTP version, by default 1.1
+    #   redirection_limit:: the number of redirections allowed, by default 10
+    #
+    module Request
+      module_function
+      # Constructs and submits an http request to the uri using the specified
+      # request method and options (see above).  Currently only get and post
+      # are supported as request methods. A block may be given to receive the
+      # Net::HTTP and request just prior to submission.
+      #
+      # Returns the Net::HTTP response.
+      def request(request_method, uri, opts={})
+        uri = "http://#{uri}" unless uri.to_s =~ /^http/
+        uri = URI.parse(uri) unless uri.kind_of?(URI)
+        uri.path = "/" if uri.path.empty?
+        headers = opts[:headers] || {}
+        params = opts[:params] || {}
+        # construct the request
+        request = case request_method.to_s
+        when /^get$/i then construct_get(uri, headers, params)
+        when /^post$/i then construct_post(uri, headers, params)
+        else
+          raise ArgumentError, "unsupported request method: #{request_method}"
+        end
+        # set the http version
+        version = opts[:version] || 1.1
+        version_method = "version_#{version.to_s.gsub(".", "_")}".to_sym
+        if ::Net::HTTP.respond_to?(version_method)
+          ::Net::HTTP.send(version_method)
+        else
+          raise ArgumentError, "unsupported HTTP version: #{version}"
+        end
+        # submit the request
+        res = ::Net::HTTP.new(uri.host, uri.port).start do |http|
+          yield(http, request) if block_given?
+          http.request(request)
+        end
+        # fetch redirections
+        redirection_limit = opts[:redirection_limit] || 10
+        redirection_limit ? fetch_redirection(res, redirection_limit) : res
+      end
+      # Shortcut to submit a get request.
+      def get(uri, opts={})
+        request(:get, uri, opts)
+      end
+      # Shortcut to submit a post request.
+      def post(uri, opts={})
+        request(:post, uri, opts)
+      end
+      def escape(str)
+        Rack::Utils.escape(str)
+      end
+      # Constructs a Net::HTTP::Post query, setting headers and parameters.
+      #
+      # ==== Supported Content Types:
+      #
+      # - application/x-www-form-urlencoded (the default)
+      # - multipart/form-data
+      #
+      # The multipart/form-data content type may specify a boundary.  If no
+      # boundary is specified, a randomly generated boundary will be used
+      # to delimit the parameters.
+      #
+      #   post = construct_post(
+      #     URI.parse('http://some.url/'),
+      #     {:content_type => 'multipart/form-data; boundary=1234'},
+      #     {:key => 'value'})
+      #
+      #   post.body
+      #   # => %Q{--1234\r
+      #   # Content-Disposition: form-data; name="key"\r
+      #   # \r
+      #   # value\r
+      #   # --1234--\r
+      #   # }
+      #
+      # (Note the carriage returns are required in multipart content)
+      #
+      # The content-length header is determined automatically from the
+      # formatted request body; manually specified content-length headers
+      # will be overridden.
+      #
+      def construct_post(uri, headers, params)
+        req = ::Net::HTTP::Post.new( "#{uri.path}#{format_query(uri)}" )
+        headers = headerize_keys(headers)
+        content_type = headers['Content-Type']
+        case content_type
+        when nil, /^application\/x-www-form-urlencoded$/i
+          req.body = format_www_form_urlencoded(params)
+          headers['Content-Type'] ||= "application/x-www-form-urlencoded"
+          headers['Content-Length'] = req.body.length
+        when /^multipart\/form-data(;\s*boundary=(.*))?$/i
+          # extract the boundary if it exists
+          boundary = $2 || rand.to_s[2..20]
+          req.body = format_multipart_form_data(params, boundary)
+          headers['Content-Type'] = "multipart/form-data; boundary=#{boundary}"
+          headers['Content-Length'] = req.body.length
+        else
+          raise ArgumentError, "unsupported Content-Type for POST: #{content_type}"
+        end
+        headers.each_pair {|key, value| req[key] = value }
+        req
+      end
+      # Constructs a Net::HTTP::Get query.  All parameters in uri and params are
+      # encoded and added to the request URI.
+      #
+      #   get = construct_get(URI.parse('http://some.url/path'), {}, {:key => 'value'})
+      #   get.path            # => "/path?key=value"
+      #
+      def construct_get(uri, headers, params)
+        req = ::Net::HTTP::Get.new( "#{uri.path}#{format_query(uri, params)}" )
+        headerize_keys(headers).each_pair {|key, value| req[key] = value }
+        req
+      end
+      # Checks the type of the response; if it is a redirection, fetches the
+      # redirection.  Otherwise return the response.
+      #
+      # Notes:
+      # - Fetch will recurse up to the input redirection limit (default 10)
+      # - Responses that are not Net::HTTPRedirection or Net::HTTPSuccess
+      #   raise an error.
+      def fetch_redirection(res, limit=10)
+        raise 'exceeded the redirection limit' if limit < 1
+        case res
+        when ::Net::HTTPRedirection
+          redirect = ::Net::HTTP.get_response( URI.parse(res['location']) )
+          fetch_redirection(redirect, limit - 1)
+        when ::Net::HTTPSuccess
+          res
+        else
+          raise StandardError, res.error!
+        end
+      end
+      # Constructs a URI query string from the uri and the input parameters.
+      # Multiple values for a parameter may be specified using an array.
+      #
+      #   format_query(URI.parse('http://some.url/path'), {:key => 'value'})
+      #   # => "?key=value"
+      #
+      #   format_query(URI.parse('http://some.url/path?one=1'), {:two => '2'})
+      #   # => "?one=1&two=2"
+      #
+      def format_query(uri, params={})
+        query = []
+        query << uri.query if uri.query
+        params.each_pair do |key, values|
+          values = [values] unless values.kind_of?(Array)
+          values.each { |value| query << "#{escape(key)}=#{escape(value)}" }
+        end
+        "#{query.empty? ? '' : '?'}#{query.join('&')}"
+      end
+      # Formats params as 'application/x-www-form-urlencoded' for use as the
+      # body of a post request.  Multiple values for a parameter may be
+      # specified using an array.  The result is obviously URI encoded.
+      #
+      #   format_www_form_urlencoded(:key => 'value with spaces')
+      #   # => "key=value+with+spaces"
+      #
+      def format_www_form_urlencoded(params={})
+        query = []
+        params.each_pair do |key, values|
+          values = [values] unless values.kind_of?(Array)
+          values.each { |value| query << "#{escape(key)}=#{escape(value)}" }
+        end
+        query.join('&')
+      end
+      # Formats params as 'multipart/form-data' using the specified boundary,
+      # for use as the body of a post request.  Multiple values for a parameter
+      # may be specified using an array.  All newlines include a carriage
+      # return for proper formatting.
+      #
+      #   format_multipart_form_data(:key => 'value')
+      #   # => %Q{--1234567890\r
+      #   # Content-Disposition: form-data; name="key"\r
+      #   # \r
+      #   # value\r
+      #   # --1234567890--\r
+      #   # }
+      #
+      # To specify a file, use a hash of file-related headers.
+      #
+      #   format_multipart_form_data(:key => {
+      #     'Content-Type' => 'text/plain',
+      #     'Filename' => "path/to/file.txt"}
+      #   )
+      #   # => %Q{--1234567890\r
+      #   # Content-Disposition: form-data; name="key"; filename="path/to/file.txt"\r
+      #   # Content-Type: text/plain\r
+      #   # \r
+      #   # \r
+      #   # --1234567890--\r
+      #   # }
+      #
+      def format_multipart_form_data(params, boundary="1234567890")
+        body = []
+        params.each_pair do |key, values|
+          values = [values] unless values.kind_of?(Array)
+          values.each do |value|
+            body << case value
+            when Hash
+              hash = headerize_keys(value)
+              filename = hash.delete('Filename') || ""
+              content = File.exists?(filename) ? File.read(filename) : ""
+              header = "Content-Disposition: form-data; name=\"#{key.to_s}\"; filename=\"#{filename}\"\r\n"
+              hash.each_pair { |key, value| header << "#{key}: #{value}\r\n" }
+              "#{header}\r\n#{content}\r\n"
+            else
+              %Q{Content-Disposition: form-data; name="#{key.to_s}"\r\n\r\n#{value.to_s}\r\n}
+            end
+          end
+        end
+        body.collect {|p| "--#{boundary}\r\n#{p}" }.join('') + "--#{boundary}--\r\n"
+      end
+      # Helper to headerize the keys of a hash to headers.
+      # See Utils#headerize.
+      def headerize_keys(hash)
+        result = {}
+        hash.each_pair do |key, value|
+          result[Utils.headerize(key)] = value
+        end
+        result
+      end
+    end
+  end
+end

data/lib/tap/http/submit.rb ADDED Viewed

@@ -0,0 +1,31 @@
+require 'tap/http/request'
+module Tap
+  module Http
+    # Tap::Http::Submit::manifest submits a captured http request
+    class Submit < Tap::Task
+      def process(opts)
+        opts = symbolize(opts)
+        request_method = opts.delete(:request_method) || 'GET'
+        uri = opts.delete(:uri)
+        log request_method, uri
+        res = Request.request(request_method, uri, opts)
+        log(nil, res.message)
+        res.body
+      end
+      protected
+      # taken from ActiveSupport
+      def symbolize(hash) # :nodoc:
+        hash.inject({}) do |options, (key, value)|
+          options[(key.to_sym rescue key) || key] = value
+          options
+        end
+      end
+    end
+  end
+end

data/lib/tap/test/http_test.rb CHANGED Viewed

@@ -1,7 +1,8 @@
+require 'rack'
 require 'webrick'
-require 'singleton'
+require 'thread'
 require 'tap/test/subset_test'
-require 'pp'
+require 'stringio'
 module Tap
   module Test
@@ -11,42 +12,24 @@ module Tap
     # validate echoed requests.
     #
     module HttpTest
-      # Server echos back all requests.
-      class Server
-        include Singleton
-        include WEBrick
-        attr_accessor :server_thread, :server
+      class MockServer
+        def initialize(body, status=200, headers={})
+          @response = [status, headers, [body]]
+        end
-        def initialize
-          # set the default log level to warn to prevent general access logs, unless otherwise specified
-          log_level = ENV["WEB_LOG_LEVEL"] ? ENV["WEB_LOG_LEVEL"].upcase  : "WARN"
-          logger = Log.new($stderr, Log.const_get( log_level ) )
-          self.server = HTTPServer.new(:Port => 2000,
-            :Logger => logger,
-            :AccessLog => [
-              [ logger, AccessLog::COMMON_LOG_FORMAT ],
-              [ logger, AccessLog::REFERER_LOG_FORMAT ]])
-          server.mount_proc("/") do |req, res|
-            res.body << req.request_line
-            res.body << req.raw_header.join('')
-            # an extra line must be added to delimit the headers from the body.
-            if req.body
-              res.body << "\r\n"
-              res.body << req.body
-            end
-            res['Content-Type'] = "text/html"
-          end
+        def call(env)
+          @response
         end
-        # Starts the server on a new thread.
-        def start_web_server
-          self.server_thread ||= Thread.new { server.start }
+      end
+      class EchoServer
+        def self.call(env)
+          body = env['rack.input'].read
+          headers = {}
+          env.each_pair {|key, value| headers[key] = [value] unless key =~ /^rack/ }
+          [200, headers, [body]]
         end
       end
@@ -54,96 +37,28 @@ module Tap
         base.send(:include, Tap::Test::SubsetTest)
       end
-      # WEB subset of tests.  Starts HTTPTest::Server if necessary
-      # and yields to the block.
-      def web_test
-        subset_test("WEB", "w") do
-          Server.instance.start_web_server
-          yield
-        end
+      def default_config(log_dev=StringIO.new(''))
+        common_logger = WEBrick::Log.new(log_dev, WEBrick::Log.const_get(:WARN) )
+        {
+          :Port => 2000,
+          :Logger => common_logger,
+          :AccessLog => common_logger
+        }
       end
-      REQUEST_ATTRIBUTES = %w{
-        request_method  http_version
-        host port path
-        script_name path_info
-        header cookies query
-        accept accept_charset
-        accept_encoding accept_language
-        user
-        addr peeraddr
-        attributes
-        keep_alive}
-      UNCHECKED_REQUEST_ATTRIBUTES = %w{
-        request_line
-        unparsed_uri
-        request_uri
-        request_time
-        raw_header
-        query_string}
-      # Parses expected and actual as an http request (using Tap::Net.parse_http_request)
-      # and asserts that all of the REQUEST_ATTRIBUTES are equal.  See the parse_http_request
-      # documentation for some important notes, particularly involving "\r\n" vs "\n" and
-      # post requests.
-      def assert_request_equal(expected, actual)
-        e = WEBrick::HTTPRequest.new(WEBrick::Config::HTTP)
-        e.parse( StringIO.new(expected) )
-        a = WEBrick::HTTPRequest.new(WEBrick::Config::HTTP)
-        a.parse( StringIO.new(actual) )
-        errors = []
-        REQUEST_ATTRIBUTES.each do |attribute|
-          exp = e.send(attribute)
-          act = a.send(attribute)
-          next if exp == act
-          errors << "<#{PP.singleline_pp(exp, '')}> expected for #{attribute} but was:\n<#{PP.singleline_pp(act, '')}>."
-        end
-        if errors.empty?
-          # this rather unecessary assertion is used simply to
-          # make assert_request_equal cause an assertion.
-          assert errors.empty?
-        else
-          flunk errors.join("\n")
+      def web_test(app=EchoServer, config=default_config)
+        subset_test("WEB", "w") do
+          begin
+            server = ::WEBrick::HTTPServer.new(config);
+            server.mount("/", Rack::Handler::WEBrick, app);
+            Thread.new { server.start }
+            yield
+          ensure
+            server.shutdown
+          end
         end
       end
-      # Convenience method that strips str, strips each line of str, and
-      # then rejoins them using "\r\n" as is typical of HTTP messages.
-      #
-      #   strip_align %Q{
-      #     GET /echo HTTP/1.1
-      #     Accept: */*
-      #     Host: localhost:2000}
-      #
-      #   # => "GET /echo HTTP/1.1\r\nAccept: */*\r\nHost: localhost:2000\r\n"
-      def strip_align(str)
-        str.strip.split(/\r?\n/).collect do |line|
-          "#{line.strip}\r\n"
-        end.compact.join('')
-      end
-      # Turns a hash of parameters into an encoded HTTP query string.
-      # Multiple values for a given key can be specified by an array.
-      #
-      #   to_query('key' => 'value', 'array' => ['one', 'two']) # => "array=one&array=two&key=value"
-      #
-      # Note: the order of the parameters in the result is determined
-      # by hash.each_pair and is thus fairly unpredicatable.
-      def to_query(hash)
-        query = []
-        hash.each_pair do |key,values|
-          values = values.kind_of?(Array) ? values : [values]
-          values.each { |value| query << "#{key}=#{value}" }
-        end
-        URI.encode(query.join('&'))
-      end
     end
   end
 end