kronk 1.6.2 → 1.7.0

data/History.rdoc

@@ -1,6 +1,34 @@
+=== 1.7.0 / 2011-10-09
+
+* Major Enhancements:
+
+  * EventMachine backend support for Player.
+
+* Enhancements:
+
+  * Full URL support for player input.
+
+  * Move and map path transaction support with matches and splats.
+
+  * Block support for player input and result.
+
+  * Request proxy :address key changed to :host.
+
+  * Many Player performance improvements, especially on Ubuntu.
+
+* Bugfixes:
+
+  * Support relative URLs in redirects.
+
+  * More accurate data diff metadata output.
+
+  * Lots of Diff::Output and DataString cleanup.
+
+  * Assign URI options to redirects.
+
 === 1.6.2 / 2011-09-12
 
-* Bugfixes
+* Bugfixes:
 
   * Handling of stream output with Diff#formatted is nil.
 

data/Manifest.txt

@@ -5,10 +5,14 @@ README.rdoc
 Rakefile
 bin/kronk
 lib/kronk.rb
+lib/kronk/async.rb
+lib/kronk/async/em_ext.rb
+lib/kronk/async/request.rb
+lib/kronk/async/response.rb
 lib/kronk/cmd.rb
 lib/kronk/constants.rb
 lib/kronk/core_ext.rb
-lib/kronk/data_renderer.rb
+lib/kronk/data_string.rb
 lib/kronk/diff.rb
 lib/kronk/diff/ascii_format.rb
 lib/kronk/diff/color_format.rb
@@ -25,6 +29,7 @@ lib/kronk/player/request_parser.rb
 lib/kronk/player/suite.rb
 lib/kronk/player/stream.rb
 lib/kronk/plist_parser.rb
+lib/kronk/queue_runner.rb
 lib/kronk/request.rb
 lib/kronk/response.rb
 lib/kronk/test.rb

data/README.rdoc

@@ -11,10 +11,14 @@ Kronk was made possible by the sponsoring of AT&T Interactive.
 
 * Parse and display or diff data from http response body and/or headers.
 
-* Include or exclude specific data points with file-glob-like paths.
+* Include, exclude or map specific data points with file-glob-like paths.
+
+* Query and logfile playback with custom output.
 
 * Supports json, rails-ish xml, and plist.
 
+* URI-specific configuration.
+
 * Support for custom data parsers and diff formatters.
 
 * Launch IRB console with the retrieved response data.
@@ -25,12 +29,8 @@ Kronk was made possible by the sponsoring of AT&T Interactive.
 
 * Easy-to-read output (color or ascii, line numbers, etc).
 
-* URI-specific configuration.
-
 * Helper methods for test suites.
 
-* Query and logfile playback with custom output.
-
 * Yes, it works on Windows.
 
 == SYNOPSIS:
@@ -79,12 +79,12 @@ Kronk pulls it's config from $HOME/.kronk/rc and supports the following:
 
 Set the file to save the last http response retrieved in:
 
-  :cache_file: ~/.kronk_cache
+  cache_file: ~/.kronk_cache
 
 Content types to match to a parser. Keys are used as a part of
 a regexp and values are evaluated by const_get.
 
-  :content_types:
+  content_types:
     xml:   XMLParser
     plist: PlistParser
     json:  JSON
@@ -92,23 +92,23 @@ a regexp and values are evaluated by const_get.
 Number of lines of context to use for diff. Full diff is returned when
 set to false:
 
-  :context: 5     # show 5 lines before and after diff
-  :context: false # show the full file
+  context: 5     # show 5 lines before and after diff
+  context: false # show the full file
 
-How to format the diff output. Supports the "special" values
-:ascii_diff and :color_diff or any string that will correctly
+How to format the diff output. Supports the special values
+'ascii' and 'color' or any string that will correctly
 resolve to a constant:
 
-  :diff_format: :ascii_diff
+  diff_format: ascii
 
 Adding User-Agent aliases is useful and simple!
 
-  :user_agents:
+  user_agents:
     win_ie9: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.0)
 
 Disabling cookies altogether may be done as so:
 
-  :use_cookies: false
+  use_cookies: false
 
 Setting URI-specific config options - useful if you always need to send
 headers or use one-off configs. Each URI option key is dropped in a regexp
@@ -116,40 +116,40 @@ to match against the given URIs so this also works for local files.
 See Kronk::Request.retrieve and Kronk::Response#selective_data
 for all options supported:
 
-  :uri_options:
+  uri_options:
     example.com:
-      :parser:      XMLParser
-      :http_method: POST
+      parser:      XMLParser
+      http_method: POST
 
-      :follow_redirects: true
+      follow_redirects: true
 
-      :ignore_data:
+      ignore_data:
         - path/to/data/point
 
-      :headers:
+      headers:
         X-Something: custom header value
 
-      :query:
+      query:
         offset: 1
         limit:  10
 
 Require specific files or gems in the ruby runtime:
 
-  :requires:
+  requires:
     - kronk-my_custom_parser
     - kronk-my_custom_diff_formatter
 
 Show line numbers in the output:
 
-  :show_lines: true
+  show_lines: true
 
 Assign a default host to use when none is specified:
 
-  :default_host: http://localhost:3000
+  default_host: http://localhost:3000
 
 Set the number of spaces for indentation:
 
-  :indentation: 2
+  indentation: 2
 
 === Bash Completion:
 
@@ -158,10 +158,15 @@ Bash completion is available by sourcing the file returned by:
   $ kronk --completion
   [gem path]/script/kronk_completion
 
-== DATA TRAVERSING:
+== DATA MANIPULATION:
 
 One of Kronk's most powerful features is its ability to segregate data.
-From the command line, this is done by passing data paths after '--':
+From the command line, this is done by passing data paths after '--'
+
+=== Selecting and Deleting:
+
+The first kind of data transformation is the ability to select and delete
+data by path:
 
   $ kronk http://host.com -- data/path1 data/path2/1/child
 
@@ -240,6 +245,37 @@ end of the path in the following fashion:
     }
   }
 
+=== Mapping and Moving:
+
+In Kronk, mapping equates to selecting and renaming a given path in an empty
+data structure. Moving refers to deleting the path value and placing it
+back in the original data structure at the new path.
+
+  # Mapping is done with the > operator:
+  $ kronk host.com -- data/path3 "data/path(1|2)>moved%1"
+
+  {
+    "data": {
+      "path3": "value3"
+    }
+    "moved1": "value1",
+    "moved2": "value2"
+  }
+
+
+  # Moving is done with the >> operator:
+  $ kronk host.com -- "data/path(1|2)>moved%1"
+
+  {
+    "data": {
+      "path3": "value3",
+      "path4": "value4"
+    }
+    "moved1": "value1",
+    "moved2": "value2"
+  }
+
+=== Special Characters:
 
 There are additionally a variety of wildcard and special characters that
 are supported:
@@ -263,7 +299,17 @@ are supported:
 
 * \\ escapes any special path character
 
-Check out Kronk::Path and Kronk::DataSet for more details on data traversing.
+* > is used to exclusively map one path to another
+
+* >> moves a path to another within the original data structure
+
+* %NUM is used in the target path to reference a given match
+
+* %% is used to reference splats from the original path and assign it in the
+  target path
+
+Check out Kronk::Path and Kronk::Path::Transaction for more details
+on data manipulation.
 
 Note: Bash may try to parse your data paths, especially if they start with
 wildcards or if they contain the pipe "|" character, so you may need to put

data/Rakefile

@@ -25,9 +25,10 @@ Hoe.spec 'kronk' do
   self.extra_deps << ['json',      '~>1.5']
   self.extra_deps << ['cookiejar', '~>0.3.0']
 
-  self.extra_dev_deps << ['plist',    '~>3.1.0']
-  self.extra_dev_deps << ['nokogiri', '~>1.4']
-  self.extra_dev_deps << ['mocha',    '~>0.9.12']
+  self.extra_dev_deps << ['plist',           '~>3.1.0']
+  self.extra_dev_deps << ['nokogiri',        '~>1.4']
+  self.extra_dev_deps << ['mocha',           '~>0.9.12']
+  self.extra_dev_deps << ['em-http-request', '~>1.0.0']
 end
 
 

data/TODO.rdoc

@@ -1,18 +1,20 @@
 = TODO
 
-== Path
+* Investigate Kronk console.
 
-* Support move and map for path transactions.
+== Done
 
-== Player
+* Investigate the use of EM for QueueRunner.
+
+* Refactor Kronk::Player into a Player and a QueueRunner.
+
+* Support move and map for path transactions.
 
 * Read full URIs for player input but strip off the protocol and host.
 
 * When using player and no host is given, support reading a full URI for
   player input.
 
-== Done
-
 * Support data diffing for arrays (including data struct).
 
 * Allow for showing diffed sections of a diff only.

data/bin/kronk

@@ -1,15 +1,6 @@
 #!/usr/bin/env ruby
 
-$: << File.join(File.dirname(__FILE__), "../lib") if $0 =~ %r{bin/kronk$}
-
-begin
-  require 'kronk'
-
-rescue LoadError => e
-  raise unless e.message =~ %r{no such file to load -- kronk}
-
-  $: << File.join(File.dirname(__FILE__), "../lib")
-  require 'kronk'
-end
+$: << File.join(File.dirname(__FILE__), "../lib") if $0 == "bin/kronk"
 
+require 'kronk'
 Kronk::Cmd.run

data/lib/kronk/async/em_ext.rb

@@ -0,0 +1,34 @@
+require 'em-http-request'
+
+module EventMachine
+  class HttpClient
+    attr_accessor :raw_response
+
+    alias em_parse_response_header parse_response_header
+
+    def parse_response_header header, version, status
+      out = em_parse_response_header header, version, status
+
+      rheader = @response_header
+
+      @raw_response = "HTTP/#{rheader.http_version} "
+      @raw_response << "#{rheader.status} #{rheader.http_reason}\r\n"
+
+      header.each do |key, val|
+        @raw_response << "#{key}: #{val}\r\n"
+      end
+
+      @raw_response << "\r\n"
+
+      out
+    end
+
+
+    alias em_on_decoded_body_data on_decoded_body_data
+
+    def on_decoded_body_data data
+      @raw_response << data
+      em_on_decoded_body_data data
+    end
+  end
+end

data/lib/kronk/async/request.rb

@@ -0,0 +1,73 @@
+class Kronk
+  class Request
+
+    class EMError < Kronk::Exception; end
+
+    ##
+    # Retrieve this requests' response asynchronously with em-http-request.
+    # Returns a EM::HttpConnection instance.
+    #
+    # Passing a block will yield a Kronk::Response instance and/or
+    # an Exception instance if an error was caught.
+    #
+    #   req  = Request.new "example.com"
+    #   em_req = req.retrieve_async do |kronk_response, err|
+    #     # do something with Kronk::Response instance here
+    #   end
+    #
+    #   em_req.callback { ... }
+    #   em_req.error    { ... }
+
+    def retrieve_async &block
+      header_opts = @headers.dup
+
+      if @auth && !@auth.empty?
+        header_opts['Authorization'] ||= []
+        header_opts['Authorization'][0] = @auth[:username] if @auth[:username]
+        header_opts['Authorization'][1] = @auth[:password] if @auth[:password]
+      end
+
+      conn = async_http
+
+      start_time = Time.now
+      req  = conn.setup_request @http_method,
+                :head => header_opts, :body => @body, &block
+
+      req.callback do |resp|
+        elapsed_time   = Time.now - start_time
+        @response      = Response.new resp.raw_response, nil, self
+        @response.time = elapsed_time
+        yield @response, nil
+      end if block_given?
+
+      req.errback do |c|
+        err = c.error ?
+              EMError.new(c.error) :
+              Kronk::NotFoundError.new("#{@uri} could not be found")
+
+        yield nil, err
+      end
+
+      req
+    end
+
+
+    ##
+    # Return an EM::HttpRequest instance.
+
+    def async_http
+      unless @proxy.empty?
+        proxy_opts = @proxy.dup
+        proxy_opts[:authorization] = [
+          proxy_opts.delete(:username),
+          proxy_opts.delete(:password)
+        ] if proxy_opts[:username] || proxy_opts[:password]
+      end
+
+      EventMachine::HttpRequest.new @uri,
+        :connect_timeout    => @timeout,
+        :inactivity_timeout => @timeout,
+        :proxy              => proxy_opts
+    end
+  end
+end

data/lib/kronk/async/response.rb

@@ -0,0 +1,70 @@
+class Kronk
+  class Response
+
+    class AsyncHandler < EM::Connection
+      attr_accessor :buffer
+
+      def initialize req
+        @buffer   = ""
+        @callback = nil
+        @_req     = req
+        @_res     = nil
+      end
+
+
+      def callback &block
+        @callback = block
+      end
+
+
+      def receive_data str
+        @buffer << str
+      end
+
+
+      def unbind
+        return unless @callback
+
+        @_res = Kronk::Response.new @buffer, nil, @_req
+        err = Kronk::Request::EMError.new "IO read error" if error?
+
+        @callback.call @_res, err
+
+      rescue => e
+        @callback.call @_res, e
+      end
+    end
+
+
+    ##
+    # Response.new with asynchronous IO input.
+    # Returns an EM::Connection subclass (AsyncHandler) instance.
+    #
+    # Passing a block will yield a Kronk::Response instance and/or
+    # an Exception instance if an error was caught.
+    #
+    #   conn = Response.from_async_io do |resp, err|
+    #     # do something with Kronk::Response instance here
+    #   end
+
+    def self.from_async_io io, req=nil, &block
+      conn = EM.attach io, AsyncHandler, req
+      conn.comm_inactivity_timeout = 2
+      conn.callback(&block)
+      conn
+    end
+
+
+    ##
+    # Follow the redirect and return a new Response instance.
+    # Returns nil if not redirect-able.
+    #
+    # Passing a block will yield a Kronk::Response instance and/or
+    # an Exception instance if an error was caught.
+
+    def follow_redirect_async opts={}, &block
+      return if !redirect?
+      Request.new(self.location, opts).retrieve_async(&block)
+    end
+  end
+end

data/lib/kronk/async.rb

@@ -0,0 +1,118 @@
+require 'kronk'
+require 'kronk/async/em_ext'
+require 'kronk/async/request'
+require 'kronk/async/response'
+
+class Kronk
+
+  ##
+  # Returns an EM::MultiRequest instance from a url, file, or IO pair.
+  # Calls the given block with a Kronk::Diff object on completion or error.
+  # Assigns @response, @responses, @diff. Must be called from an EM loop.
+  #
+  #   kronk.compare_async uri1, uri2 do |diff, err|
+  #     # handle diff, responses, or error
+  #   end
+
+  def compare_async uri1, uri2
+    multi = EM::MultiRequest.new
+
+    str1 = str2 = ""
+    res1 = res2 = nil
+    err1 = err2 = nil
+
+    conn1 = request_async uri1 do |res, err|
+      err1 = err and next if err
+      res1 = res
+      str1 = res.stringify
+    end
+
+    conn2 = request_async uri2 do |res, err|
+      err2 = err and next if err
+      res2 = res
+      str2 = res.stringify
+    end
+
+    multi.add :left,  conn1
+    multi.add :right, conn2
+
+    multi.callback do
+      next yield(nil, (err1 || err2)) if err1 || err2
+
+      @responses = [res1, res2]
+      @response  = res2
+
+      opts = {:labels => [res1.uri, res2.uri]}.merge @options
+      @diff = Diff.new str1, str2, opts
+
+      yield @diff
+    end
+
+    multi
+
+  rescue => e
+    yield nil, e
+  end
+
+
+  ##
+  # Returns an EventMachine Connection instance from a url, file, or IO.
+  # Calls the given block with a Kronk::Response object on completion or error.
+  # Assigns @response, @responses, @diff. Must be called from an EM loop.
+  #
+  #   kronk.request_async uri do |resp, err|
+  #     # handle response or error
+  #   end
+
+  def request_async uri
+    options = Kronk.config[:no_uri_options] ? @options : options_for_uri(uri)
+
+    rdir = options[:follow_redirects]
+
+    handler = Proc.new do |resp, err|
+      next yield(resp, err) if err
+
+      resp.parser         = options[:parser] if options[:parser]
+      resp.stringify_opts = options
+
+      if resp.redirect? && (rdir == true || Fixnum === rdir && rdir > 0)
+        Cmd.verbose "Following redirect to #{resp.location}"
+
+        rdir = rdir - 1 if Fixnum === rdir
+        opts = options_for_uri resp.location
+        resp.follow_redirect_async(opts, &handler)
+
+      else
+        @responses = [resp]
+        @response  = resp
+        @diff      = nil
+
+        yield resp if block_given?
+
+        resp
+      end
+    end
+
+    if IO === uri
+      Cmd.verbose "Reading IO #{uri}"
+      Response.from_async_io(uri, &handler)
+
+    elsif StringIO === uri
+      Cmd.verbose "Reading IO #{uri}"
+      handler.call Response.new(uri)
+
+    elsif File.file? uri.to_s
+      Cmd.verbose "Reading file:  #{uri}\n"
+      handler.call Response.read_file(uri)
+
+    else
+      req = Request.new uri, options
+      Cmd.verbose "Retrieving URL:  #{req.uri}\n"
+      conn = req.retrieve_async(&handler)
+      conn.callback{ Kronk.history << uri }
+    end
+
+  rescue => e
+    yield nil, e
+  end
+end