em-http-request 0.3.0 → 1.0.0.beta.1

data/.gitignore

@@ -3,3 +3,4 @@
 Makefile
 mkmf.log
 Gemfile.lock
+misc

data/Changelog.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 1.0.0.beta.1 / 2011-02-20 - The big rewrite
+
+- Switched parser from Ragel to http_parser.rb
+- Removed em_buffer C extension
+- Added support for HTTP keepalive
+- Added support for HTTP pipelining
+- ~60% performance improvement across the board: less GC time!
+- Refactored & split all tests
+- Basic 100-Continue handling on POST/PUT
+
 ## 0.3.0 / 2011-01-15
 
 - IMPORTANT: default to non-persistent connections (timeout => 0 now requires :keepalive => true)

data/README.md

@@ -1,175 +1,58 @@
-EM-HTTP-Request
-===============
+# EM-HTTP-Request
 
-Asynchronous HTTP client for Ruby, based on EventMachine runtime.
+Async (EventMachine) HTTP client, with support for:
 
-- Ragel HTTP parser for speed & performance
-- Simple interface for single & parallel requests via deferred callbacks
+- Asynchronous HTTP API for single & parallel request execution
+- Keep-Alive and HTTP pipelining support
+- Auto-follow 3xx redirects with max depth
 - Automatic gzip & deflate decoding
-- Basic-Auth & OAuth support
-- Custom timeout support
-- Stream response processing
-- Proxy support (with SSL Tunneling): CONNECT, direct & SOCKS5
-- Auto-follow 3xx redirects with custom max depth
-- Bi-directional communication with web-socket services
-- [Native mocking support](http://wiki.github.com/igrigorik/em-http-request/mocking-httprequest) and through [Webmock](http://github.com/bblimke/webmock)
+- Streaming response processing
+- Streaming file uploads
+- HTTP proxy and SOCKS5 support
+- Basic Auth & OAuth
+- Connection-level & Global middleware support
+- Ryan Dahl's HTTP parser via [http_parser.rb](https://github.com/tmm1/http_parser.rb)
+- Works wherever EventMachine runs
 
-Getting started
----------------
+## Getting started
 
-      gem install em-http-request
-      irb:0> require 'em-http'
+    gem install em-http-request
 
-Or checkout [screencast / demo](http://everburning.com/news/eventmachine-screencast-em-http-request/) of using EM-HTTP-Request.
+- Introductory [screencast](http://everburning.com/news/eventmachine-screencast-em-http-request/)
+- [Issuing GET/POST/etc requests](https://github.com/igrigorik/em-http-request/wiki/Issuing-Requests)
+- [Issuing parallel requests with Multi interface](https://github.com/igrigorik/em-http-request/wiki/Parallel-Requests)
+- [Handling Redirects & Timeouts](https://github.com/igrigorik/em-http-request/wiki/Redirects-and-Timeouts)
+- [Keep-Alive and HTTP Pipelining](https://github.com/igrigorik/em-http-request/wiki/Keep-Alive-and-HTTP-Pipelining)
+- [Stream processing responses & uploads](https://github.com/igrigorik/em-http-request/wiki/Streaming)
+- [Issuing requests through HTTP & SOCKS5 proxies](https://github.com/igrigorik/em-http-request/wiki/Proxy)
+- [Basic Auth & OAuth](https://github.com/igrigorik/em-http-request/wiki/Basic-Auth-and-OAuth)
+- [GZIP & Deflate decoding](https://github.com/igrigorik/em-http-request/wiki/Compression)
+- [EM-HTTP Middleware](https://github.com/igrigorik/em-http-request/wiki/Middleware)
 
-Libraries & Applications using em-http
---------------------------------------
+## Extensions
 
-- [chirpstream](http://github.com/joshbuddy/chirpstream) - EM client for Twitters Chirpstream API
-- [RDaneel](http://github.com/hasmanydevelopers/RDaneel) - Ruby crawler which respects robots.txt
-- [rsolr-async](http://github.com/mwmitchell/rsolr-async) - An asynchronus connection adapter for RSolr
-- [PubSubHubbub](http://github.com/igrigorik/PubSubHubbub) - Asynchronous PubSubHubbub ruby client
-- [Firering](http://github.com/EmmanuelOga/firering) - Eventmachine powered Campfire API
-- and many others.. drop me a link if you want yours included!
-
-Simple client example
----------------------
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new('http://127.0.0.1/').get :query => {'keyname' => 'value'}, :timeout => 10
-
-        http.callback {
-          p http.response_header.status
-          p http.response_header
-          p http.response
-
-          EventMachine.stop
-        }
-      }
-
-Multi-request example
----------------------
-
-Fire and wait for multiple requests to complete via the MultiRequest interface.
-
-      EventMachine.run {
-        multi = EventMachine::MultiRequest.new
-
-        # add multiple requests to the multi-handler
-        multi.add(EventMachine::HttpRequest.new('http://www.google.com/').get)
-        multi.add(EventMachine::HttpRequest.new('http://www.yahoo.com/').get)
-
-        multi.callback  {
-          p multi.responses[:succeeded]
-          p multi.responses[:failed]
-
-          EventMachine.stop
-        }
-      }
-
-Basic-Auth example
-------------------
-
-Full basic author support. For OAuth, check examples/oauth-tweet.rb file.
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new('http://www.website.com/').get :head => {'authorization' => ['user', 'pass']}
-
-        http.errback { failed }
-        http.callback {
-          p http.response_header
-          EventMachine.stop
-        }
-      }
-
-
-POSTing data example
---------------------
+Several higher-order Ruby projects have incorporated em-http and other Ruby HTTP clients:
 
-For multi-part uploads, please see [this gist](https://gist.github.com/778639).
+- [EM-Synchrony](https://github.com/igrigorik/em-synchrony) - Collection of convenience classes and primitives to help untangle evented code (Ruby 1.9 + Fibers).
+- [Rack-Client](https://github.com/halorgium/rack-client) - Use Rack API for server, test, and client side. Supports Rack middleware!
+    - [Example in action](https://gist.github.com/802391)
+- [Faraday](https://github.com/technoweenie/faraday) - Modular HTTP client library using middleware heavily inspired by Rack.
+    - [Example in action](https://gist.github.com/802395)
 
-      EventMachine.run {
-        http1 = EventMachine::HttpRequest.new('http://www.website.com/').post :body => {"key1" => 1, "key2" => [2,3]}
-        http2 = EventMachine::HttpRequest.new('http://www.website.com/').post :body => "some data"
+## Testing
 
-        # ...
-      }
+- [WebMock](https://github.com/bblimke/webmock) - Library for stubbing and setting expectations on HTTP requests in Ruby.
+    - Example of [using WebMock, VCR & EM-HTTP](https://gist.github.com/802553)
 
-Streaming body processing
--------------------------
+## Other libraries & applications using EM-HTTP
 
-Allows you to consume an HTTP stream of content in real-time. Each time a new piece of content is pushed
-to the client, it is passed to the stream callback for you to operate on.
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new('http://www.website.com/').get
-        http.stream { |chunk| print chunk }
-      }
-
-Streaming files from disk
--------------------------
-Allows you to efficiently stream a (large) file from disk via EventMachine's FileStream interface.
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new('http://www.website.com/').post :file => 'largefile.txt'
-        http.callback { |chunk| puts "Upload finished!" }
-      }
-
-Proxy example
--------------
-Full transparent proxy support with support for SSL tunneling.
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new('http://www.website.com/').get :proxy => {
-          :host => 'www.myproxy.com',
-          :port => 8080,
-          :authorization => ['username', 'password'] # authorization is optional
-      }
-
-SOCKS5 Proxy example
--------------
-Tunnel your requests via connect via SOCKS5 proxies (ssh -D port somehost).
-
-    EventMachine.run {
-      http = EventMachine::HttpRequest.new('http://www.website.com/').get :proxy => {
-        :host => 'www.myproxy.com',
-        :port => 8080,
-        :type => :socks
-    }
-
-Auto-follow 3xx redirects
--------------------------
-
-Specify the max depth of redirects to follow, default is 0.
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new('http://www.google.com/').get :redirects => 1
-        http.callback { p http.last_effective_url }
-      }
-
-WebSocket example
------------------
-
-[Bi-directional communication with WebSockets](http://www.igvita.com/2009/12/22/ruby-websockets-tcp-for-the-browser/): simply pass in a ws:// resource and the client will negotiate the connection upgrade for you. On successful handshake the callback is invoked, and any incoming messages will be passed to the stream callback. The client can also send data to the server at will by calling the "send" method!
-
-      EventMachine.run {
-        http = EventMachine::HttpRequest.new("ws://yourservice.com/websocket").get :timeout => 0
-
-        http.errback { puts "oops" }
-        http.callback {
-          puts "WebSocket connected!"
-          http.send("Hello client")
-        }
-
-        http.stream { |msg|
-          puts "Recieved: #{msg}"
-          http.send "Pong: #{msg}"
-        }
-
-        http.disconnect { puts "oops, dropped connection?" }
-      }
+- [chirpstream](http://github.com/joshbuddy/chirpstream) - EM client for Twitters Chirpstream API
+- [rsolr-async](http://github.com/mwmitchell/rsolr-async) - An asynchronus connection adapter for RSolr
+- [PubSubHubbub](http://github.com/igrigorik/PubSubHubbub) - Asynchronous PubSubHubbub ruby client
+- [Firering](http://github.com/EmmanuelOga/firering) - Eventmachine powered Campfire API
+- [RDaneel](http://github.com/hasmanydevelopers/RDaneel) - Ruby crawler which respects robots.txt
+- and many others.. drop me a link if you want yours included!
 
-License
--------
+### License
 
-(MIT License) - Copyright (c) 2011 Ilya Grigorik
+(MIT License) - Copyright (c) 2011 Ilya Grigorik

data/Rakefile

@@ -1,80 +1,9 @@
 require 'bundler'
+Bundler::GemHelper.install_tasks
 
-Bundler.setup
-Bundler.require :default, :development
-
-require 'rake'
-require 'rake/clean'
-require 'rake/gempackagetask'
 require 'rspec/core/rake_task'
 
-require 'fileutils'
-include FileUtils
-
-# copied from EventMachine.
-MAKE = ENV['MAKE'] || if RUBY_PLATFORM =~ /mswin/ # mingw uses make.
-  'nmake'
-else
-  'make'
-end
-
-# Default Rake task is compile
-task :default => :compile
-
-# Rebuild parser Ragel
-task :ragel do
-  Dir.chdir "ext/http11_client" do
-    target = "http11_parser.c"
-    File.unlink target if File.exist? target
-    sh "ragel http11_parser.rl | rlgen-cd -G2 -o #{target}"
-    raise "Failed to build C source" unless File.exist? target
-  end
-end
-
 desc "Run all RSpec tests"
 RSpec::Core::RakeTask.new(:spec)
 
-def make(makedir)
-  Dir.chdir(makedir) { sh MAKE }
-end
-
-def extconf(dir)
-  Dir.chdir(dir) { ruby "extconf.rb" }
-end
-
-def setup_extension(dir, extension)
-  ext = "ext/#{dir}"
-  ext_so = "#{ext}/#{extension}.#{Config::MAKEFILE_CONFIG['DLEXT']}"
-  ext_files = FileList[
-    "#{ext}/*.c",
-    "#{ext}/*.h",
-    "#{ext}/extconf.rb",
-    "#{ext}/Makefile",
-    "lib"
-  ]
-
-  task "lib" do
-    directory "lib"
-  end
-
-  desc "Builds just the #{extension} extension"
-
-  mf = (extension + '_makefile').to_sym
-
-  task mf do |t|
-    extconf "#{ext}"
-  end
-
-  task extension.to_sym => [mf] do
-    make "#{ext}"
-    cp ext_so, "lib"
-  end
-end
-
-setup_extension("buffer", "em_buffer")
-setup_extension("http11_client", "http11_client")
-
-task :compile => [:em_buffer, :http11_client]
-
-CLEAN.include ['build/*', '**/*.o', '**/*.so', '**/*.a', '**/*.log', 'pkg']
-CLEAN.include ['ext/buffer/Makefile', 'lib/em_buffer.*', 'lib/http11_client.*']
+task :default => :spec

data/em-http-request.gemspec

@@ -5,6 +5,7 @@ require "em-http/version"
 Gem::Specification.new do |s|
   s.name        = "em-http-request"
   s.version     = EventMachine::HttpRequest::VERSION
+
   s.platform    = Gem::Platform::RUBY
   s.authors     = ["Ilya Grigorik"]
   s.email       = ["ilya@igvita.com"]
@@ -13,20 +14,19 @@ Gem::Specification.new do |s|
   s.description = s.summary
   s.rubyforge_project = "em-http-request"
 
-  s.add_dependency "eventmachine", ">= 0.12.9"
-  s.add_dependency "addressable", ">= 2.0.0"
-  s.add_dependency "escape_utils"
+  s.add_dependency "eventmachine"
+  s.add_dependency "addressable", ">= 2.2.3"
+  s.add_dependency "http_parser.rb", ">= 0.5.1"
+  s.add_dependency "em-socksify"
 
   s.add_development_dependency "rspec"
   s.add_development_dependency "rake"
-  s.add_development_dependency "em-websocket"
   s.add_development_dependency "rack"
+  s.add_development_dependency "yajl-ruby"
   s.add_development_dependency "mongrel", "~> 1.2.0.pre2"
 
-  s.extensions = ["ext/buffer/extconf.rb", "ext/http11_client/extconf.rb"]
-
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
-end
+end

data/examples/fetch.rb

@@ -1,30 +1,30 @@
-require 'rubygems'
-require 'eventmachine'
-require '../lib/em-http'
-
-urls = ARGV
-if urls.size < 1
-  puts "Usage: #{$0} <url> <url> <...>"
-  exit
-end
-
-pending = urls.size
-
-EM.run do
-  urls.each do |url|
-    http = EM::HttpRequest.new(url).get
-    http.callback {
-      puts "#{url}\n#{http.response_header.status} - #{http.response.length} bytes\n"
-      puts http.response
-
-      pending -= 1
-      EM.stop if pending < 1
-    }
-    http.errback {
-      puts "#{url}\n" + http.error
-
-      pending -= 1
-      EM.stop if pending < 1
-    }
-  end
-end
+require 'rubygems'
+require 'eventmachine'
+require '../lib/em-http'
+
+urls = ARGV
+if urls.size < 1
+  puts "Usage: #{$0} <url> <url> <...>"
+  exit
+end
+
+pending = urls.size
+
+EM.run do
+  urls.each do |url|
+    http = EM::HttpRequest.new(url).get
+    http.callback {
+      puts "#{url}\n#{http.response_header.status} - #{http.response.length} bytes\n"
+      puts http.response
+
+      pending -= 1
+      EM.stop if pending < 1
+    }
+    http.errback {
+      puts "#{url}\n" + http.error
+
+      pending -= 1
+      EM.stop if pending < 1
+    }
+  end
+end

data/examples/fibered-http.rb

@@ -1,39 +1,39 @@
-require 'eventmachine'
-require 'em-http'
-require 'fiber'
-
-# Using Fibers in Ruby 1.9 to simulate blocking IO / IO scheduling
-# while using the async EventMachine API's
-
-def async_fetch(url)
-  f = Fiber.current
-  http = EventMachine::HttpRequest.new(url).get :timeout => 10
-
-  http.callback { f.resume(http) }
-  http.errback  { f.resume(http) }
-
-  return Fiber.yield
-end
-
-EventMachine.run do
-  Fiber.new{
-
-    puts "Setting up HTTP request #1"
-    data = async_fetch('http://www.google.com/')
-    puts "Fetched page #1: #{data.response_header.status}"
-
-    puts "Setting up HTTP request #2"
-    data = async_fetch('http://www.yahoo.com/')
-    puts "Fetched page #2: #{data.response_header.status}"
-
-    EventMachine.stop
-  }.resume
-end
-
-puts "Done"
-
-#  Setting up HTTP request #1
-#  Fetched page #1: 302
-#  Setting up HTTP request #2
-#  Fetched page #2: 200
+require 'eventmachine'
+require 'em-http'
+require 'fiber'
+
+# Using Fibers in Ruby 1.9 to simulate blocking IO / IO scheduling
+# while using the async EventMachine API's
+
+def async_fetch(url)
+  f = Fiber.current
+  http = EventMachine::HttpRequest.new(url).get :timeout => 10
+
+  http.callback { f.resume(http) }
+  http.errback  { f.resume(http) }
+
+  return Fiber.yield
+end
+
+EventMachine.run do
+  Fiber.new{
+
+    puts "Setting up HTTP request #1"
+    data = async_fetch('http://www.google.com/')
+    puts "Fetched page #1: #{data.response_header.status}"
+
+    puts "Setting up HTTP request #2"
+    data = async_fetch('http://www.yahoo.com/')
+    puts "Fetched page #2: #{data.response_header.status}"
+
+    EventMachine.stop
+  }.resume
+end
+
+puts "Done"
+
+#  Setting up HTTP request #1
+#  Fetched page #1: 302
+#  Setting up HTTP request #2
+#  Fetched page #2: 200
 #  Done