pcap_tools 0.0.5 → 0.0.6

data/.gitignore

@@ -0,0 +1,3 @@
+*.gem
+*.pcap
+*.xml

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org/'
+
+gemspec
+

data/Gemfile.lock

@@ -0,0 +1,24 @@
+PATH
+  remote: .
+  specs:
+    pcap_tools (0.0.6)
+      nokogiri (= 1.6.0)
+      popen4 (= 0.1.2)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    Platform (0.4.0)
+    mini_portile (0.5.2)
+    nokogiri (1.6.0)
+      mini_portile (~> 0.5.0)
+    open4 (1.3.0)
+    popen4 (0.1.2)
+      Platform (>= 0.4.0)
+      open4 (>= 0.4.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  pcap_tools!

data/README.markdown

@@ -1,56 +1,107 @@
 # What is it ?
 
-It's a ruby library to help tcpdump file processing : do some offline analysis on tcpdump files.
+PCapTools is a ruby library to process pcap file from [wireshark](http://www.wireshark.org/) or [tcpdump](http://www.tcpdump.org/).
+
+PCapTools uses [tshark](http://www.wireshark.org/docs/man-pages/tshark.html) to process the pcap file, and run some analysis on it. Tshark is bundled with [wireshark](http://www.wireshark.org/download.html).
+
+There are two ways to use PCapTools
+
+* as a command line tools
+* as a ruby library
+
+## As a command line tools
 
 Main functionnalities :
 
 * Rebuild tcp streams
-* Extract and parse http request
+* Extract and parse http requests
+
+### Install
+
+Ensure `tshark` is installed. If not, check your wireshark install.
+
+    tshark -v
+
+Install pcap_tools
+
+    gem install pcap_tools
+
+### Command line options
+
+    pcap_tools --help
+
+    Usage: pcap_tools_http [options] pcap_files
+        --no-body                    Do not display body
+        --tshark_path                Path to tshark executable
+        --one-tcp-stream [index]     Display only one tcp stream
+        --mode [MODE]                Parsing mode : http, tcp, frame, tcp_count. Default http
+
+### Typical use
+
+    tcpdump -w out.pcap -s0 port 80
+    pcap_tools out.pcap
+
+
+## As a ruby library
 
-# How use it
+### Install
 
-## Make a tcpdump
+Ensure `tshark` is installed. If not, check your wireshark install.
 
-* `tcpdump -w out.pcap -s 4096 <filter>`
-* Get the output file out.pcap
+    tshark -v
 
-Please adjust the 4096 value, to the max packet size to capture.
+Declare dependency to `pcap_tools` in your Gemfile.
 
-## Write a ruby script
+    gem 'pcap_tools'
 
-    require 'pcap_tools'
+### How to use it
 
-    # Load tcpdump file
-    capture = PCAPRUB::Pcap.open_offline('out.pcap')
+The best example is the [pcap_tools command line script](https://github.com/bpaquet/pcap_tools/blob/master/bin/pcap_tools).
 
-## Available functions
+Pcap_tools is an event processor : `tshark` returns an XML Flow, which is parsed with a SAX processor. Each packet is processed by a TCP processor, which build streams. Each streams is processed by an HTTP processor, which rebuild HTTP request / response.
+
+#### Loading files
+
+    PcapTools::Loader::load_file(f, {:tshark => OPTIONS[:tshark_path]}) do |index, packet|
+    end
+
+Each packet is a ruby object containing the main attributes found in the packet, extracted with tshark.
+
+You have to use a packet processor to process this packet. The main one is `PcapTools::TcpProcessor`.
 
 ### Extract tcp streams
 
-This function rebuild tcp streams from an array of pcap capture object.
+    processor = PcapTools::TcpProcessor.new
+    PcapTools::Loader::load_file(f, {:tshark => OPTIONS[:tshark_path]}) do |index, packet|
+      processor.inject index, packet
+    end
 
-    tcp_streams = PcapTools::extract_tcp_streams(captures)
+The [TCPProcessor](https://github.com/bpaquet/pcap_tools/blob/master/lib/pcap_tools/packet_processors/tcp.rb) rebuild streams from IP raw packets. To use the streams, you have to add some streams processors.
 
-`tcp_streams` is an array of hash, each hash has tree keys :
+    processor.add_stream_processor PcapTools::TcpStreamRebuilder.new
 
-* `:type` : `:in` or `:out`, if the packet was sent or received
-* `:time` : timestamp of packet
-* `:data` : payload of packet
+The [TcpStreamRebuilder](https://github.com/bpaquet/pcap_tools/blob/master/lib/pcap_tools/stream_processors/http.rb) reassembles contiguous packet, for example the packets containing a big HTTP Response.
+
+    processor.add_stream_processor PcapTools::HttpExtractor.new
 
-Remarks :
+The [HttpExtractor](https://github.com/bpaquet/pcap_tools/blob/master/lib/pcap_tools/stream_processors/rebuilder.rb) build HTTP request and response from streams.
 
-* Packets are in the rigth ordere
-* Packets are not merged (eg an http response can be splitted on serval consecutive packets,
-with the same type `:in` or `:out`).
-To reassemble packet of the same type, please use `stream.rebuild_packets`
+Please read the code to build your own stream processor. Do not be afraid, it's easy :)
 
-### Extract http calls
+Note : the TCPProcessor is not able to rebuild tcp stream which do not start in the pcap file. For example, if you launch tcpdump after a long running Oracle DB connection, TCPProcessor will not show the Oracle DB connection.
 
-This function extract http calls from a tcp stream, returned from the `extract_tcp_streams` function.
+### Data format
 
-    http_calls = PcapTools::extract_http_calls(stream)
+Streams objects
 
-`http_calls` is an array of `http_call`.
+A `tcp_streams` is an array of hash, each hash has some keys :
+
+* `:type` : `:in` or `:out`, if the packet was sent or received
+* `:time` : timestamp of the packet.
+* `:data` : payload of packet.
+* `:size` : payload size.
+
+HTTP Objects
 
 A `http_call` is an array of two objects :
 
@@ -76,15 +127,3 @@ The request and response object have some new attributes
 For the response object body, the following "Content-Encoding" type are honored :
 
 * gzip
-
-### Extract http calls from captures
-
-The two in one : extract http calls from an array of captures objects
-
-    http_calls = PcapTools::extract_http_calls_from_captures(captures)
-
-### Load multiple files
-
-Load multiple pcap files, in time order. Useful when you use `tcpdump -C 5 -W 100000`, to split captured data into pieces of 5M
-
-    captures = PcapTools::load_mutliple_files '*pcap*'

data/bin/pcap_tools

@@ -3,7 +3,7 @@
 require 'pcap_tools'
 require 'optparse'
 
-options = {
+OPTIONS = {
   :mode => :http,
 }
 
@@ -11,60 +11,133 @@ OptionParser.new do |opts|
   opts.banner = "Usage: pcap_tools_http [options] pcap_files"
 
   opts.on("--no-body", "Do not display body") do
-    options[:no_body] = true
+    OPTIONS[:no_body] = true
   end
 
-  opts.on("--mode [MODE]", [:http, :tcp], "parsing mode") do |m|
-    options[:mode] = m
+  opts.on("--tshark_path", "Path to tshark executable") do |x|
+    OPTIONS[:tshark_path] = x
   end
 
-end.parse!
-
-data = ARGV.map{|f| puts "Loading #{f}"; PcapTools::Parser::load_file(f)}
+  opts.on("--one-tcp-stream [index]", Integer, "Display only one tcp stream") do |x|
+    OPTIONS[:one_tcp_stream] = x
+  end
 
-tcps = PcapTools::extract_tcp_streams(data)
+  opts.on("--mode [MODE]", [:http, :tcp, :frame, :tcp_count], "Parsing mode : http, tcp, frame, tcp_count. Default http") do |m|
+    OPTIONS[:mode] = m
+  end
 
-puts "Tcp streams extracted : #{tcps.size}"
-puts "Parsing mode : #{options[:mode]}"
-puts
+end.parse!
 
 def format_time t
   "#{t} #{t.nsec / 1000}"
 end
 
-if options[:mode] == :http
-  tcps.each do |tcp|
-    PcapTools::extract_http_calls(tcp).each do |req, resp|
-      puts ">>>> #{req["pcap-src"]}:#{req["pcap-src-port"]} > #{req["pcap-dst"]}:#{req["pcap-dst-port"]} #{format_time req.time}"
-      puts "#{req.method} #{req.path}"
-      req.each_capitalized_name.reject{|x| x =~ /^Pcap/ }.each do |x|
-        puts "#{x}: #{req[x]}"
+puts "Mode : #{OPTIONS[:mode]}"
+
+processor = nil
+
+if OPTIONS[:mode] == :frame
+
+  processor = PcapTools::FrameProcessor.new
+
+else
+
+  class TcpCounter
+
+    def initialize
+      @counter = 0
+    end
+
+    def process_stream stream
+      @counter += 1
+      stream
+    end
+
+    def finalize
+      puts "Number of TCP Streams : #{@counter}"
+    end
+
+  end
+
+
+  processor = PcapTools::TcpProcessor.new
+  processor.add_stream_processor TcpCounter.new
+  processor.add_stream_processor PcapTools::TcpOneStreamFilter.new OPTIONS[:one_tcp_stream]
+
+  if OPTIONS[:mode] == :tcp
+
+    class TcpPrinter
+
+      def process_stream stream
+        puts "<<<< new connection >>>> [ Wirershark stream index #{stream[:index]} ]"
+        stream[:data].each do |packet|
+          type = packet[:type] == :out ? ">>>>" : "<<<<<"
+          puts "#{type} #{packet[:from]}:#{packet[:from_port]} > #{packet[:to]}:#{packet[:to_port]}, size #{packet[:data].size}  #{format_time packet[:time]}"
+          puts packet[:data] unless OPTIONS[:no_body]
+          puts
+        end
+      end
+
+      def finalize
+      end
+
+    end
+
+    processor.add_stream_processor TcpPrinter.new
+
+  end
+
+  if OPTIONS[:mode] == :http
+
+    class HttpPrinter
+
+      def initialize
+        @counter = 0
       end
-      puts
-      puts req.body unless options[:no_body]
-      puts "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< #{format_time resp.time}"
-      if resp
-        puts "#{resp.code} #{resp.message}"
-        resp.each_capitalized_name.reject{|x| x =~ /^Pcap/ }.each do |x|
-          puts "#{x}: #{resp[x]}"
+
+      def process_stream stream
+        stream.each do |index, req, resp|
+          @counter += 1
+          puts ">>>> #{req["pcap-src"]}:#{req["pcap-src-port"]} > #{req["pcap-dst"]}:#{req["pcap-dst-port"]} #{format_time req.time} [ Wirershark stream index #{index} ]"
+          puts "#{req.method} #{req.path}"
+          req.each_capitalized_name.reject{|x| x =~ /^Pcap/ }.each do |x|
+            puts "#{x}: #{req[x]}"
+          end
+          puts
+          puts req.body unless OPTIONS[:no_body]
+          if resp
+            puts "<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< #{format_time resp.time}"
+            puts "#{resp.code} #{resp.message}"
+            resp.each_capitalized_name.reject{|x| x =~ /^Pcap/ }.each do |x|
+              puts "#{x}: #{resp[x]}"
+            end
+            puts
+            puts resp.body unless OPTIONS[:no_body]
+          else
+            puts "No response found"
+          end
+          puts
         end
-        puts
-        puts resp.body unless options[:no_body]
-      else
-        puts "No response in pcap file"
       end
-      puts
+
+      def finalize
+        puts "Number of HTTP Request / response : #{@counter}"
+      end
+
     end
+
+    processor.add_stream_processor PcapTools::TcpStreamRebuilder.new
+    processor.add_stream_processor PcapTools::HttpExtractor.new
+    processor.add_stream_processor HttpPrinter.new
+
   end
+
 end
 
-if options[:mode] == :tcp
-  tcps.each do |tcp|
-    tcp.each do |packet|
-      type = packet[:type] == :out ? ">>>>" : "<<<<<"
-      puts "#{type} #{packet[:from]}:#{packet[:from_port]} > #{packet[:to]}:#{packet[:to_port]}, size #{packet[:data].size}  #{format_time packet[:time]}"
-      puts packet[:data]
-      puts
-    end
+ARGV.each do |f|
+  PcapTools::Loader::load_file(f, {:tshark => OPTIONS[:tshark_path]}) do |index, packet|
+    processor.inject index, packet
   end
-end
+end
+
+processor.finalize

data/lib/pcap_tools.rb

@@ -1,189 +1,14 @@
-require 'rubygems'
 require 'net/http'
+require 'bindata'
 require 'zlib'
 
-require File.join(File.dirname(__FILE__), 'pcap_parser')
+require_relative 'pcap_tools/loader'
 
-module Net
+require_relative 'pcap_tools/patches/http.rb'
 
-  class HTTPRequest
-    attr_accessor :time
-  end
+require_relative 'pcap_tools/packet_processors/frame'
+require_relative 'pcap_tools/packet_processors/tcp'
 
-  class HTTPResponse
-    attr_accessor :time
-
-    def body= body
-      @body = body
-      @read = true
-    end
-
-  end
-
-end
-
-module PcapTools
-
-  class TcpStream < Array
-
-    def insert_tcp sym, packet
-      data = packet.payload
-      return if data.size == 0
-      timestamp = Time.at(packet.parent.parent.parent.ts_sec, packet.parent.parent.parent.ts_usec)
-      self << {:type => sym, :data => data, :from => packet.parent.src_addr, :to => packet.parent.dst_addr, :from_port => packet.src_port, :to_port => packet.dst_port, :time => timestamp}
-    end
-
-    def rebuild_packets
-      out = TcpStream.new
-      current = nil
-      self.each do |packet|
-        if current
-          if packet[:type] == current[:type]
-            current[:data] += packet[:data]
-          else
-            out << current
-            current = packet.clone
-          end
-        else
-          current = packet.clone
-        end
-      end
-      out << current if current
-      out
-    end
-
-  end
-
-  def extract_http_calls_from_captures captures
-    calls = []
-    extract_tcp_streams(captures).each do |tcp|
-      calls.concat(extract_http_calls(tcp))
-    end
-    calls
-  end
-
-  module_function :extract_http_calls_from_captures
-
-  def extract_tcp_streams captures
-    packets = []
-    captures.each do |capture|
-      capture.each do |packet|
-        packets << packet
-      end
-    end
-
-    streams = []
-    packets.each_with_index do |packet, k|
-      if packet.respond_to?(:type) && packet.type == "TCP" && packet.syn == 1 && packet.ack == 0
-        kk = k
-        tcp = TcpStream.new
-        while kk < packets.size
-          packet2 = packets[kk]
-          if packet2.respond_to?(:type) && packet.type == "TCP"
-            if packet.dst_port == packet2.dst_port && packet.src_port == packet2.src_port
-              tcp.insert_tcp :out, packet2
-              break if packet.fin == 1 || packet2.fin == 1
-            end
-            if packet.dst_port == packet2.src_port && packet.src_port == packet2.dst_port
-              tcp.insert_tcp :in, packet2
-              break if packet.fin == 1 || packet2.fin == 1
-            end
-          end
-          kk += 1
-        end
-        streams << tcp
-      end
-    end
-    streams
-  end
-
-  module_function :extract_tcp_streams
-
-  def extract_http_calls stream
-    rebuilded = stream.rebuild_packets
-    calls = []
-    data_out = ""
-    data_in = nil
-    k = 0
-    while k < rebuilded.size
-      begin
-        req = HttpParser::parse_request(rebuilded[k])
-        resp = k + 1 < rebuilded.size ? HttpParser::parse_response(rebuilded[k + 1]) : nil
-        calls << [req, resp]
-      rescue Exception => e
-        warn "Unable to parse http call : #{e}"
-      end
-      k += 2
-    end
-    calls
-  end
-
-  module_function :extract_http_calls
-
-  module HttpParser
-
-    def parse_request stream
-      headers, body = split_headers(stream[:data])
-      line0 = headers.shift
-      m = /(\S+)\s+(\S+)\s+(\S+)/.match(line0) or raise "Unable to parse first line of http request #{line0}"
-      clazz = {'POST' => Net::HTTP::Post, 'HEAD' => Net::HTTP::Head, 'GET' => Net::HTTP::Get, 'PUT' => Net::HTTP::Put}[m[1]] or raise "Unknown http request type #{m[1]}"
-      req = clazz.new m[2]
-      req['Pcap-Src'] = stream[:from]
-      req['Pcap-Src-Port'] = stream[:from_port]
-      req['Pcap-Dst'] = stream[:to]
-      req['Pcap-Dst-Port'] = stream[:to_port]
-      req.time = stream[:time]
-      req.body = body
-      req['user-agent'] = nil
-      req['accept'] = nil
-      add_headers req, headers
-      req.body.size == req['Content-Length'].to_i or raise "Wrong content-length for http request, header say #{req['Content-Length'].chomp}, found #{req.body.size}"
-      req
-    end
-
-    module_function :parse_request
-
-    def parse_response stream
-      headers, body = split_headers(stream[:data])
-      line0 = headers.shift
-      m = /^(\S+)\s+(\S+)\s+(.*)$/.match(line0) or raise "Unable to parse first line of http response #{line0}"
-      resp = Net::HTTPResponse.send(:response_class, m[2]).new(m[1], m[2], m[3])
-      resp.time = stream[:time]
-      add_headers resp, headers
-      if resp.chunked?
-        resp.body = read_chunked("\r\n" + body)
-      else
-        resp.body = body
-        resp.body.size == resp['Content-Length'].to_i or raise "Wrong content-length for http response, header say #{resp['Content-Length'].chomp}, found #{resp.body.size}"
-      end
-      resp.body = Zlib::GzipReader.new(StringIO.new(resp.body)).read if resp['Content-Encoding'] == 'gzip'
-      resp
-    end
-
-    module_function :parse_response
-
-    private
-
-    def self.add_headers o, headers
-      headers.each do |line|
-        m = /\A([^:]+):\s*/.match(line) or raise "Unable to parse line #{line}"
-        o[m[1]] = m.post_match
-      end
-    end
-
-    def self.split_headers str
-      index = str.index("\r\n\r\n")
-      return str[0 .. index].split("\r\n"), str[index + 4 .. -1]
-    end
-
-    def self.read_chunked str
-      return "" if str == "\r\n"
-      m = /\r\n([0-9a-fA-F]+)\r\n/.match(str) or raise "Unable to read chunked body in #{str.split("\r\n")[0]}"
-      len = m[1].hex
-      return "" if len == 0
-      m.post_match[0..len - 1] + read_chunked(m.post_match[len .. -1])
-    end
-
-  end
-
-end
+require_relative 'pcap_tools/stream_processors/one_stream_filter'
+require_relative 'pcap_tools/stream_processors/rebuilder'
+require_relative 'pcap_tools/stream_processors/http'

data/lib/pcap_tools/loader.rb

@@ -0,0 +1,118 @@
+require 'popen4'
+require 'ox'
+require 'fileutils'
+
+module PcapTools
+
+  module Loader
+
+    class MyParser < ::Ox::Sax
+
+      def initialize block
+        @current_packet_index = 0
+        @current_packet = nil
+        @current_processing = nil
+        @current_proto_name = nil
+        @current_field_name = nil
+        @block = block
+      end
+
+      def attr name, value
+        if @current_processing == :proto && name == :name
+          @current_proto_name = value
+          @current_packet[:protos] << value
+        elsif @current_processing == :field && name == :name
+          @current_field_name = value
+          # p @current_field_name
+        elsif name == :show
+          if @current_proto_name == "geninfo" && @current_field_name == "timestamp"
+            @current_packet[:time] = Time.parse value
+          elsif @current_proto_name == "ip" && @current_field_name == "ip.src"
+            @current_packet[:from] = value
+          elsif @current_proto_name == "ip" && @current_field_name == "ip.dst"
+            @current_packet[:to] = value
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.len"
+            @current_packet[:size] = value.to_i
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.stream"
+            @current_packet[:stream] = value.to_i
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.srcport"
+            @current_packet[:from_port] = value.to_i
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.dstport"
+            @current_packet[:to_port] = value.to_i
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.flags.fin"
+            @current_packet[:tcp_flags][:fin] = value == "1"
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.flags.reset"
+            @current_packet[:tcp_flags][:rst] = value == "1"
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.flags.ack"
+            @current_packet[:tcp_flags][:ack] = value == "1"
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.flags.syn"
+            @current_packet[:tcp_flags][:syn] = value == "1"
+          end
+        elsif name == :value
+          if @current_proto_name == "fake-field-wrapper" && @current_field_name == "data"
+            @current_packet[:data] = [value].pack("H*")
+          elsif @current_proto_name == "tcp" && @current_field_name == "tcp.segment_data"
+            @current_packet[:data] = [value].pack("H*")
+          end
+        end
+      end
+
+      def start_element name, attrs = []
+        case name
+          when :packet
+            @current_packet = {
+              :tcp_flags => {},
+              :packet_index => @current_packet_index + 1,
+              :protos => [],
+            }
+          when :proto
+            @current_processing = :proto
+          when :field
+            @current_processing = :field
+          when :pdml
+          else
+            raise "Unknown element [#{name}]"
+        end
+      end
+
+
+      def end_element name
+        if name == :packet
+          # p @current_packet
+          if @current_packet[:protos].include? "malformed"
+            $stderr.puts "Malformed packet #{@current_packet_index}"
+            return
+          end
+          raise "No data found in packet #{@current_packet_index}, protocols found #{@current_packet[:protos]}" if @current_packet[:data].nil? && @current_packet[:size] > 0
+          @current_packet.delete :protos
+          @block.call @current_packet_index, @current_packet
+          @current_packet_index += 1
+        end
+      end
+
+    end
+
+    def self.load_file f, options = {}, &block
+      tshark_executable = options[:tshark] || "tshark"
+      accepted_protocols = ["geninfo", "tcp", "ip", "eth", "sll", "frame"]
+      accepted_protocols += options[:accepted_protocols] if options[:accepted_protocols]
+      profile_name = "pcap_tools"
+      profile_dir = "#{ENV['HOME']}/.wireshark/profiles/#{profile_name}"
+      unless File.exist? "#{profile_dir}/disabled_protos"
+        status = POpen4::popen4("#{tshark_executable} -G protocols") do |stdout, stderr, stdin, pid|
+          list = stdout.read.split("\n").map { |x| x.split(" ").last }.reject { |x| accepted_protocols.include? x }
+          FileUtils.mkdir_p profile_dir
+          File.open("#{profile_dir}/disabled_protos", "w") { |io| io.write(list.join("\n") + "\n") }
+        end
+        raise "Tshark execution error when listing protocols" unless status.exitstatus == 0
+      end
+      status = POpen4::popen4("#{tshark_executable} -n -C #{profile_name} -T pdml -r #{f}") do |stdout, stderr, stdin, pid|
+        Ox.sax_parse(MyParser.new(block), stdout)
+        $stderr.puts stderr.read
+      end
+      raise "Tshark execution error with file #{f}" unless status.exitstatus == 0
+    end
+
+  end
+
+end

data/lib/pcap_tools/packet_processors/frame.rb

@@ -0,0 +1,19 @@
+module PcapTools
+
+  class FrameProcessor
+
+    def initialize
+      @counter = 0
+    end
+
+    def inject index, packet
+      @counter += 1
+    end
+
+    def finalize
+      puts "Number of frames : #{@counter}"
+    end
+
+  end
+
+end

data/lib/pcap_tools/packet_processors/tcp.rb

@@ -0,0 +1,51 @@
+require 'time'
+
+module PcapTools
+
+  class TcpProcessor
+
+    def initialize
+      @streams = {}
+      @stream_processors = []
+    end
+
+    def add_stream_processor processor
+      @stream_processors << processor
+    end
+
+    def inject index, packet
+      stream_index = packet[:stream]
+      if stream_index
+        if packet[:tcp_flags][:syn] && packet[:tcp_flags][:ack] === false
+          @streams[stream_index] = {
+            :first => packet,
+            :data => [],
+          }
+        elsif packet[:tcp_flags][:fin] || packet[:tcp_flags][:rst]
+          if @streams[stream_index]
+            current = {:index => stream_index, :data => @streams[stream_index][:data]}
+            @stream_processors.each do |p|
+              current = p.process_stream current
+              break unless current
+            end
+            @streams.delete stream_index
+          end
+        else
+          if @streams[stream_index]
+            packet[:type] = (packet[:from] == @streams[stream_index][:first][:from] && packet[:from_port] == @streams[stream_index][:first][:from_port]) ? :out : :in
+            packet.delete :tcp_flags
+            @streams[stream_index][:data] << packet if packet[:size] > 0
+          end
+        end
+      end
+    end
+
+    def finalize
+      @stream_processors.each do |p|
+        p.finalize
+      end
+    end
+
+  end
+
+end

data/lib/pcap_tools/patches/http.rb

@@ -0,0 +1,17 @@
+module Net
+
+  class HTTPRequest
+    attr_accessor :time
+  end
+
+  class HTTPResponse
+    attr_accessor :time
+
+    def body=(body)
+      @body = body
+      @read = true
+    end
+
+  end
+
+end

data/lib/pcap_tools/stream_processors/http.rb

@@ -0,0 +1,99 @@
+module PcapTools
+
+  class HttpExtractor
+
+    def process_stream stream
+      calls = []
+      k = 0
+      while k < stream[:data].size
+        begin
+          req = parse_request(stream[:data][k])
+          resp = k + 1 < stream[:data].size ? parse_response(stream[:data][k + 1]) : nil
+          calls << [stream[:index], req, resp]
+        rescue Exception => e
+          warn "Unable to parse http call in stream #{stream[:index]} : #{e}"
+        end
+        k += 2
+      end
+      calls
+    end
+
+    def finalize
+    end
+
+    private
+
+    def parse_request stream
+      headers, body = split_headers(stream[:data])
+      line0 = headers.shift
+      m = /(\S+)\s+(\S+)\s+(\S+)/.match(line0) or raise "Unable to parse first line of http request #{line0}"
+      clazz = {
+        'POST' => Net::HTTP::Post,
+        'HEAD' => Net::HTTP::Head,
+        'GET' => Net::HTTP::Get,
+        'PUT' => Net::HTTP::Put
+      }[m[1]] or raise "Unknown http request type [#{m[1]}]"
+      req = clazz.new m[2]
+      req['Pcap-Src'] = stream[:from]
+      req['Pcap-Src-Port'] = stream[:from_port]
+      req['Pcap-Dst'] = stream[:to]
+      req['Pcap-Dst-Port'] = stream[:to_port]
+      req.time = stream[:time]
+      req.body = body
+      req['user-agent'] = nil
+      req['accept'] = nil
+      add_headers req, headers
+      if req['Content-Length']
+        req.body.size == req['Content-Length'].to_i or raise "Wrong content-length for http request, header say [#{req['Content-Length'].chomp}], found #{req.body.size}"
+      end
+      req
+    end
+
+    def parse_response stream
+      headers, body = split_headers(stream[:data])
+      line0 = headers.shift
+      m = /^(\S+)\s+(\S+)\s+(.*)$/.match(line0) or raise "Unable to parse first line of http response [#{line0}]"
+      resp = Net::HTTPResponse.send(:response_class, m[2]).new(m[1], m[2], m[3])
+      resp.time = stream[:time]
+      add_headers resp, headers
+      if resp.chunked?
+        resp.body = read_chunked("\r\n" + body)
+      else
+        resp.body = body
+        if resp['Content-Length']
+          resp.body.size == resp['Content-Length'].to_i or raise "Wrong content-length for http response, header say [#{resp['Content-Length'].chomp}], found #{resp.body.size}"
+        end
+      end
+      begin
+        resp.body = Zlib::GzipReader.new(StringIO.new(resp.body)).read if resp['Content-Encoding'] == 'gzip'
+      rescue Zlib::GzipFile::Error
+        warn "Response body is not in gzip: [#{resp.body}]"
+      end
+      resp
+    end
+
+    def add_headers o, headers
+      headers.each do |line|
+        m = /\A([^:]+):\s*/.match(line) or raise "Unable to parse header line [#{line}]"
+        o[m[1]] = m.post_match
+      end
+    end
+
+    def split_headers str
+      index = str.index("\r\n\r\n")
+      return str[0 .. index].split("\r\n"), str[index + 4 .. -1]
+    end
+
+    def read_chunked str
+      if str.nil? || (str == "\r\n")
+        return ''
+      end
+      m = /\r\n([0-9a-fA-F]+)\r\n/.match(str) or raise "Unable to read chunked body in #{str.split("\r\n")[0]}"
+      len = m[1].hex
+      return '' if len == 0
+      m.post_match[0..len - 1] + read_chunked(m.post_match[len .. -1])
+    end
+
+  end
+
+end

data/lib/pcap_tools/stream_processors/one_stream_filter.rb

@@ -0,0 +1,19 @@
+module PcapTools
+
+  class TcpOneStreamFilter
+
+    def initialize target
+      @target = target
+    end
+
+    def process_stream stream
+      return nil if @target && stream[:index] != @target
+      stream
+    end
+
+    def finalize
+    end
+
+  end
+
+end

data/lib/pcap_tools/stream_processors/rebuilder.rb

@@ -0,0 +1,33 @@
+module PcapTools
+
+  class TcpStreamRebuilder
+
+    def process_stream stream
+      out = []
+      current = nil
+      stream[:data].each do |packet|
+        if current
+          if packet[:type] == current[:type]
+            current[:times] << {:offset => current[:size], :time => packet[:time]}
+            current[:data] += packet[:data]
+            current[:size] += packet[:size]
+          else
+            out << current
+            current = packet.clone
+            current[:times] = [{:offset => 0, :time => packet[:time]}]
+          end
+        else
+          current = packet.clone
+          current[:times] = [{:offset => 0, :time => packet[:time]}]
+        end
+      end
+      out << current if current
+      {:index => stream[:index], :data => out}
+    end
+
+    def finalize
+    end
+
+  end
+
+end

data/pcap_tools.gemspec

@@ -2,7 +2,7 @@ require 'rake'
 
 Gem::Specification.new do |s|
   s.name        = 'pcap_tools'
-  s.version     = '0.0.5'
+  s.version     = '0.0.6'
   s.authors     = ['Bertrand Paquet']
   s.email       = 'bertrand.paquet@gmail.com'
   s.summary     = 'Tools for extracting data from pcap files'
@@ -11,5 +11,6 @@ Gem::Specification.new do |s|
   s.files       = `git ls-files`.split($/)
   s.license     = 'BSD'
 
-  s.add_runtime_dependency('bindata', '>= 1.6.0')
+  s.add_dependency('popen4', '0.1.2')
+  s.add_dependency('ox', '2.0.11')
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pcap_tools
 version: !ruby/object:Gem::Version
-  version: 0.0.5
+  version: 0.0.6
   prerelease: 
 platform: ruby
 authors:
@@ -9,24 +9,40 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-11-22 00:00:00.000000000 Z
+date: 2013-12-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: bindata
+  name: popen4
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - '='
       - !ruby/object:Gem::Version
-        version: 1.6.0
+        version: 0.1.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - '='
       - !ruby/object:Gem::Version
-        version: 1.6.0
+        version: 0.1.2
+- !ruby/object:Gem::Dependency
+  name: ox
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.0.11
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 2.0.11
 description: 
 email: bertrand.paquet@gmail.com
 executables:
@@ -34,10 +50,19 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- .gitignore
+- Gemfile
+- Gemfile.lock
 - README.markdown
 - bin/pcap_tools
-- lib/pcap_parser.rb
 - lib/pcap_tools.rb
+- lib/pcap_tools/loader.rb
+- lib/pcap_tools/packet_processors/frame.rb
+- lib/pcap_tools/packet_processors/tcp.rb
+- lib/pcap_tools/patches/http.rb
+- lib/pcap_tools/stream_processors/http.rb
+- lib/pcap_tools/stream_processors/one_stream_filter.rb
+- lib/pcap_tools/stream_processors/rebuilder.rb
 - pcap_tools.gemspec
 homepage: https://github.com/bpaquet/pcap_tools
 licenses:
@@ -60,9 +85,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Tools for extracting data from pcap files
 test_files: []
-has_rdoc: 

data/lib/pcap_parser.rb

@@ -1,218 +0,0 @@
-require 'bindata'
-
-module PcapTools
-
-  module Parser
-
-    module HasParent
-
-      attr_accessor :parent
-
-    end
-
-    class PcapFile < BinData::Record
-      endian :little
-
-      struct :header do
-        uint32 :magic
-        uint16 :major
-        uint16 :minor
-        int32 :this_zone
-        uint32 :sig_figs
-        uint32 :snaplen
-        uint32 :linktype
-      end
-
-      array :packets, :read_until => :eof do
-        uint32 :ts_sec
-        uint32 :ts_usec
-        uint32 :incl_len
-        uint32 :orig_len
-        string :data, :length => :incl_len
-      end
-
-    end
-
-    # Present IP addresses in a human readable way
-    class IPAddr < BinData::Primitive
-      array :octets, :type => :uint8, :initial_length => 4
-
-      def set(val)
-        ints = val.split(/\./).collect { |int| int.to_i }
-        self.octets = ints
-      end
-
-      def get
-        self.octets.collect { |octet| "%d" % octet }.join(".")
-      end
-    end
-
-    # TCP Protocol Data Unit
-    class TCP_PDU < BinData::Record
-      mandatory_parameter :packet_length
-
-      endian :big
-
-      uint16 :src_port
-      uint16 :dst_port
-      uint32 :seq
-      uint32 :ack_seq
-      bit4 :doff
-      bit4 :res1
-      bit2 :res2
-      bit1 :urg
-      bit1 :ack
-      bit1 :psh
-      bit1 :rst
-      bit1 :syn
-      bit1 :fin
-      uint16 :window
-      uint16 :checksum
-      uint16 :urg_ptr
-      string :options, :read_length => :options_length_in_bytes
-      string :payload, :read_length => lambda { packet_length - payload.rel_offset }
-
-      def options_length_in_bytes
-        (doff - 5 ) * 4
-      end
-
-      def type
-        "TCP"
-      end
-
-      include HasParent
-
-    end
-
-    # UDP Protocol Data Unit
-    class UDP_PDU < BinData::Record
-      mandatory_parameter :packet_length
-
-      endian :big
-
-      uint16 :src_port
-      uint16 :dst_port
-      uint16 :len
-      uint16 :checksum
-      string :payload, :read_length => lambda { packet_length - payload.rel_offset }
-
-      def type
-        "UDP"
-      end
-
-      include HasParent
-    end
-
-    # IP Protocol Data Unit
-    class IP_PDU < BinData::Record
-      endian :big
-
-      bit4 :version, :asserted_value => 4
-      bit4 :header_length
-      uint8 :tos
-      uint16 :total_length
-      uint16 :ident
-      bit3 :flags
-      bit13 :frag_offset
-      uint8 :ttl
-      uint8 :protocol
-      uint16 :checksum
-      ip_addr :src_addr
-      ip_addr :dst_addr
-      string :options, :read_length => :options_length_in_bytes
-      choice :payload, :selection => :protocol do
-        tcp_pdu 6, :packet_length => :payload_length_in_bytes
-        udp_pdu 17, :packet_length => :payload_length_in_bytes
-        string :default, :read_length => :payload_length_in_bytes
-      end
-
-      def header_length_in_bytes
-        header_length * 4
-      end
-
-      def options_length_in_bytes
-        header_length_in_bytes - options.rel_offset
-      end
-
-      def payload_length_in_bytes
-        total_length - header_length_in_bytes
-      end
-
-      def type
-        "IP"
-      end
-
-      include HasParent
-    end
-
-    class MacAddr < BinData::Primitive
-      array :octets, :type => :uint8, :initial_length => 6
-
-      def set(val)
-        ints = val.split(/\./).collect { |int| int.to_i }
-        self.octets = ints
-      end
-
-      def get
-        self.octets.collect { |octet| "%02x" % octet }.join(":")
-      end
-    end
-
-    IPV4 = 0x0800
-    class Ethernet < BinData::Record
-      endian :big
-
-      mac_addr :dst
-      mac_addr :src
-      uint16 :protocol
-      choice :payload, :selection => :protocol do
-        ip_pdu IPV4
-        rest :default
-      end
-
-      include HasParent
-    end
-
-    class LinuxCookedCapture < BinData::Record
-      endian :big
-
-      uint16 :type
-      uint16 :address_type
-      uint16 :address_len
-      array :octets, :type => :uint8, :initial_length => 8
-      uint16 :protocol
-      choice  :payload, :selection => :protocol do
-        ip_pdu IPV4
-        rest :default
-      end
-
-      include HasParent
-    end
-
-    def load_file f
-      packets = []
-      File.open(f, 'rb') do |io|
-        content = PcapFile.read(io)
-        raise 'Wrong endianess' unless content.header.magic.to_i.to_s(16) == "a1b2c3d4"
-        content.packets.each do |original_packet|
-          packet = case content.header.linktype
-          when 113 then LinuxCookedCapture.read(original_packet.data)
-          when 1 then Ethernet.read(original_packet.data)
-          else raise "Unknown network #{content.header.linktype}"
-          end
-          packet.parent = original_packet
-          while packet.respond_to?(:payload) && packet.payload.is_a?(BinData::Choice)
-            packet.payload.parent = packet
-            packet = packet.payload
-          end
-          packets << packet
-        end
-      end
-      packets
-    end
-
-    module_function :load_file
-
-  end
-
-end