dorothy2 0.0.3 → 1.0.0

data/README.md

@@ -13,7 +13,7 @@ Dorothy2 is a continuation of my Bachelor degree's final project ([Dorothy: insi
 The main framework's structure remained almost the same, and it has been fully detailed in my degree's final project or in this short [paper](http://www.honeynet.it/wp-content/uploads/Dorothy/EC2ND-Dorothy.pdf). More information about the whole project can be found on the Italian Honeyproject [website](http://www.honeynet.it).
 
 
-The framework is manly composed by four big elements that can be even executed separately:
+The framework is mainly composed by four big elements that can be even executed separately:
 
 * The Dorothy analysis engine (included in this gem)
 
@@ -33,7 +33,7 @@ The framework is manly composed by four big elements that can be even executed s
 
 The first three modules are (or will be soon) publicly released under GPL 2/3 license as tribute to the the [Honeynet Project Alliance](http://www.honeynet.org).
 All the information generated by the framework - i.e. binary info, timestamps, dissected network analysis - are stored into a postgres DB (Dorothive) in order to be used for further analysis.
-A no-SQL database (CouchDB) is also used to mass strore all the traffic dumps thanks to the [pcapr/xtractr](https://code.google.com/p/pcapr/wiki/Xtractr) technology.
+A no-SQL database (CouchDB) is also used to mass store all the traffic dumps thanks to the [pcapr/xtractr](https://code.google.com/p/pcapr/wiki/Xtractr) technology.
 
 I started to code this project in late 2009 while learning Ruby at the same time. Since then, I´ve been changing/improving it as long as my Ruby coding skills were improving. Because of that, you may find some parts of code not-really-tidy :)
 
@@ -55,6 +55,11 @@ Dorothy needs the following software (not expressly in the same host) in order t
 * [pcapr-local](https://github.com/mudynamics/pcapr-local )  (only used by doroParser)
 * MaxMind libraries (only used by doroParser)
 
+Regarding the Operating System
+
+* Dorothy has been designed to run on any *nix system. So far it was successfully tested on OSX and Linux.
+* The virtual machines used as sandboxes are meant to be Windows based (successfully tested on XP)
+* Only pcapr-local strictly requires Linux, if you want to use a Mac for executing this gem (like I do), install it into the NAM (as this guide suggests)
 
 ## Installation
 
@@ -116,7 +121,7 @@ It is recommended to follow this step2step process:
 
         #gem install pcapr-local
 
-* Start pcapr-local by using the dorothy's account and configure it. When prompted, insert the folder path used to store the network dumps
+* Start pcapr-local by using the dorothy's system account and configure it. When prompted, insert the folder path used to store the network dumps
 
         $startpcapr
         ....
@@ -164,7 +169,7 @@ or
 2. Configure a dedicated postgres user for Dorothy (or use the default postgres user instead, up to you :)
 
 > Note:
-> If you want to use Postgres "as is", and then configure Dorothy to use "postgres" degault the user, configure a password for this user at least (by default it comes with no password)
+> If you want to use Postgres "as is", and then configure Dorothy to use "postgres" default the user, configure a password for this user at least (by default it comes with no password)
 
 3. Install the following packages
 
@@ -222,8 +227,8 @@ The first time you execute Dorothy, it will ask you to fill those information in
       --infoflow, -i:   Print the analysis flow
     --source, -s <s>:   Choose a source (from the ones defined in etc/sources.yml)
         --daemon, -d:   Stay in the background, by constantly pooling datasources
- --SandboxUpdate, -S:   Update Dorothive with the new Sandbox file
- --DorothiveInit, -D:   (RE)Install the Dorothy Database (Dorothive)
+  --SandboxUpdate, -S:   Update Dorothive with the new Sandbox file
+  --DorothiveInit, -D:   (RE)Install the Dorothy Database (Dorothive)
           --help, -h:   Show this message
 
 
@@ -231,6 +236,10 @@ The first time you execute Dorothy, it will ask you to fill those information in
 >
      $dorothy_start -v -s malwarefolder
 
+After the execution, if everything went fine, you will find the analysis output (screens/pcap/bin) into the analysis folder that you have configured e.g. dorothy/opt/analyzed/[:digit:]/
+Other information will be stored into Dorothive.
+If executed in daemon mode, Dorothy2 will poll the datasources every X seconds (where X is defined by the "dtimeout:" field in the configuration file) looking for new binaries.
+
 ### DoroParser usage:
 
     $dparser_start [options]
@@ -245,6 +254,9 @@ The first time you execute Dorothy, it will ask you to fill those information in
      $dparser_start -d start
      $dparser_stop
 
+
+After the execution, if everything went fine, doroParser will store all the donwloaded files into the binary's analysis folder e.g. dorothy/opt/analyzed/[:digit:]/downloads
+Other information -i.e. Network data- will be stored into Dorothive.
 If executed in daemon mode, DoroParser will poll the database every X seconds (where X is defined by the "dtimeout:" field in the configuration file) looking for new pcaps that has been inserted.
 
 ###6. Debugging problems
@@ -268,6 +280,18 @@ Below there are some tips about how understand the root-cause of your crash.
 
 ------------------------------------------
 
+## Acknowledgements
+
+Thanks to all the people who have contributed in making the Dorothy2 project up&running:
+
+* Marco C. (research)
+* Davide C. (Dorothive)
+* Andrea V. (WGUI)
+* Domenico C. - Patrizia P. (Dorothive/JDrone)
+* [All](https://www.honeynet.it/research) the graduating students from [UniMI](http://cdlonline.di.unimi.it/) who have contributed.
+* Sabrina P. (our students "headhunter" :)
+* Jorge C. and Nelson M. (betatesting/first release feedbacks)
+
 ## Contributing
 
 1. Fork it

data/TODO

@@ -0,0 +1,21 @@
+##############
+#DOROTHY-TODO#
+##############
+
+-PORT TO Ruby 2.0
+-WGUI
+
+-BINARY STATIC ANALYSIS
+-ANALYZE SYSTEM CHANGES
+-SYSTEM ANALYSIS -VMWARE API: QueryChangedDiskAreas
+-LIST PROCESSES-> pm.ListProcessesInGuest(:vm => vm, :auth => auth).inspect
+
+-CODE- CATCH CTRL-C AND EXIT GRACEFULLY
+-INTERACTIVE CONSOLE FOR NETWORK ANALYSIS
+
+-REVIEW DOROTHIVE (binary fullpath?)
+
+-ADD EMAIL AS SOURCETYPE (use ruby mail gem for retreiving the emails, and parse them)
+
+-REPORT PLUGIN
+ -REPORT - MAEC

data/bin/dorothy_start

@@ -37,7 +37,7 @@ opts = Trollop.options do
   opt :source, "Choose a source (from the ones defined in etc/sources.yml)", :type => :string
   opt :daemon, "Stay in the backround, by constantly pooling datasources"
   opt :SandboxUpdate, "Update Dorothive with the new Sandbox file"
-  opt :DorothiveInit, "(RE)Install the Dorothy Database (Dorothive)"
+  opt :DorothiveInit, "(RE)Install the Dorothy Database (Dorothive)", :type => :string
 
 end
 
@@ -103,6 +103,12 @@ end
 sfile = home + '/etc/sources.yml'
 sboxfile = home + '/etc/sandboxes.yml'
 
+if opts[:DorothiveInit]
+  Util.init_db(opts[:DorothiveInit])
+  puts "[Dorothy]".yellow + " Database loaded, now you can restart Dorothy!"
+  exit(0)
+end
+
 #INIT DB Connector
 begin
   db = Insertdb.new
@@ -120,11 +126,6 @@ rescue => e
 end
 
 
-if opts[:DorothiveInit]
-  Util.init_db
-  exit(0)
-end
-
 if opts[:SandboxUpdate]
   puts "[Dorothy]".yellow + " Loading #{sboxfile} into Dorothive"
   DoroConfig.init_sandbox(sboxfile)

data/bin/dparser_start

@@ -63,11 +63,23 @@ LOGGER_PARSER.sev_threshold = DoroSettings.env[:loglevel]
 LOGGER = DoroLogger.new(logout, DoroSettings.env[:logage])
 LOGGER.sev_threshold = DoroSettings.env[:loglevel]
 
+begin
+
+rescue
+  exit(1)
+
+end
+
+
+
 begin
   DoroParser.start(daemon)
 rescue => e
   puts "[PARSER]".yellow + " An error occurred: ".red + $!
-  puts "[PARSER]".yellow + " For more information check the logfile" + $! if daemon
+  if daemon
+  puts "[PARSER]".yellow + " For more information check the logfile" + $!
+  puts "[PARSER]".yellow + "Dorothy-Parser has been stopped"
+  end
   LOGGER_PARSER.error "Parser", "An error occurred: " + $!
   LOGGER_PARSER.debug "Parser", "#{e.inspect} --BACKTRACE: #{e.backtrace}"
   LOGGER_PARSER.info "Parser", "Dorothy-Parser has been stopped"

data/etc/ddl/dorothive.ddl

@@ -128,26 +128,6 @@ CREATE TYPE layer7_protocols AS ENUM (
 
 ALTER TYPE dorothy.layer7_protocols OWNER TO postgres;
 
---
--- Name: sample_type; Type: TYPE; Schema: dorothy; Owner: postgres
---
-
-CREATE TYPE sample_type AS ENUM (
-    'mz',
-    'pe',
-    'elf'
-);
-
-
-ALTER TYPE dorothy.sample_type OWNER TO postgres;
-
---
--- Name: TYPE sample_type; Type: COMMENT; Schema: dorothy; Owner: postgres
---
-
-COMMENT ON TYPE sample_type IS 'Sample file type';
-
-
 --
 -- Name: sanbox_type; Type: TYPE; Schema: dorothy; Owner: postgres
 --
@@ -245,7 +225,6 @@ SELECT pg_catalog.setval('analyses_id_seq', 1, true);
 CREATE TABLE samples (
     hash character(64) NOT NULL,
     size integer NOT NULL,
-    type sample_type,
     path character(256),
     filename character(256),
     md5 character(64),
@@ -276,14 +255,6 @@ COMMENT ON COLUMN samples.hash IS 'SHA256 checksum hash';
 
 COMMENT ON COLUMN samples.size IS 'Sample size';
 
-
---
--- Name: COLUMN samples.type; Type: COMMENT; Schema: dorothy; Owner: postgres
---
-
-COMMENT ON COLUMN samples.type IS 'Sample type';
-
-
 --
 -- Name: CONSTRAINT size_notneg ON samples; Type: COMMENT; Schema: dorothy; Owner: postgres
 --
@@ -298,7 +269,7 @@ COMMENT ON CONSTRAINT size_notneg ON samples IS 'Sample size must not be negativ
 CREATE TABLE traffic_dumps (
     hash character(64) NOT NULL,
     size integer NOT NULL,
-    pcapr_id character(64),
+    pcapr_id character(32),
     "binary" character varying,
     parsed boolean
 );
@@ -1323,7 +1294,7 @@ COPY roles (id, type, comment) FROM stdin;
 -- Data for Name: samples; Type: TABLE DATA; Schema: dorothy; Owner: postgres
 --
 
-COPY samples (hash, size, type, path, filename, md5, long_type) FROM stdin;
+COPY samples (hash, size, path, filename, md5, long_type) FROM stdin;
 \.
 
 

data/lib/doroParser.rb

@@ -17,7 +17,6 @@
 
 
 require 'rubygems'
-require 'mu/xtractr'
 require 'md5'
 require 'rbvmomi'
 require 'rest_client'
@@ -32,15 +31,17 @@ require 'pg'
 require 'iconv'
 require 'tmail'
 require 'ipaddr'
+require 'net/http'
+require 'json'
 
 require File.dirname(__FILE__) + '/dorothy2/environment'
+require File.dirname(__FILE__) + '/mu/xtractr'
 require File.dirname(__FILE__) + '/dorothy2/DEM'
 require File.dirname(__FILE__) + '/dorothy2/do-utils'
 require File.dirname(__FILE__) + '/dorothy2/do-logger'
 require File.dirname(__FILE__) + '/dorothy2/deep_symbolize'
 
 
-
 module DoroParser
 #Host roles 
 
@@ -85,12 +86,34 @@ module DoroParser
 
 
       begin
-        xtractr = Doroxtractr.create "http://#{DoroSettings.pcapr[:host]}:#{DoroSettings.pcapr[:port]}/pcaps/1/pcap/#{dump['pcapr_id'].gsub(/\s+/, "")}"
+
+        #check if the pcap has been correctly indexed by pcapr
+        xtractr = Doroxtractr.create "http://#{DoroSettings.pcapr[:host]}:#{DoroSettings.pcapr[:port]}/pcaps/1/pcap/#{dump['pcapr_id'].rstrip}"
 
       rescue => e
-        LOGGER_PARSER.fatal "PARSER", "Can't create a XTRACTR instance, try with nextone"
+        LOGGER_PARSER.fatal "PARSER", "Can't connect to the PCAPR server."
         LOGGER_PARSER.debug "PARSER", "#{$!}"
-        LOGGER_PARSER.debug "PARSER", e
+        LOGGER_PARSER.debug "PARSER", e.backtrace if VERBOSE
+        return false
+      end
+
+      #it may happen that Pcapr has created an instance, but it is still indexing the pcap.
+      #The following section is to avoid a crash while quering such (still-empty instance)
+      #In addition, an added check is inserted, to see if the pcapr instance really match the pcap filename
+      begin
+      pcapr_query = URI.parse "http://#{DoroSettings.pcapr[:host]}:#{DoroSettings.pcapr[:port]}/pcaps/1/about/#{dump['pcapr_id'].rstrip}"
+      pcapr_response = Net::HTTP.get_response(pcapr_query)
+      pcapname = File.basename(JSON.parse(pcapr_response.body)["filename"], ".pcap")
+
+      t ||= $1 if pcapname =~ /[0-9]*\-(.*)$/
+      raise NameError.new if  t != dump['sample'].rstrip
+
+      rescue NameError
+        LOGGER_PARSER.error "PARSER", "The pcapr filename mismatchs the one present in Dorothive!. Skipping."
+        next
+
+      rescue
+        LOGGER_PARSER.error "PARSER", "Can't find the PCAP into Pcapr, maybe it has not been indexed yet. Skipping."
         next
       end
 
@@ -103,18 +126,14 @@ module DoroParser
 
           flowdeep = xtractr.flows("flow.id:#{flow.id}")
 
-
-
           #Skipping if NETBIOS spreading activity:
           if flow.dport == 135 or flow.dport == 445
             LOGGER_PARSER.info "PARSER", "Netbios connections, skipping flow" unless NONETBIOS
             next
           end
 
-
           title = flow.title[0..200].gsub(/'/,"") #xtool bug ->')
 
-
           #insert hosts (geo) info into db
           #TODO: check if is a localaddress
           localip = xtractr.flows.first.src.address
@@ -258,10 +277,6 @@ module DoroParser
 
                     end
 
-
-
-
-
                   end
 
                 #case MAIL
@@ -269,8 +284,6 @@ module DoroParser
                   LOGGER_PARSER.info "SMTP", "FOUND an SMTP request..".white
                   #insert mail
                   #by from to subject data id time connection
-
-
                   streamdata.each do |m|
                     mailfrom = 'null'
                     mailto = 'null'
@@ -303,8 +316,6 @@ module DoroParser
                     @insertdb.insert("emails", mailvalues )
                   end
 
-
-
                 #case FTP
                 when "FTP" then
                   LOGGER_PARSER.info "FTP", "FOUND an FTP request".white
@@ -324,9 +335,7 @@ module DoroParser
                     end
                   end
 
-
                 else
-
                   LOGGER_PARSER.info "PARSER", "Unknown traffic, try see if it is IRC traffic"
 
                   if Parser.guess(streamdata.inspect).class.inspect =~ /IRC/
@@ -358,14 +367,12 @@ module DoroParser
                     end
                   end
 
-
                   @p.each do |d|
 
                     begin
 
                       dns = DoroDNS.new(d)
 
-
                       dnsvalues = ["default", dns.name, dns.cls_i.inspect, dns.qry?, dns.ttl, flowid, dns.address.to_s, dns.data, dns.type_i.inspect]
 
                       LOGGER_PARSER.debug "DB", " Inserting DNS data from #{flow.dst.address.to_s}".blue if VERBOSE
@@ -404,8 +411,8 @@ module DoroParser
 
         rescue => e
 
-          LOGGER_PARSER.error "PARSER", "Error while analyzing flow #{flow.id}"
-          LOGGER_PARSER.debug "PARSER", "#{e.inspect} BACKTRACE: #{e.backtrace}"
+          LOGGER_PARSER.error "PARSER", "Error while analyzing flow #{flow.id}: #{e.inspect}"
+          LOGGER_PARSER.debug "PARSER", "#{e.backtrace}" if VERBOSE
           LOGGER_PARSER.info "PARSER", "Flow #{flow.id} will be skipped"
           next
         end

data/lib/dorothy2/BFM.rb

@@ -7,15 +7,15 @@
 ###BINARY FETCHER MODULE###
 ###					          	###
 ###########################
-
+#The BFM module is in charge of retreiving the binary from the sources configured in the sources.yml file.
+#It receive the source hash, and return the downloaded binaries objects.
 module Dorothy
 
-
   class DorothyFetcher
     attr_reader :bins
 
-
-    def initialize(source)  #source struct: Hash, {:dir => "#{HOME}/bins/honeypot", :typeid=> 0 ..}
+    #Source struct: Hash, {:dir => "#{HOME}/bins/honeypot", :typeid=> 0 ..}
+    def initialize(source)
       ndownloaded = 0
 
       @bins = []
@@ -26,7 +26,6 @@ module Dorothy
         when "ssh" then
           LOGGER.info "BFM", " Fetching trojan from > Honeypot"
           #file = "/opt/dionaea/var/dionaea/binaries/"
-
           #puts "Start to download malware"
 
           files = []
@@ -37,7 +36,6 @@ module Dorothy
                 unless files.include? "#{source["localdir"]}/" + File.basename(name)
                   ndownloaded += 1
                   files.push "#{source["localdir"]}/" + File.basename(name)
-                  #			puts ""
                 end
                 #		print "#{File.basename(name)}: #{sent}/#{total}\r"
                 #		$stdout.flush
@@ -45,26 +43,22 @@ module Dorothy
               LOGGER.info "BFM", "#{ndownloaded} files downloaded"
             end
 
-
           rescue => e
             LOGGER.error "BFM", "An error occurred while downloading malwares from honeypot sensor: " + $!
             LOGGER.error "BFM", "Error: #{$!}, #{e.inspect}, #{e.backtrace}"
           end
 
           #DIRTY WORKAROUND for scp-ing only files without directory
-
           FileUtils.mv(Dir.glob(source["localdir"] + "/binaries/*"), source["localdir"])
           Dir.rmdir(source["localdir"] + "/binaries")
 
 
           begin
-
             unless DoroSettings.env[:testmode]
               Net::SSH.start(source["ip"], source["user"], :password => source["pass"], :port => source["port"]) do |ssh|
                 ssh.exec "mv #{source["remotedir"]}/* #{source["remotedir"]}/../analyzed "
               end
             end
-
           rescue
             LOGGER.error "BFM", "An error occurred while erasing parsed malwares in the honeypot sensor: " + $!
           end
@@ -87,8 +81,6 @@ module Dorothy
       end
     end
 
-
-
     private
     def load_malw(f, typeid, sourceinfo = nil)
 
@@ -100,7 +92,7 @@ module Dorothy
         return false
       end
 
-      samplevalues = [bin.sha, bin.size, bin.dbtype, bin.dir_bin, filename, bin.md5, bin.type ]
+      samplevalues = [bin.sha, bin.size, bin.dir_bin, filename, bin.md5, bin.type ]
       sighvalues = [bin.sha, typeid, bin.ctime, "null"]
 
       begin

data/lib/dorothy2/do-utils.rb

@@ -16,16 +16,16 @@ module Dorothy
       File.exist?(file)
     end
 
-    def init_db(force=false)
-      LOGGER.warn "DB", "The database is going to be initialized, all the data present will be lost. Continue?(write yes)"
+    def init_db(ddl=DoroSettings.dorothive[:ddl], force=false)
+      LOGGER.warn "DB", "The database is going to be initialized with the file #{ddl}. If the Dorothive is already present, " + "all the its data will be lost".red + ". Continue?(write yes)"
       answ = "yes"
       answ = gets.chop unless force
 
       if answ == "yes"
         begin
           #ugly, I know, but couldn't find a better and easier way..
-          raise 'An error occurred' unless system "psql -h #{DoroSettings.dorothive[:dbhost]} -U #{DoroSettings.dorothive[:dbuser]} -f #{DoroSettings.dorothive[:ddl]}"
-          LOGGER.info "DB", "Database correctly initialized."
+          raise 'An error occurred' unless system "psql -h #{DoroSettings.dorothive[:dbhost]} -U #{DoroSettings.dorothive[:dbuser]} -f #{ddl} 1> /dev/null"
+          LOGGER.info "DB", "Database correctly initialized. Now you can restart Dorothy!"
         rescue => e
           LOGGER.error "DB", $!
           LOGGER.debug "DB", e.inspect
@@ -248,7 +248,6 @@ module Dorothy
       @binpath = file
       @filename = File.basename file
       @extension = File.extname file
-      @dbtype = "null"  #TODO: remove type column in sample table
 
       File.open(file, 'rb') do |fh1|
         while buffer1 = fh1.read(1024)

data/lib/dorothy2/version.rb

@@ -1,3 +1,3 @@
 module Dorothy2
-  VERSION = "0.0.3"
+  VERSION = "1.0.0"
 end

data/lib/dorothy2.rb

@@ -5,7 +5,6 @@
 ##for irb debug:
 ##from $home, irb and :
 ##load 'lib/dorothy2.rb'; include Dorothy; LOGGER = DoroLogger.new(STDOUT, "weekly"); DoroSettings.load!('etc/dorothy.yml')
-#$LOAD_PATH.unshift '/opt/local/lib/ruby/gems/1.8/gems/ruby-filemagic-0.4.2/lib'
 
 require 'net/ssh'
 require 'net/scp'
@@ -152,8 +151,8 @@ module Dorothy
       vsm.copy_file("#{bin.md5}#{bin.extension}",filecontent)
 
       #Start Sniffer
-      dumpname = bin.md5
-      pid = @nam.start_sniffer(guestvm[2],DoroSettings.nam[:interface], dumpname, DoroSettings.nam[:pcaphome]) #dumpname = vmfile.pcap
+      dumpname = anal_id.to_s + "-" + bin.md5
+      pid = @nam.start_sniffer(guestvm[2],DoroSettings.nam[:interface], dumpname, DoroSettings.nam[:pcaphome])
       LOGGER.info "NAM","VM#{guestvm[0]} ".yellow + "Start sniffing module"
       LOGGER.debug "NAM","VM#{guestvm[0]} ".yellow + "Tcpdump instance #{pid} started" if VERBOSE
 
@@ -216,8 +215,7 @@ module Dorothy
 
       #Downloading PCAP
       LOGGER.info "NAM", "VM#{guestvm[0]} ".yellow + "Downloading #{dumpname}.pcap to #{bin.dir_pcap}"
-      #t = DoroSettings.nam[:pcaphome] + "/" + dumpname + ".pcap"
-      Ssh.download(DoroSettings.nam[:host], DoroSettings.nam[:user],DoroSettings.nam[:pass], DoroSettings.nam[:pcaphome] + "/" + dumpname + ".pcap", bin.dir_pcap)
+      Ssh.download(DoroSettings.nam[:host], DoroSettings.nam[:user],DoroSettings.nam[:pass], DoroSettings.nam[:pcaphome] + "/#{dumpname}.pcap", bin.dir_pcap)
 
       #Downloading Screenshots from esx
       LOGGER.info "NAM", "VM#{guestvm[0]} ".yellow + "Downloading Screenshots"
@@ -231,11 +229,10 @@ module Dorothy
       #UPDATE DOROTHIBE DB#
       #####################
 
-      pcapfile =  bin.dir_pcap + dumpname + ".pcap"
-      dump = Loadmalw.new(pcapfile)
+      dump = Loadmalw.new(bin.dir_pcap + dumpname + ".pcap")
 
       #pcaprpath = bin.md5 + "/pcap/" + dump.filename
-      pcaprid = Loadmalw.calc_pcaprid(dump.filename, dump.size)
+      pcaprid = Loadmalw.calc_pcaprid(dump.filename, dump.size).rstrip
 
       LOGGER.debug "NAM", "VM#{guestvm[0]} ".yellow + "Pcaprid: " + pcaprid if VERBOSE
 

data/lib/mu/xtractr/about.rb

@@ -0,0 +1,57 @@
+# "THE BEER-WARE LICENSE" (Revision 42):
+# Mu[http://www.mudynamics.com] wrote this file. As long as you retain this 
+# notice you can do whatever you want with this stuff. If we meet some day, 
+# and you think this stuff is worth it, you can buy us a beer in return. 
+#
+# All about pcapr
+# * http://www.pcapr.net
+# * http://groups.google.com/group/pcapr-forum
+# * http://twitter.com/pcapr
+#
+# Mu Dynamics
+# * http://www.mudynamics.com
+# * http://labs.mudynamics.com
+
+module Mu
+class Xtractr
+# = About
+# Contains the meta data about the index including the number of packets,
+# flows, hosts, services and also the duration (in seconds) of the indexed 
+# pcaps.
+#
+#  xtractr.about.duration
+#  xtractr.about.packets
+class About
+    # Returns the version of the xtractr server
+    attr_reader :version
+    
+    # Returns the ##packets in the index
+    attr_reader :packets
+    
+    # Returns the ##flows in the index
+    attr_reader :flows
+    
+    # Returns the ##hosts in the index
+    attr_reader :hosts
+    
+    # Returns the ##services in the index
+    attr_reader :services
+    
+    # Returns the total duration of all the pcaps in the index
+    attr_reader :duration
+    
+    def initialize json # :nodoc:
+        @version  = json['version']
+        @packets  = json['packets']
+        @flows    = json['flows']
+        @hosts    = json['hosts']
+        @services = json['services']
+        @duration = json['duration']
+    end
+    
+    def inspect # :nodoc:
+        "#<about \##{flows} flows, \##{packets} packets>"
+    end
+end
+end # Xtractr
+end # Mu

data/lib/mu/xtractr/content.rb

@@ -0,0 +1,68 @@
+# "THE BEER-WARE LICENSE" (Revision 42):
+# Mu[http://www.mudynamics.com] wrote this file. As long as you retain this 
+# notice you can do whatever you want with this stuff. If we meet some day, 
+# and you think this stuff is worth it, you can buy us a beer in return. 
+#
+# All about pcapr
+# * http://www.pcapr.net
+# * http://groups.google.com/group/pcapr-forum
+# * http://twitter.com/pcapr
+#
+# Mu Dynamics
+# * http://www.mudynamics.com
+# * http://labs.mudynamics.com
+
+module Mu
+class Xtractr
+# = Content
+# Content is the next level of abstraction beyond Message. When a stream is
+# fetched from xtractr, all registered stream processors are invoked on the
+# various messages. For example, the HTTP content processor, pulls out the
+# response body from HTTP requests and responses, dechunks them and potentially
+# unzips the content. The resulting content represents the HTML file or a JPEG
+# image that can be saved off.
+#
+#  xtractr.packets('http.content.type:gif').first.flow.stream.contents.each do |c|
+#      c.save
+#  end
+class Content
+    # The name of the content (like a jpeg or pdf file).
+    attr_accessor :name
+    
+    # The encoding (base64, gzip, deflate, etc) of this content.
+    attr_accessor :encoding
+    
+    # The mime type of this content.
+    attr_accessor :type
+    
+    # The message from which this content was extracted.
+    attr_reader   :message
+    
+    # The actual body of the content (gunzip'd PDF file, for example)
+    attr_accessor :body
+    
+    def initialize message # :nodoc:
+        @name    = "content.#{message.stream.flow.id}.#{message.index}"
+        @type    = 'application/unknown'
+        @body    = nil
+        @message = message
+    end
+    
+    # Save the content to a file. If the filename is not provided then the
+    # content name is used instead. This is a convenience method used for
+    # method chaining.
+    #  flow.stream.contents.first.save
+    def save filename=nil
+        open(filename || name, "w") do |ios|
+            ios.write body
+        end
+        return self
+    end
+    
+    def inspect # :nodoc:
+        preview = body[0..32].inspect
+        "#<content #{name} #{type} #{encoding} #{preview}>"
+    end
+end
+end # Xtractr
+end # Mu