pcapr-local 0.1.13 → 0.2.0

data/README.md

@@ -2,63 +2,146 @@
 
 ## Introduction
 
-pcapr.Local is a tool for browsing and managing a large repository of packet capture files (pcaps). After you install and configure pcapr.Local with the location of your pcaps, it automatically indexes those pcaps and enables you to navigate your collection using a web browser. Pcapr.Local extends and integrates with [Xtractr](http://code.google.com/p/pcapr/wiki/Xtractr), so it uses the Xtractr web UI hosted on pcapr.net. However, because the UI is configured to talk to the local Xtractr instance managed by pcapr.Local, your data never leaves your network.
-
-In addition to managing pcaps, pcapr.Local helps you leverage your custom Wireshark dissectors to create Scenarios in Mu Studio. Just download the PAR file (Pcap ARchive) file created by pcapr.Local and import it into Mu Studio, where your Wireshark data guides Scenario creation.
+pcapr.Local is a tool for browsing and managing a large repository of packet 
+capture files (pcaps). After you install and configure pcapr.Local with the 
+location of your pcaps, it automatically indexes those pcaps and enables you 
+to navigate your collection using a web browser. pcapr.Local extends and 
+integrates with [Xtractr](http://code.google.com/p/pcapr/wiki/Xtractr), so it 
+uses the Xtractr web UI hosted on pcapr.net. However, because the UI is 
+configured to talk to the local Xtractr instance managed by pcapr.Local, your 
+data never leaves your network.
+
+In addition to managing pcaps, pcapr.Local helps you leverage your custom 
+Wireshark dissectors and command-line options to create Scenarios in Mu 
+Studio. Just download the PAR file (Pcap ARchive) file created by pcapr.Local 
+and import it into Mu Studio, where your Wireshark data guides Scenario 
+creation.
+
+You can learn more about pcapr.Local in our announcement 
+[blog](http://labs.mudynamics.com/2011/04/18/announcing-pcaprlocal/).
+
+![pcapr.Local](http://blog.mudynamics.com/wp-content/uploads/2011/04/pcaprlocal.png)
 
 ## Dependencies
 
 ### Supported Environments
 
-Linux (any flavor). You can install on a dedicated Linux system or in a virtual machine (VM). 
+Linux (any flavor). You can install on a dedicated Linux system or in a 
+virtual machine (VM).
 
 ### Ruby & Rubygems
 
-Ruby (1.8.6, 1.8.7, 1.9.2) + Rubygems (1.3.7 or higher). When using Ruby 1.8.6, you must install rubygems 1.3.7. Rubygems officially ceased support for ruby 1.8.6 as of the rubygems 1.4.0 release, so any version 1.4.x or higher will not install on a ruby 1.8.6 system.
+Ruby (1.8.6, 1.8.7, 1.9.2) + Rubygems (1.3.7 or higher). When using Ruby 
+1.8.6, you must install rubygems 1.3.7. Rubygems officially ceased support for 
+ruby 1.8.6 as of the rubygems 1.4.0 release, so any version 1.4.x or higher 
+will not install on a ruby 1.8.6 system.
 
 ### CouchDB
-Local and remote installations supported. If you have configured a username and password for the CouchDB service, you'll need to provide those user credentials during the pcapr.Local gem installation. On Ubuntu/Debian you can install CouchDB with:
 
-   $ sudo apt-get install couchdb
+Local and remote installations supported. If you have configured a username 
+and password for the CouchDB service, you'll need to provide those user 
+credentials during the pcapr.Local gem installation. On Ubuntu/Debian you can 
+install CouchDB with:
+
+    $ sudo apt-get install couchdb
 
 ### Wireshark (any version)
 
-Pcapr.Local will automatically use the installed version of tshark (a component of Wireshark) to create the pcap indexes. When using a package manager (such as aptitude on Ubuntu), you might need to install tshark command line utility separately if it's not included as part of the Wireshark installation. 
+pcapr.Local will automatically use the installed version of `tshark` (a 
+component of Wireshark) to create the pcap indexes. When using a package 
+manager (such as aptitude on Ubuntu), you might need to install `tshark` 
+command line utility separately if it's not included as part of the Wireshark 
+installation.
 
 ### Zip (any version)
-Pcapr.Local requires zip to create PAR files from your indexed pcaps.
+
+pcapr.Local requires zip to create PAR files from your indexed pcaps.
 
 ## Running pcapr.Local
 
-1. Install the gem. 
-2. Run the "startpcapr" executable that is installed with the gem:
+1.    Install the gem. 
+2.    Run the `startpcapr` executable that is installed with the gem:
+
+        $ startpcapr
+
+3.    This configuration script asks you some basic questions and records your
+      answers in a config file at `~/.pcapr_local/config` that will be used on 
+      subsequent invocations. After collecting configuration information, the 
+      server process returns a prompt but continues running in the background. 
+      To monitor the process, tail the pcapr.Local log file with:
 
-    $ startpcapr
+        $ tail -F ~/pcapr.Local/log/server.log
 
-This configuration script asks you some basic questions and records your answers in a config file at ~/.pcapr_local/config that will be used on subsequent invocations. After collecting configuration information, the server process returns a prompt but continues running in the background. To monitor the process, tail the pcapr.Local log file with:
+4.    Add your packet capture files to the `pcaps` directory you configured
+      (default `~/pcapr.Local/pcaps`) and wait a few minutes for pcapr.Local to
+      index them.
 
-    $ tail -F ~/pcapr.Local/log/server.log
+5.    Point your web browser to http://localhost:8080 (or whatever you 
+      configured).
 
-3. Add your packet capture files to the pcap directory you configured (default ~/pcapr.Local/pcaps) and wait a few minutes for pcap.Local to index them. 
-4. Point your web browser to http://localhost:8080 (or whatever you configured).
-5. Stop the pcapr.Local server with:
+6.    Stop the pcapr.Local server with:
 
-    $ stoppcapr
+        $ stoppcapr
 
 ## Creating PAR Files
 
-A PAR file (Pcap ARchive) is a format that can be imported into Mu Studio to create a Scenario. Although a PAR file is equivalent to a pcap file for the purposes of Scenario creation, because a PAR contains dissection data from your local Wireshark installation, you'll get the full benefits of any custom dissectors used by that installation. Additionally, when you import a PAR file you'll bypass flow selection and go directly to the Scenario Editor.
+A PAR file (Pcap ARchive) is a format that can be imported into Mu Studio to 
+create a Scenario. Although a PAR file is equivalent to a pcap file for the 
+purposes of Scenario creation, because a PAR contains dissection data from 
+your local Wireshark installation, you'll get the full benefits of any custom 
+dissectors used by that installation. Additionally, when you import a PAR file 
+you'll bypass flow selection and go directly to the Scenario Editor.
 
 ### In the Web UI
 
-Point your web browser to http://localhost:8080 (or whatever you configured), then select a pcap to view its details. At the bottom of the details page, click the Download PAR File link.
+Point your web browser to http://localhost:8080 (or whatever you configured), 
+then select a pcap to view its details. At the bottom of the details page, 
+click the Download PAR File link.
 
 ### On the Command Line
 
-The gem bundles a CLI tool for creating PAR files called 'pcap2par'. To use, just provide a path to your pcap:
+The gem bundles a CLI tool for creating PAR files called `pcap2par`. To use, 
+just provide a path to your pcap:
  
-    $  pcap2par my_traffic.pcap
+    $ pcap2par my_traffic.pcap
+
+This creates a PAR file called `export.par` in the current directory. You can 
+optionally specify the name of the output file as a second argument:
+
+    $ pcap2par my_traffic.pcap ~/par_files/my_traffic.par 
+
+## Custom Wireshark Command-Line Options
+
+pcapr.Local supports supplying custom options to `tshark` if the options are 
+added to `~/.pcapr_local/config`. If you need any of these options to make 
+things decode successfully in your environment, you can add them to the 
+tshark.options stanza in the pcapr.Local configuration file.
+
+      "tshark": {
+        "options": "-d tcp.port==2121,ftp"
+      }
+
+See the [tshark man page](http://www.wireshark.org/docs/man-pages/tshark.html) 
+for more information.
+
+### User Specified Decodes (Decode As)
+
+If you use nonstandard TCP / UDP ports or Ethernet / IP protocol numbers in 
+your environment, `tshark` can decode the protocols correctly when you 
+manually map them to the proper dissectors.
+
+The following command-line option example adds `2121/tcp` as an additional port 
+for FTP traffic:
+
+    -d tcp.port==2121,ftp
+
+### Custom Lua Dissectors
+
+If you use custom protocols which are not present in the standard version of 
+`tshark`, you can decode these by creating a Lua script dissector.
 
-This creates a PAR file called "export.par" in the current directory. You can optionally specify the name of the output file as a second argument:
+The following command-line option example loads a Lua script from 
+`custom_dissector.lua`, and adds it to Wireshark's existing dissector library 
+to support additional protocols:
 
-    $ pcap2par my_traffic.pcap ~/par_files/my_traffic.par 
+    -X lua_script:custom_dissector.lua

data/Rakefile

@@ -3,14 +3,16 @@ require 'rake'
 
 require 'jeweler'
 Jeweler::Tasks.new do |gem|
-  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  # gem is a Gem::Specification
+  # http://docs.rubygems.org/read/chapter/20
   gem.name = "pcapr-local"
   gem.homepage = "http://github.com/pcapr-local/pcapr-local"
   gem.license = "MIT"
   gem.summary = %Q{Manage your pcap collection}
   gem.description = %Q{Index, Browse, and Query your vast pcap collection.}
-  gem.email = "nbaggott@gmail.com"
+  gem.email = "rubygems@mudynamics.com"
   gem.authors = ["Mu Dynamics"]
+  
   gem.add_dependency "rest-client", ">= 1.6.1"
   gem.add_dependency "couchrest", "~> 1.0.1"
   gem.add_dependency "sinatra", "~> 1.1.0"
@@ -18,15 +20,11 @@ Jeweler::Tasks.new do |gem|
   gem.add_dependency "thin", "~> 1.2.7"
   gem.add_dependency "rack", "~> 1.2.1"
   gem.add_dependency "rack-contrib", "~> 1.1.0"
-  # Include your dependencies below. Runtime dependencies are required when using your gem,
-  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
-  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
-  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+  
   gem.add_development_dependency "shoulda", ">= 0"
   gem.add_development_dependency "bundler", "~> 1.0.0"
   gem.add_development_dependency "jeweler", "~> 1.5.2"
   gem.add_development_dependency "rcov", ">= 0"
-
 end
 Jeweler::RubygemsDotOrgTasks.new
 

data/VERSION

@@ -1 +1 @@
-0.1.13
+0.2.0

data/lib/mu/scenario/pcap.rb

@@ -15,6 +15,7 @@ module Pcap
     TSHARK_READ_TIMEOUT     = 10.0 # seconds
     TSHARK_LINES_PER_PACKET = 16384
     TSHARK_OPTS = "-n -o tcp.desegment_tcp_streams:false"
+    TSHARK_OPTS_SUFFIX = TSHARK_OPTS
     TSHARK_SIZE_OPTS = "-n -o 'column.format: cum_size, \"%B\"'"
     TSHARK_PSML_OPTS = %Q{#{TSHARK_OPTS} -o 'column.format: "Protocol", "%p", "Info", "%i"'}
 
@@ -24,6 +25,13 @@ module Pcap
     EXCLUDE_FROM_SIZE_CHECK = ['rtp'].freeze
 
     class PcapTooLarge < StandardError; end
+    
+    def self.reset_options options
+        return unless options
+        tshark_opts = options << ' ' << TSHARK_OPTS_SUFFIX
+        remove_const(:TSHARK_OPTS) if const_defined?(:TSHARK_OPTS)
+        const_set(:TSHARK_OPTS, tshark_opts)
+    end
 
     def self.validate_pcap_size(path)
         tshark_filter = EXCLUDE_FROM_SIZE_CHECK.map{ |proto| "not #{proto}" }.join " and "

data/lib/pcapr_local.rb

@@ -69,7 +69,14 @@ module PcaprLocal
     def self.get_db config
         PcaprLocal::DB.get_db config.fetch("couch")
     end
-
+    
+    def self.set_tshark_options config
+        options = config['tshark']['options']
+        return unless options
+        
+        Mu::Scenario::Pcap.reset_options(options)
+    end
+    
     def self.start config=nil
         config ||= PcaprLocal::Config.config
         
@@ -93,6 +100,9 @@ module PcaprLocal
 
         # Create pid file that will be deleted when we shutdown.
         create_pid_file config['pidfile']
+        
+        # Override the default tshark options
+        set_tshark_options(config)
 
         # Get database instance
         db = get_db config

data/lib/pcapr_local/config.rb

@@ -9,14 +9,14 @@ module Config
 
     DEFAULT_CONFIG = {
         # Shared config.
-        "install_dir" => "#{HOME}/pcapr.Local/",
-        "pcap_dir"    => "#{HOME}/pcapr.Local/pcaps",
-        "index_dir"   => "#{HOME}/pcapr.Local/indexes",
+        "install_dir"     => "#{HOME}/pcapr.Local/",
+        "pcap_dir"        => "#{HOME}/pcapr.Local/pcaps",
+        "index_dir"       => "#{HOME}/pcapr.Local/indexes",
 
         # UI (Sinatra)
         "app" => {
-            "host" => '127.0.0.1',
-            "port" => 8080,
+            "host"        => '127.0.0.1',
+            "port"        => 8080,
         },
 
         # Pcap scanning
@@ -27,15 +27,20 @@ module Config
 
         # Couch
         "couch" => {
-            "uri" => 'http://127.0.0.1:5984/',
-            "database" => "pcapr_local"
+            "uri"         => 'http://127.0.0.1:5984/',
+            "database"    => 'pcapr_local'
         },
 
         # Xtractr
         "xtractr" => {
             "path"         => 'xtractr',
             "idle_timeout" => 60  # kill xtractr browser after n seconds of idle time 
-        }
+        },
+        
+        # tshark
+        "tshark" => {
+            "options"      => ''
+        },
     }
 
 

data/pcapr-local.gemspec

@@ -5,13 +5,13 @@
 
 Gem::Specification.new do |s|
   s.name = %q{pcapr-local}
-  s.version = "0.1.13"
+  s.version = "0.2.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Mu Dynamics"]
-  s.date = %q{2011-04-04}
+  s.date = %q{2011-12-08}
   s.description = %q{Index, Browse, and Query your vast pcap collection.}
-  s.email = %q{nbaggott@gmail.com}
+  s.email = %q{rubygems@mudynamics.com}
   s.executables = ["pcap2par", "startpcapr", "stoppcapr", "xtractr"]
   s.extra_rdoc_files = [
     "LICENSE.txt",
@@ -228,30 +228,6 @@ Gem::Specification.new do |s|
   s.require_paths = ["lib"]
   s.rubygems_version = %q{1.3.7}
   s.summary = %q{Manage your pcap collection}
-  s.test_files = [
-    "test/mu/pcap/reader/tc_http_family.rb",
-    "test/mu/pcap/tc_ethernet.rb",
-    "test/mu/pcap/tc_header.rb",
-    "test/mu/pcap/tc_ipv4.rb",
-    "test/mu/pcap/tc_ipv6.rb",
-    "test/mu/pcap/tc_packet.rb",
-    "test/mu/pcap/tc_pair.rb",
-    "test/mu/pcap/tc_pkthdr.rb",
-    "test/mu/pcap/tc_reader.rb",
-    "test/mu/pcap/tc_tcp.rb",
-    "test/mu/pcap/tc_udp.rb",
-    "test/mu/pcap/tc_wrapper.rb",
-    "test/mu/scenario/pcap/tc_fields.rb",
-    "test/mu/scenario/pcap/tc_rtp.rb",
-    "test/mu/scenario/tc_pcap.rb",
-    "test/mu/tc_pcap.rb",
-    "test/mu/testcase.rb",
-    "test/pcapr_local/tc_api.rb",
-    "test/pcapr_local/test_scanner.rb",
-    "test/pcapr_local/test_xtractr.rb",
-    "test/pcapr_local/testcase.rb",
-    "test/test_pcapr_local.rb"
-  ]
 
   if s.respond_to? :specification_version then
     current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION

metadata

@@ -4,9 +4,9 @@ version: !ruby/object:Gem::Version
   prerelease: false
   segments: 
   - 0
-  - 1
-  - 13
-  version: 0.1.13
+  - 2
+  - 0
+  version: 0.2.0
 platform: ruby
 authors: 
 - Mu Dynamics
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-04-04 00:00:00 -07:00
+date: 2011-12-08 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -179,7 +179,7 @@ dependencies:
   type: :development
   version_requirements: *id011
 description: Index, Browse, and Query your vast pcap collection.
-email: nbaggott@gmail.com
+email: rubygems@mudynamics.com
 executables: 
 - pcap2par
 - startpcapr
@@ -427,26 +427,5 @@ rubygems_version: 1.3.7
 signing_key: 
 specification_version: 3
 summary: Manage your pcap collection
-test_files: 
-- test/mu/pcap/reader/tc_http_family.rb
-- test/mu/pcap/tc_ethernet.rb
-- test/mu/pcap/tc_header.rb
-- test/mu/pcap/tc_ipv4.rb
-- test/mu/pcap/tc_ipv6.rb
-- test/mu/pcap/tc_packet.rb
-- test/mu/pcap/tc_pair.rb
-- test/mu/pcap/tc_pkthdr.rb
-- test/mu/pcap/tc_reader.rb
-- test/mu/pcap/tc_tcp.rb
-- test/mu/pcap/tc_udp.rb
-- test/mu/pcap/tc_wrapper.rb
-- test/mu/scenario/pcap/tc_fields.rb
-- test/mu/scenario/pcap/tc_rtp.rb
-- test/mu/scenario/tc_pcap.rb
-- test/mu/tc_pcap.rb
-- test/mu/testcase.rb
-- test/pcapr_local/tc_api.rb
-- test/pcapr_local/test_scanner.rb
-- test/pcapr_local/test_xtractr.rb
-- test/pcapr_local/testcase.rb
-- test/test_pcapr_local.rb
+test_files: []
+