recog 0.01

data/.gitignore

@@ -0,0 +1,3 @@
+# ignore rvm files
+.ruby-version
+.ruby-gemset

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format documentation

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+gem 'nokogiri'
+
+group :test do
+  gem 'rspec', '~> 2.14.1'
+  gem 'cucumber', '~> 1.3.8'
+  gem 'aruba', '~> 0.5.3'
+end

data/Gemfile.lock

@@ -0,0 +1,42 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    aruba (0.5.3)
+      childprocess (>= 0.3.6)
+      cucumber (>= 1.1.1)
+      rspec-expectations (>= 2.7.0)
+    builder (3.2.2)
+    childprocess (0.3.9)
+      ffi (~> 1.0, >= 1.0.11)
+    cucumber (1.3.10)
+      builder (>= 2.1.2)
+      diff-lcs (>= 1.1.3)
+      gherkin (~> 2.12)
+      multi_json (>= 1.7.5, < 2.0)
+      multi_test (>= 0.0.2)
+    diff-lcs (1.2.5)
+    ffi (1.9.3)
+    gherkin (2.12.2)
+      multi_json (~> 1.3)
+    mini_portile (0.5.2)
+    multi_json (1.8.2)
+    multi_test (0.0.3)
+    nokogiri (1.6.1)
+      mini_portile (~> 0.5.0)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.7)
+    rspec-expectations (2.14.4)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.4)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  aruba (~> 0.5.3)
+  cucumber (~> 1.3.8)
+  nokogiri
+  rspec (~> 2.14.1)

data/LICENSE

@@ -0,0 +1,23 @@
+Copyright (c) 2014, Rapid7
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this
+  list of conditions and the following disclaimer in the documentation and/or
+  other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,63 @@
+Recog: A Recognition Framework
+=====
+
+Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simply to extract useful information from web server banners, snmp system description fields, and a whole lot more. Recog is open source, please see the [LICENSE](https://github.com/recog/LICENSE) file for more information.
+
+
+## Installation
+
+Recog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (1.9.3+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
+
+    $ git clone git@github.com:rapid7/recog.git
+    $ cd recog
+    $ bundle install
+
+## Maturity
+
+Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase in Recog is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects. 
+
+## Fingerprints
+
+The fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/recog/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
+
+A fingerprint file consists of an XML document like the following:
+
+    01: <?xml version="1.0"?>
+    02: 
+    03: <fingerprints matches="ssh.banner">
+    04: 
+    05: <fingerprint pattern="^RomSShell_([\d\.]+)$">
+    06:  <description>Allegro RomSShell SSH</description>
+    07:  <example>RomSShell_4.62</example>
+    08:  <param pos="0" name="service.vendor" value="Allegro"/>
+    09:  <param pos="0" name="service.product" value="RomSShell"/>
+    10:  <param pos="1" name="service.version"/>
+    11: </fingerprint>
+    12: 
+    13: </fingerprints>
+
+The first line should always consist of the XML version declaration. The first element should always be a <fingerpints/> block with a `matches` attribute indicating what this fingerprint file is supposed to match. The `matches` attribute is normally in the form of protocol.field. 
+
+Inside of the <fingerprints/> element there should be one or more <fingerprint/> elements. Every fingerprint should contain a `pattern` attribute, which contains the regular expression to be used against the match key. 
+
+Inside of the fingerprint, a <description/> element should contain a human-readable string describing this fingerprint. 
+
+The <example/> element should contain a successful match for the fingerprint's `pattern`. Multiple <example/> elements are preferred, as these elements are used for the built-in regression testing suite.
+
+the <param/> elements contain a `pos` attribute, which indicates what capture field from the `pattern` should be extracted, or `0` for a static string. The `name` attribute is the key that will be reported in the case of a successful match and the `value` will either be a static string for `pos` values of `0` or missing and taken from the captured field.
+
+Once a fingerprint has been added, the <examples/> entries can be tested by executing `bin/recog_verify.rb` against the fingerprint file:
+    
+    $ bin/recog_verify.rb xml/ssh_banners.xml
+    
+Matches can be tested on the command-line in a similar fashion:
+    
+    $ echo 'OpenSSH_6.6p1 Ubuntu-2ubuntu1' | bin/recog_match.rb xml/ssh_banners.xml -
+    MATCH: {"service.version"=>"6.6p1", "openssh.comment"=>"Ubuntu-2ubuntu1", "service.vendor"=>"OpenBSD", "service.family"=>"OpenSSH", "service.product"=>"OpenSSH", "data"=>"OpenSSH_6.6p1 Ubuntu-2ubuntu1"}
+
+
+
+
+
+
+

data/bin/recog_export.rb

@@ -0,0 +1,81 @@
+#!/usr/bin/env ruby
+
+$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), "..", "lib")))
+require 'optparse'
+require 'ostruct'
+require 'recog'
+
+def squash_lines(str)
+  str.split(/\n/).join(' ').gsub(/\s+/, ' ')
+end
+
+def export_text(options)
+end
+
+def export_ruby(options)
+  $stdout.puts "# Recog fingerprint database export [ #{File.basename(options.xml_file)} ] on #{Time.now.to_s}"
+  $stdout.puts "fp_str   = '' # Set this value to the match string"
+  $stdout.puts "fp_match = {} # Match results are stored here"
+  $stdout.puts ""
+  $stdout.puts "case fp_str"
+  options.db.fingerprints.each do |fp|
+    puts "  # #{squash_lines fp.name}"
+    puts "  when /#{fp.regex.to_s}/"
+    fp.tests.each do |test|
+      puts "    # Example: #{squash_lines test}"
+    end
+    fp.params.each_pair do |k,v|
+      if v[0] == 0
+        puts "    fp_match[#{k.inspect}] = #{v[1].inspect}"
+      else
+        puts "    fp_match[#{k.inspect}] = $#{v[0].to_s}"
+      end
+    end
+    puts ""
+  end
+  $stdout.puts "end"
+end
+
+
+options = OpenStruct.new(etype: :ruby)
+
+option_parser = OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options] XML_FINGERPRINTS_FILE"
+  opts.separator "Exports an XML fingerprint database to another format."
+  opts.separator ""
+  opts.separator "Options"
+
+  opts.on("-t", "--type type", 
+          "Choose a type of export.",
+          "  [r]uby (default - export a ruby case statement with regular expressions)",
+          "  [t]ext (export a text description of the fingerprints)") do |etype|
+    case etype.downcase
+    when /^r/
+      options.etype = :ruby
+    when /^t/
+      options.etype = :text
+    end
+  end
+
+  opts.on("-h", "--help", "Show this message.") do
+    puts opts
+    exit
+  end
+end
+option_parser.parse!(ARGV)
+
+if ARGV.count != 1
+  puts option_parser
+  exit
+end
+
+options.xml_file = ARGV.shift
+options.db = Recog::DB.new(options.xml_file)
+
+case options.etype
+when :ruby
+  export_ruby(options)
+when :text
+  export_text(options)
+end
+

data/bin/recog_match.rb

@@ -0,0 +1,51 @@
+#!/usr/bin/env ruby
+
+$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), "..", "lib")))
+require 'optparse'
+require 'ostruct'
+require 'recog'
+require 'recog/matcher_factory'
+
+options = OpenStruct.new(color: false, detail: false, fail_fast: false)
+
+option_parser = OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options] XML_FINGERPRINT_FILE <BANNERS_FILE>"
+  opts.separator "Identifies the matches and misses between the fingerprints and the banners file."
+  opts.separator ""
+  opts.separator "Options"
+
+  opts.on("-f", "--format FORMATTER", 
+          "Choose a formatter.",
+          "  [s]ummary (default - failure/match msgs)",
+          "  [d]etail  (msgs with total counts)") do |format|
+    if format.start_with? 'd'
+      options.detail = true
+    end
+  end
+
+  opts.on("--fail-fast [NUM]",
+          "Stop after number of failures (default: 10).") do |num|
+    options.fail_fast = true
+    options.stop_after = (num.to_i == 0) ? 10 : num.to_i
+  end
+
+  opts.on("-c", "--color", "Enable color in the output.") do
+    options.color = true
+  end
+
+  opts.on("-h", "--help", "Show this message.") do
+    puts opts
+    exit
+  end
+end
+option_parser.parse!(ARGV)
+
+if ARGV.count != 2
+  puts option_parser
+  exit
+end
+
+ndb = Recog::DB.new(ARGV.shift)
+options.fingerprints = ndb.fingerprints
+matcher = Recog::MatcherFactory.build(options)
+matcher.match_banners(ARGV.shift || "-")

data/bin/recog_verify.rb

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+
+$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), "..", "lib")))
+require 'optparse'
+require 'ostruct'
+require 'recog'
+require 'recog/verifier_factory'
+
+options = OpenStruct.new(color: false, detail: false)
+
+option_parser = OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options] XML_FINGERPRINTS_FILE"
+  opts.separator "Verifies that each fingerprint passes its internal tests."
+  opts.separator ""
+  opts.separator "Options"
+
+  opts.on("-f", "--format FORMATTER", 
+          "Choose a formatter.",
+          "  [s]ummary (default - failure/warning msgs and summary)",
+          "  [d]etail  (fingerprint name with tests and expanded summary)") do |format|
+    if format.start_with? 'd'
+      options.detail = true
+    end
+  end
+
+  opts.on("-c", "--color", "Enable color in the output.") do
+    options.color = true
+  end
+
+  opts.on("-h", "--help", "Show this message.") do
+    puts opts
+    exit
+  end
+end
+option_parser.parse!(ARGV)
+
+if ARGV.count != 1
+  puts option_parser
+  exit
+end
+
+ndb = Recog::DB.new(ARGV.shift)
+options.fingerprints = ndb.fingerprints
+verifier = Recog::VerifierFactory.build(options)
+verifier.verify_tests

data/features/match.feature

@@ -0,0 +1,16 @@
+Feature: Match
+  Scenario: Finds matches
+    When I run `match.rb matching_banners_fingerprints.xml banners.xml`
+    Then it should pass with:
+      """
+      MATCH: {"pureftpd.config"=>"[privsep] [TLS] ", "service.family"=>"Pure-FTPd", "service.product"=>"Pure-FTPd", "data"=>"---------- Welcome to Pure-FTPd [privsep] [TLS] ----------"}
+      MATCH: {"os.vendor"=>"Sun", "os.family"=>"Solaris", "os.product"=>"Solaris", "os.device"=>"General", "host.name"=>"polaris", "os.version"=>"5.8", "data"=>"polaris FTP server (SunOS 5.8) ready."}
+      """
+
+  Scenario: Fails at finding matches
+    When I run `match.rb failing_banners_fingerprints.xml banners.xml`
+    Then it should pass with:
+      """
+      FAIL: ---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
+      FAIL: polaris FTP server (SunOS 5.8) ready
+      """

data/features/support/env.rb

@@ -0,0 +1,5 @@
+require 'aruba/cucumber'
+
+Before do
+  @dirs = ["features/xml"]
+end

data/features/verify.feature

@@ -0,0 +1,31 @@
+Feature: Verify
+  Scenario: No tests
+    When I run `verify.rb no_tests.xml`
+    Then it should pass with:
+      """
+      SUMMARY: Test completed with 0 successful, 0 warnings, and 0 failures
+      """
+
+  Scenario: Successful tests
+    When I run `verify.rb successful_tests.xml`
+    Then it should pass with:
+      """
+      SUMMARY: Test completed with 2 successful, 0 warnings, and 0 failures
+      """
+
+  Scenario: Tests with warnings
+    When I run `verify.rb tests_with_warnings.xml`
+    Then it should pass with:
+      """
+      WARN: 'Pure-FTPd' failed to match \"---------- Welcome to Pure-FTPd ----------\" key 'pureftpd.config'' with (?-mix:^-{10} Welcome to Pure-FTPd (.*)-{10}$)'
+      SUMMARY: Test completed with 1 successful, 1 warnings, and 0 failures
+      """
+
+  Scenario: Tests with failures
+    When I run `verify.rb tests_with_failures.xml`
+    Then it should pass with:
+      """
+      FAIL: 'foo test' failed to match "bar" with (?-mix:^foo$)'
+      FAIL: '' failed to match "This almost matches" with (?-mix:^This matches$)'
+      SUMMARY: Test completed with 0 successful, 0 warnings, and 2 failures
+      """

data/features/xml/banners.xml

@@ -0,0 +1,2 @@
+---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
+polaris FTP server (SunOS 5.8) ready.

data/features/xml/failing_banners_fingerprints.xml

@@ -0,0 +1,20 @@
+<?xml version="1.0"?>
+<fingerprints>
+  <fingerprint pattern="^=\(.\*.\)=-\.:\. \(\( Welcome to PureFTPd (\d+\..+) \)\) \.:\.-=\(.\*.\)=-$">
+    <example>=(&lt;*&gt;)=-.:. (( Welcome to PureFTPd 1.1.0 )) .:.-=(&lt;*&gt;)=-</example>
+    <description>Older Pure-FTPd versions</description>
+    <param pos="0" name="service.family" value="Pure-FTPd"/>
+    <param pos="0" name="service.product" value="Pure-FTPd"/>
+    <param pos="1" name="service.version"/>
+  </fingerprint>
+  <fingerprint pattern="^(\S+) FTP Server \(Solaris (\S+)\) ready\.?$" flags="REG_ICASE">
+    <description>SunOS/Solaris</description>
+    <example>example.com FTP server (Solaris 5.7) ready.</example>
+    <param pos="0" name="os.vendor" value="Sun"/>
+    <param pos="0" name="os.family" value="Solaris"/>
+    <param pos="0" name="os.product" value="Solaris"/>
+    <param pos="0" name="os.device" value="General"/>
+    <param pos="1" name="host.name"/>
+    <param pos="2" name="os.version"/>
+  </fingerprint>
+</fingerprints>

data/features/xml/matching_banners_fingerprints.xml

@@ -0,0 +1,22 @@
+<?xml version="1.0"?>
+<fingerprints>
+  <fingerprint pattern="^-{10} Welcome to Pure-FTPd (.*)-{10}$">
+    <example>---------- Welcome to Pure-FTPd ----------</example>
+    <description>Pure-FTPd
+      Config data can be zero or more of: [privsep] [TLS]
+      </description>
+    <param pos="1" name="pureftpd.config"/>
+    <param pos="0" name="service.family" value="Pure-FTPd"/>
+    <param pos="0" name="service.product" value="Pure-FTPd"/>
+  </fingerprint>
+  <fingerprint pattern="^(\S+) FTP Server \(SunOS (\S+)\) ready\.?$" flags="REG_ICASE">
+    <description>SunOS/Solaris</description>
+    <example>example.com FTP server (SunOS 5.7) ready.</example>
+    <param pos="0" name="os.vendor" value="Sun"/>
+    <param pos="0" name="os.family" value="Solaris"/>
+    <param pos="0" name="os.product" value="Solaris"/>
+    <param pos="0" name="os.device" value="General"/>
+    <param pos="1" name="host.name"/>
+    <param pos="2" name="os.version"/>
+  </fingerprint>
+</fingerprints>

data/features/xml/no_tests.xml

@@ -0,0 +1,53 @@
+<?xml version="1.0"?>
+<!--
+SMTP response lines to the EHLO command are matched against these patterns
+(1 line at a time) to fingerprint SMTP servers.
+
+See comment at the top of smtp_banners.xml for additional info.
+-->
+
+<fingerprints>
+   <fingerprint pattern="^500[ -]Syntax error, command &quot;XXXX&quot; unrecognized$">
+      <description>
+         Cisco PIX changes the command letters to 'X' before passing
+         them to the real SMTP server.
+      </description>
+      <param pos="0" name="service.vendor" value="Cisco"/>
+      <param pos="0" name="service.family" value="PIX"/>
+      <param pos="0" name="service.product" value="PIX"/>
+   </fingerprint>
+
+   <!--
+   Don't try to infer a fingerprint from XEXCH50, because if we do, it might overwrite
+   a very precise MS IIS SMTP service or MS Exchange Server fingerprint found with the
+   help of smtp_banners.xml. Instead, this case is handled specially by the Jess rule
+   smtp-iis-xexch50-svc-fingerprint. -mrb
+
+   <fingerprint pattern="^250[ -] *XEXCH50.*$">
+      <description>
+         Microsoft Exchange/IIS server
+      </description>
+      <param pos="0" name="service.vendor" value="Microsoft"/>
+      <param pos="0" name="service.family" value="IIS"/>
+      <param pos="0" name="service.product" value="IIS"/>
+      <param pos="0" name="os.vendor" value="Microsoft"/>
+      <param pos="0" name="os.family" value="Windows"/>
+      <param pos="0" name="os.device" value="General"/>
+      <param pos="0" name="os.product" value="Windows"/>
+   </fingerprint>
+   -->
+
+   <fingerprint pattern="^221[ -]See ya in cyberspace$">
+      <description>
+         221 See ya in cyberspace
+      </description>
+      <param pos="0" name="service.vendor" value="Alt-N"/>
+      <param pos="0" name="service.family" value="MDaemon"/>
+      <param pos="0" name="service.product" value="MDaemon"/>
+      <param pos="0" name="os.vendor" value="Microsoft"/>
+      <param pos="0" name="os.family" value="Windows"/>
+      <param pos="0" name="os.device" value="General"/>
+      <param pos="0" name="os.product" value="Windows"/>
+      <param pos="0" name="os.arch" value="x86"/>
+   </fingerprint>
+</fingerprints>