recog 0.02 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2221758dde9a54ad057043aad3cad77ec8c8be7b
-  data.tar.gz: b2b077c53b397ebb0f32f88b38524e23ee8bae03
+  metadata.gz: 4177857ea00bc92010990ba3dd3d16a9df376773
+  data.tar.gz: 0e75995b2330ce2f6425c4c3b465bdee53644188
 SHA512:
-  metadata.gz: 268d78f87daefe24734c09a665dec59f35ce26a3bdf1255669329c3ce1668e7fbbd8a93a59d47ee0d294bfdade4e6f505f022e3c2b2f9cb2798231753804da07
-  data.tar.gz: 8b953534f3954862bec8dbc987cd3d13dc92df8d65848517139dc12a4cddf575dfefdca91351428766a6c755817e549bd4735b85de41ea8924d1ef6f9b2f339f
+  metadata.gz: 36c37a2bf118bd25a395d477d6d5b29a80e145fd2aaf09f33b522ef6678b79daaf86783293f5d06d06a63ba068aa05ca4425e77308cc6362b3ade01f41b38d03
+  data.tar.gz: 59fd715fe36dee81b72b98dfe2e2426681e67fd2d651df56feeb8918f3d215eb91a13fad2ee1ed8813fac6947061eb328e8b5314b0364c4cef187c72ab0937f1

data/.gitignore

@@ -1,3 +1,9 @@
+.yardoc
+coverage/
+doc/
+
+/Gemfile.lock
+
 # ignore rvm files
 .ruby-version
 .ruby-gemset

data/.rspec

@@ -1,2 +1,3 @@
 --color
---format documentation
+--warnings
+--require spec_helper

data/.travis.yml

@@ -1,5 +1,10 @@
 language: ruby
 rvm:
+  - 2.1.2
   - 2.0.0
   - 1.9.3
+  - jruby
+matrix:
+  allow_failures:
+    - rvm: jruby
 script: bundle exec rspec spec

data/.yardopts

@@ -0,0 +1 @@
+--markup markdown

data/Gemfile

@@ -1,9 +1,11 @@
 source 'https://rubygems.org'
 
+gemspec
+
 gem 'nokogiri'
 
 group :test do
-  gem 'rspec', '~> 2.14.1'
+  gem 'rspec', '>= 2.99'
   gem 'cucumber', '~> 1.3.8'
   gem 'aruba', '~> 0.5.3'
 end

data/README.md

@@ -3,7 +3,7 @@ Recog: A Recognition Framework
 
 Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simply to extract useful information from web server banners, snmp system description fields, and a whole lot more. Recog is open source, please see the [LICENSE](https://github.com/recog/LICENSE) file for more information.
 
-[![Build Status](https://travis-ci.org/rapid7/recog.png)](https://travis-ci.org/rapid7/recog) [![Code Climate](https://codeclimate.com/badge.png)](https://codeclimate.com/github/rapid7/recog)
+[![Build Status](https://travis-ci.org/rapid7/recog.png)](https://travis-ci.org/rapid7/recog)
 ==
 
 ## Installation
@@ -16,18 +16,18 @@ Recog consists of both XML fingerprint files and an assortment of code, mostly i
 
 ## Maturity
 
-Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase in Recog is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects. 
+Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase in Recog is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.
 
 ## Fingerprints
 
-The fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/recog/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
+The fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/rapid7/recog/blob/master/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
 
 A fingerprint file consists of an XML document like the following:
 
     01: <?xml version="1.0"?>
-    02: 
+    02:
     03: <fingerprints matches="ssh.banner">
-    04: 
+    04:
     05: <fingerprint pattern="^RomSShell_([\d\.]+)$">
     06:  <description>Allegro RomSShell SSH</description>
     07:  <example>RomSShell_4.62</example>
@@ -35,25 +35,25 @@ A fingerprint file consists of an XML document like the following:
     09:  <param pos="0" name="service.product" value="RomSShell"/>
     10:  <param pos="1" name="service.version"/>
     11: </fingerprint>
-    12: 
+    12:
     13: </fingerprints>
 
-The first line should always consist of the XML version declaration. The first element should always be a <fingerpints/> block with a `matches` attribute indicating what this fingerprint file is supposed to match. The `matches` attribute is normally in the form of protocol.field. 
+The first line should always consist of the XML version declaration. The first element should always be a <fingerpints/> block with a `matches` attribute indicating what this fingerprint file is supposed to match. The `matches` attribute is normally in the form of protocol.field.
 
-Inside of the <fingerprints/> element there should be one or more <fingerprint/> elements. Every fingerprint should contain a `pattern` attribute, which contains the regular expression to be used against the match key. 
+Inside of the <fingerprints/> element there should be one or more <fingerprint/> elements. Every fingerprint should contain a `pattern` attribute, which contains the regular expression to be used against the match key.
 
-Inside of the fingerprint, a <description/> element should contain a human-readable string describing this fingerprint. 
+Inside of the fingerprint, a <description/> element should contain a human-readable string describing this fingerprint.
 
 The <example/> element should contain a successful match for the fingerprint's `pattern`. Multiple <example/> elements are preferred, as these elements are used for the built-in regression testing suite.
 
 the <param/> elements contain a `pos` attribute, which indicates what capture field from the `pattern` should be extracted, or `0` for a static string. The `name` attribute is the key that will be reported in the case of a successful match and the `value` will either be a static string for `pos` values of `0` or missing and taken from the captured field.
 
 Once a fingerprint has been added, the <examples/> entries can be tested by executing `bin/recog_verify.rb` against the fingerprint file:
-    
+
     $ bin/recog_verify.rb xml/ssh_banners.xml
-    
+
 Matches can be tested on the command-line in a similar fashion:
-    
+
     $ echo 'OpenSSH_6.6p1 Ubuntu-2ubuntu1' | bin/recog_match.rb xml/ssh_banners.xml -
     MATCH: {"service.version"=>"6.6p1", "openssh.comment"=>"Ubuntu-2ubuntu1", "service.vendor"=>"OpenBSD", "service.family"=>"OpenSSH", "service.product"=>"OpenSSH", "data"=>"OpenSSH_6.6p1 Ubuntu-2ubuntu1"}
 

data/Rakefile

@@ -0,0 +1,22 @@
+require "bundler/gem_tasks"
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new do |t|
+    t.pattern = "spec/**/*_spec.rb"
+end
+
+require 'yard'
+require 'yard/rake/yardoc_task'
+YARD::Rake::YardocTask.new do |t|
+    t.files = ['lib/**/*.rb', '-', 'README.md']
+end
+
+require 'cucumber'
+require 'cucumber/rake/task'
+
+Cucumber::Rake::Task.new(:features) do |t|
+    t.cucumber_opts = "features --format pretty"
+end
+
+task :default => [ :spec, :features, :yard ]
+

data/bin/recog_verify.rb

@@ -14,7 +14,7 @@ option_parser = OptionParser.new do |opts|
   opts.separator ""
   opts.separator "Options"
 
-  opts.on("-f", "--format FORMATTER", 
+  opts.on("-f", "--format FORMATTER",
           "Choose a formatter.",
           "  [s]ummary (default - failure/warning msgs and summary)",
           "  [d]etail  (fingerprint name with tests and expanded summary)") do |format|

data/features/match.feature

@@ -1,6 +1,6 @@
 Feature: Match
   Scenario: Finds matches
-    When I run `match.rb matching_banners_fingerprints.xml banners.xml`
+    When I run `recog_match.rb matching_banners_fingerprints.xml banners.xml`
     Then it should pass with:
       """
       MATCH: {"pureftpd.config"=>"[privsep] [TLS] ", "service.family"=>"Pure-FTPd", "service.product"=>"Pure-FTPd", "data"=>"---------- Welcome to Pure-FTPd [privsep] [TLS] ----------"}
@@ -8,7 +8,7 @@ Feature: Match
       """
 
   Scenario: Fails at finding matches
-    When I run `match.rb failing_banners_fingerprints.xml banners.xml`
+    When I run `recog_match.rb failing_banners_fingerprints.xml banners.xml`
     Then it should pass with:
       """
       FAIL: ---------- Welcome to Pure-FTPd [privsep] [TLS] ----------

data/features/verify.feature

@@ -1,31 +1,34 @@
 Feature: Verify
   Scenario: No tests
-    When I run `verify.rb no_tests.xml`
+    When I run `recog_verify.rb no_tests.xml`
     Then it should pass with:
       """
       SUMMARY: Test completed with 0 successful, 0 warnings, and 0 failures
       """
 
   Scenario: Successful tests
-    When I run `verify.rb successful_tests.xml`
+    When I run `recog_verify.rb successful_tests.xml`
     Then it should pass with:
       """
-      SUMMARY: Test completed with 2 successful, 0 warnings, and 0 failures
+      SUMMARY: Test completed with 4 successful, 0 warnings, and 0 failures
       """
 
   Scenario: Tests with warnings
-    When I run `verify.rb tests_with_warnings.xml`
+    When I run `recog_verify.rb tests_with_warnings.xml`
     Then it should pass with:
       """
-      WARN: 'Pure-FTPd' failed to match \"---------- Welcome to Pure-FTPd ----------\" key 'pureftpd.config'' with (?-mix:^-{10} Welcome to Pure-FTPd (.*)-{10}$)'
+      WARN: 'Pure-FTPd' has no test cases
       SUMMARY: Test completed with 1 successful, 1 warnings, and 0 failures
       """
 
   Scenario: Tests with failures
-    When I run `verify.rb tests_with_failures.xml`
+    When I run `recog_verify.rb tests_with_failures.xml`
     Then it should pass with:
       """
       FAIL: 'foo test' failed to match "bar" with (?-mix:^foo$)'
       FAIL: '' failed to match "This almost matches" with (?-mix:^This matches$)'
-      SUMMARY: Test completed with 0 successful, 0 warnings, and 2 failures
+      FAIL: 'bar test' failed to find expected capture group os.version '5.0'
+      SUMMARY: Test completed with 0 successful, 0 warnings, and 3 failures
       """
+
+

data/features/xml/no_tests.xml

@@ -1,53 +1,3 @@
 <?xml version="1.0"?>
-<!--
-SMTP response lines to the EHLO command are matched against these patterns
-(1 line at a time) to fingerprint SMTP servers.
-
-See comment at the top of smtp_banners.xml for additional info.
--->
-
 <fingerprints>
-   <fingerprint pattern="^500[ -]Syntax error, command &quot;XXXX&quot; unrecognized$">
-      <description>
-         Cisco PIX changes the command letters to 'X' before passing
-         them to the real SMTP server.
-      </description>
-      <param pos="0" name="service.vendor" value="Cisco"/>
-      <param pos="0" name="service.family" value="PIX"/>
-      <param pos="0" name="service.product" value="PIX"/>
-   </fingerprint>
-
-   <!--
-   Don't try to infer a fingerprint from XEXCH50, because if we do, it might overwrite
-   a very precise MS IIS SMTP service or MS Exchange Server fingerprint found with the
-   help of smtp_banners.xml. Instead, this case is handled specially by the Jess rule
-   smtp-iis-xexch50-svc-fingerprint. -mrb
-
-   <fingerprint pattern="^250[ -] *XEXCH50.*$">
-      <description>
-         Microsoft Exchange/IIS server
-      </description>
-      <param pos="0" name="service.vendor" value="Microsoft"/>
-      <param pos="0" name="service.family" value="IIS"/>
-      <param pos="0" name="service.product" value="IIS"/>
-      <param pos="0" name="os.vendor" value="Microsoft"/>
-      <param pos="0" name="os.family" value="Windows"/>
-      <param pos="0" name="os.device" value="General"/>
-      <param pos="0" name="os.product" value="Windows"/>
-   </fingerprint>
-   -->
-
-   <fingerprint pattern="^221[ -]See ya in cyberspace$">
-      <description>
-         221 See ya in cyberspace
-      </description>
-      <param pos="0" name="service.vendor" value="Alt-N"/>
-      <param pos="0" name="service.family" value="MDaemon"/>
-      <param pos="0" name="service.product" value="MDaemon"/>
-      <param pos="0" name="os.vendor" value="Microsoft"/>
-      <param pos="0" name="os.family" value="Windows"/>
-      <param pos="0" name="os.device" value="General"/>
-      <param pos="0" name="os.product" value="Windows"/>
-      <param pos="0" name="os.arch" value="x86"/>
-   </fingerprint>
 </fingerprints>

data/features/xml/successful_tests.xml

@@ -7,27 +7,12 @@
       <param pos="0" name="os.product" value="IOS"/>
       <param pos="1" name="os.version"/>
    </fingerprint>
-   <fingerprint pattern="^Microsoft Exchange Server 2007 IMAP4 service ready$">
-      <!--  Microsoft Exchange Server 2007 IMAP4 service ready
-        -->
-      <description>Microsoft Exchange Server 2007</description>
-      <param pos="0" name="service.vendor" value="Microsoft"/>
-      <param pos="0" name="service.family" value="Exchange Server"/>
-      <param pos="0" name="service.product" value="Exchange 2007 Server"/>
-      <param pos="0" name="os.vendor" value="Microsoft"/>
-      <param pos="0" name="os.device" value="General"/>
-      <param pos="0" name="os.family" value="Windows"/>
-      <param pos="0" name="os.product" value="Windows"/>
-    </fingerprint>
-    <fingerprint pattern="^The Microsoft Exchange IMAP4 service is ready\.?$">
-      <example>The Microsoft Exchange IMAP4 service is ready.</example>
-      <description>Microsoft Exchange Server</description>
-      <param pos="0" name="service.vendor" value="Microsoft"/>
-      <param pos="0" name="service.family" value="Exchange Server"/>
-      <param pos="0" name="service.product" value="Exchange Server"/>
-      <param pos="0" name="os.vendor" value="Microsoft"/>
-      <param pos="0" name="os.device" value="General"/>
-      <param pos="0" name="os.family" value="Windows"/>
-      <param pos="0" name="os.product" value="Windows"/>
+   <fingerprint pattern="^bar ([\d.]+)$">
+     <description>bar test</description>
+     <example os.version="1.0" >bar 1.0</example>
+     <example os.version="2.0" >bar 2.0</example>
+     <example os.version="2.1" >bar 2.1</example>
+     <param pos="1" name="os.version" />
+     <param pos="0" name="os.name" value="Bar" />
    </fingerprint>
 </fingerprints>

data/features/xml/tests_with_failures.xml

@@ -2,9 +2,19 @@
 <fingerprints>
    <fingerprint pattern="^foo$">
      <description>foo test</description>
+     <!-- Fail: doesn't match -->
      <example>bar</example>
    </fingerprint>
    <fingerprint pattern="^This matches$">
+     <!-- Warn: no name -->
+     <!-- Fail: doesn't match -->
      <example>This almost matches</example>
    </fingerprint>
+   <fingerprint pattern="^bar ([\d.]+)$">
+     <description>bar test</description>
+     <!-- Fail: expected os.version doesn't match the capture group -->
+     <example os.version="5.0" >bar 1.0</example>
+     <param pos="1" name="os.version" />
+     <param pos="0" name="os.name" value="Bar" />
+   </fingerprint>
 </fingerprints>

data/features/xml/tests_with_warnings.xml

@@ -7,4 +7,11 @@
     <param pos="0" name="service.family" value="Pure-FTPd"/>
     <param pos="0" name="service.product" value="Pure-FTPd"/>
   </fingerprint>
+  <fingerprint pattern="^-{10} Welcome to Pure-FTPd (.*)-{10}$">
+    <!-- should warn with no examples -->
+    <description>Pure-FTPd</description>
+    <param pos="1" name="pureftpd.config"/>
+    <param pos="0" name="service.family" value="Pure-FTPd"/>
+    <param pos="0" name="service.product" value="Pure-FTPd"/>
+  </fingerprint>
 </fingerprints>

data/lib/recog/db.rb

@@ -1,38 +1,54 @@
 module Recog
+
+# A collection of {Fingerprint fingerprints} for matching against a particular
+# kind of fingerprintable data, e.g. an HTTP `Server` header
 class DB
   require 'nokogiri'
   require 'recog/fingerprint'
 
-  attr_accessor :path, :fingerprints, :match_key
+  # @return [String]
+  attr_reader :path
+
+  # @return [Array<Fingerprint>] {Fingerprint} objects that can be matched
+  #   against strings that make sense for the {#match_key}
+  attr_reader :fingerprints
+
+  # @return [String] Taken from the `fingerprints/matches` element, or
+  #   defaults to the basename of {#path} without the `.xml` extension.
+  attr_reader :match_key
 
+  # @param path [String]
   def initialize(path)
-    self.path = path
+    @match_key = nil
+    @path = path
+    @fingerprints = []
+
     parse_fingerprints
   end
 
+  # @return [void]
   def parse_fingerprints
-    self.fingerprints = []
     xml = nil
-    
+
     File.open(self.path, "rb") do |fd|
-      xml = Nokogiri::XML( fd.read(fd.stat.size))
+      xml = Nokogiri::XML(fd.read(fd.stat.size))
     end
 
     xml.xpath("/fingerprints").each do |fbase|
       if fbase['matches']
-        self.match_key = fbase['matches'].to_s
+        @match_key = fbase['matches'].to_s
       end
     end
 
-    unless self.match_key
-      self.match_key = File.basename(self.path).sub(/\.xml$/, '')
+    unless @match_key
+      @match_key = File.basename(self.path).sub(/\.xml$/, '')
     end
 
     xml.xpath("/fingerprints/fingerprint").each do |fprint|
-      fingerprints << Fingerprint.new(fprint)
+      @fingerprints << Fingerprint.new(fprint)
     end
 
     xml = nil
   end
 end
-end
+end

data/lib/recog/db_manager.rb

@@ -24,4 +24,4 @@ class DBManager
   end
 
 end
-end
+end

data/lib/recog/fingerprint.rb

@@ -1,60 +1,144 @@
 module Recog
+
+# A fingerprint that can be {#match matched} against a particular kind of
+# fingerprintable data, e.g. an HTTP `Server` header
 class Fingerprint
-  attr_reader :name, :regex, :params, :tests
+  require 'recog/fingerprint/regexp_factory'
+  require 'recog/fingerprint/test'
+
+  # A human readable name describing this fingerprint
+  # @return (see #parse_description)
+  attr_reader :name
+
+  # Regular expression pulled from the {DB} xml file.
+  #
+  # @see #create_regexp
+  # @return [Regexp] the Regexp to try when calling {#match}
+  attr_reader :regex
+
+  # Collection of indexes for capture groups created by {#match}
+  #
+  # @return (see #parse_params)
+  attr_reader :params
+
+  # Collection of example strings that should {#match} our {#regex}
+  #
+  # @return (see #parse_examples)
+  attr_reader :tests
 
+  # @param xml [Nokogiri::XML::Element]
   def initialize(xml)
-    @name   = description(xml)
+    @name   = parse_description(xml)
     @regex  = create_regexp(xml)
-    @params = parse_params(xml)
-    @tests  = examples(xml)
+    @params = {}
+    @tests = []
+
+    parse_examples(xml)
+    parse_params(xml)
   end
 
-  private
+  # Attempt to match the given string.
+  #
+  # @param match_string [String]
+  # @return [Hash,nil] Keys will be host, service, and os attributes
+  def match(match_string)
+    match_data = @regex.match(match_string)
+    return if match_data.nil?
 
-  def description(xml)
-    element = xml.xpath('description')
-    element.empty? ? '' : element.first.content 
+    result = { 'matched' => @name }
+    @params.each_pair do |k,v|
+      if v[0] == 0
+        # A match offset of 0 means this param has a hardcoded value
+        result[k] = v[1]
+      else
+        result[k] = match_data[ v[0] ]
+      end
+    end
+    return result
+  end
+
+  # Ensure all the {#tests} actually match the fingerprint and return the
+  # expected capture groups.
+  #
+  # @yieldparam status [Symbol] One of `:warn`, `:fail`, or `:success` to
+  #   indicate whether a test worked
+  # @yieldparam message [String] A human-readable string explaining the
+  #   `status`
+  def verify_tests(&block)
+    if tests.size == 0
+      yield :warn, "'#{@name}' has no test cases"
+    end
+
+    tests.each do |test|
+      result = match(test.content)
+      if result.nil?
+        yield :fail, "'#{@name}' failed to match #{test.content.inspect} with #{@regex}'"
+        next
+      end
+
+      message = test
+      status = :success
+      # Ensure that all the attributes as provided by the example were parsed
+      # out correctly and match the capture group values we expect.
+      test.attributes.each do |k, v|
+        if !result.has_key?(k) || result[k] != v
+          message = "'#{@name}' failed to find expected capture group #{k} '#{v}'"
+          status = :fail
+          break
+        end
+      end
+      yield status, message
+    end
   end
 
+  private
+
+  # @param xml [Nokogiri::XML::Element]
+  # @return [Regexp]
   def create_regexp(xml)
     pattern = xml['pattern']
     flags   = xml['flags'].to_s.split(',')
     RegexpFactory.build(pattern, flags)
   end
 
-  def parse_params(xml)
-    {}.tap do |h|
-      xml.xpath('param').each do |e|
-        name  = e['name']
-        pos   = e['pos'].to_i
-        value = e['value'].to_s
-        h[name] = [pos, value]
-      end
-    end
+  # @param xml [Nokogiri::XML::Element]
+  # @return [String] Contents of the source XML's `description` tag
+  def parse_description(xml)
+    element = xml.xpath('description')
+    element.empty? ? '' : element.first.content
   end
 
-  def examples(xml)
-    xml.xpath('example').collect(&:content)
-  end
+  # @param xml [Nokogiri::XML::Element]
+  # @return [void]
+  def parse_examples(xml)
+    elements = xml.xpath('example')
 
-  module RegexpFactory
-    def self.build(pattern, flags)
-      options = build_options(flags)
-      Regexp.new(pattern, options)
+    elements.each do |elem|
+      # convert nokogiri Attributes into a hash of name => value
+      attrs = elem.attributes.values.reduce({}) { |a,e| a.merge(e.name => e.value) }
+      @tests << Test.new(elem.content, attrs)
     end
 
-    def self.build_options(flags)
-      rflags = Regexp::NOENCODING
-      flags.each do |flag|
-        case flag
-        when 'REG_DOT_NEWLINE', 'REG_LINE_ANY_CRLF'
-          rflags |= Regexp::MULTILINE
-        when 'REG_ICASE'
-          rflags |= Regexp::IGNORECASE 
-        end
+    nil
+  end
+
+  # @param xml [Nokogiri::XML::Element]
+  # @return [Hash<String,Array>] Keys are things like `"os.name"`, values are a two
+  #   element Array. The first element is an index for the capture group that returns
+  #   that thing. If the index is 0, the second element is a static value for
+  #   that thing; otherwise it is undefined.
+  def parse_params(xml)
+    @params = {}.tap do |h|
+      xml.xpath('param').each do |param|
+        name  = param['name']
+        pos   = param['pos'].to_i
+        value = param['value'].to_s
+        h[name] = [pos, value]
       end
-      rflags
     end
+
+    nil
   end
+
 end
 end