recog 2.3.3 → 2.3.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3156a27b14a7ca5bedb824bb3ea9077cde3b2144c3627e02fa346ea0ee69fa0c
-  data.tar.gz: af693b3ff10691b7817096b9c79594815be02fb9a0018f207846896eed0786c2
+  metadata.gz: be02bd17e124bbded970024eb6634e60ce5a3764da67faecb0da21157179d6ed
+  data.tar.gz: 04eacbfe28e565b359b4b798f0ef171c97274f9d5a43f273b8f09972b999ad46
 SHA512:
-  metadata.gz: d2c7faf0fe86fbc4d0ed51a1743b4d2872047937a4d76df6b47ab78d622be673cda1501c441d9c0fc49ad197d9f9bbd4b10a3224c3d8fc1ee52fe2d3a8bd5078
-  data.tar.gz: 5300fad8c8d9a238d4dbdf6c70c2ab206a2a0b741a4ffbd6d6c34891701f2ee6491cf9613e2859ddb8923bd384b92daf1f76229a8e02a7b8667adad161803652
+  metadata.gz: bc69e881e5a68c16227bff868480d9f68760ddb72ea8203e89c5ce2bef06a5558ba5a0a730f44e1976f4481b9c7e543b1ec2685a4dc9755b6cfa967d69fe8b66
+  data.tar.gz: 87a4cc900949a643cb89c7c2058939fb5e49a2d875b72687dc7cb739b3969fdbc6e5dafc7dceace9e4c89e16b0e94ac0f9b0327ef563a74ac7f247c7173f58ff

data/.gitignore

@@ -1,11 +1,17 @@
+# Ruby and tooling specific
 .yardoc
 coverage/
 doc/
 pkg/
-.idea/
-.vscode/
+
 /Gemfile.lock
 
-# ignore rvm files
-.ruby-version
-.ruby-gemset
+#Python specific
+venv
+
+# IDE specific
+.vscode/
+.idea
+
+# Misc
+**/.DS_Store

data/.ruby-gemset

@@ -0,0 +1 @@
+recog

data/.ruby-version

@@ -0,0 +1 @@
+2.6.6

data/.travis.yml

@@ -2,11 +2,14 @@ language: ruby
 sudo: false
 cache: bundler
 rvm:
-  - '2.3.8'
-  - '2.4.5'
-  - '2.5.3'
-  - '2.6.1'
+  - '2.5.8'
+  - '2.6.6'
   - 'jruby-9.1.9.0'
+jdk:
+  - openjdk8
+matrix:
+  allow_failures:
+    - rvm: 'jruby-9.1.9.0'
 before_install:
   - "echo 'gem: --no-ri --no-rdoc' > ~/.gemrc"
   - rake --version

data/Gemfile

@@ -1,13 +1,10 @@
 source 'https://rubygems.org'
 
-gemspec
+gemspec name: 'recog'
 
 gem 'nokogiri'
 
 group :test do
   gem 'rake'
-  gem 'rspec', '>= 2.99'
-  gem 'cucumber', '~> 1.3.8'
-  gem 'aruba', '~> 0.5.3'
-  gem 'regexp_parser', '~> 0.2.0'
+  gem 'regexp_parser'
 end

data/README.md

@@ -1,18 +1,21 @@
-Recog: A Recognition Framework
-=====
-
-Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, snmp system description fields, and a whole lot more. Recog is open source, please see the [LICENSE](https://raw.githubusercontent.com/rapid7/recog/master/LICENSE) file for more information.
-
+# Recog: A Recognition Framework
 [![Gem Version](https://badge.fury.io/rb/recog.svg)](http://badge.fury.io/rb/recog)
 [![Build Status](https://travis-ci.org/rapid7/recog.svg?branch=master)](https://travis-ci.org/rapid7/recog)
 
+
+Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, snmp system description fields, and a whole lot more.
+
+Recog is open source, please see the [LICENSE](https://raw.githubusercontent.com/rapid7/recog/master/LICENSE) file for more information.
+
 ## Installation
 
-Recog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (2.1+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
+Recog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
 
-    $ git clone git@github.com:rapid7/recog.git
-    $ cd recog
-    $ bundle install
+```shell
+$ git clone git@github.com:rapid7/recog.git
+$ cd recog
+$ bundle install
+```
 
 ## Maturity
 
@@ -24,7 +27,7 @@ The fingerprints within Recog are stored in XML files, each of which is designed
 
 A fingerprint file consists of an XML document like the following:
 
-```
+```xml
 <fingerprints matches="ssh.banner">
   <fingerprint pattern="^RomSShell_([\d\.]+)$">
     <description>Allegro RomSShell SSH</description>
@@ -36,15 +39,15 @@ A fingerprint file consists of an XML document like the following:
 </fingerprints>
 ```
 
-The first line should always consist of the XML version declaration. The first element should always be a `fingerpints` block with a `matches` attribute indicating what data this fingerprint file is supposed to match. The `matches` attribute is normally in the form of `protocol.field`.
+The first line should always consist of the XML version declaration. The first element should always be a `fingerprints` block with a `matches` attribute indicating what data this fingerprint file is supposed to match. The `matches` attribute is normally in the form of `protocol.field`.
 
 Inside of the `fingerprints` element there should be one or more `fingerprint` elements. Every `fingerprint` must contain a `pattern` attribute, which contains the regular expression to be used to match against the data.  An optional `flags` attribute can be specified to control how the regular expression is to be interpreted.  See [the Recog documentation for `FLAG_MAP`](http://www.rubydoc.info/gems/recog/Recog/Fingerprint/RegexpFactory#FLAG_MAP-constant) for more information.
 
 Inside of the fingerprint, a `description` element should contain a human-readable string describing this fingerprint.
 
-At least one `example` element should be present, however multiple `example` elements are preferred.  These elements are used as part of the test coverage present in rspec which validates that the provided data matches the specified regular expression.  Additionally, if the fingerprint is using the `param` elements to extract field values from the data (described next), you can add these expected extractions as attributes for the `example` elements.  In the example above, this:
+At least one `example` element should be present, however multiple `example` elements are preferred.  These elements are used as part of the test coverage present in `rspec` which validates that the provided data matches the specified regular expression.  Additionally, if the fingerprint is using the `param` elements to extract field values from the data (described next), you can add these expected extractions as attributes for the `example` elements.  In the example above, this:
 
-```
+```xml
 <example service.version="4.62">RomSShell_4.62</example>
 ```
 
@@ -54,7 +57,7 @@ The `param` elements contain a `pos` attribute, which indicates what capture fie
 
 The `example` string can be base64 encoded to permit the use of unprintable characters.  To signal this to Recog an `_encoding` attribute with the value of `base64` is added to the `example` element.  Based64 encoded text that is longer than 80 characters may be wrapped with newlines as shown below to aid in readability.
 
-````
+````xml
 <example _encoding="base64">
   dGllczGEAAAAlQQWMS4yLjg0MC4xMTM1NTYuMS40LjgwMAQuZGF0YS5yZW1vdmVkLjCEAAAAK
   AQdZG9tYWluQ29udHJvbGxlckZ1bmN0aW9uYWxpdHkxhAAAAAMEATc=
@@ -65,15 +68,15 @@ The `example` string can be base64 encoded to permit the use of unprintable char
 
 Once a fingerprint has been added, the `example` entries can be tested by executing `bin/recog_verify` against the fingerprint file:
 
-```
-    $ bin/recog_verify xml/ssh_banners.xml
+```shell
+$ bin/recog_verify xml/ssh_banners.xml
 ```
 
 Matches can be tested on the command-line in a similar fashion:
 
-```
-    $ echo 'OpenSSH_6.6p1 Ubuntu-2ubuntu1' | bin/recog_match xml/ssh_banners.xml -
-    MATCH: {"matched"=>"OpenSSH running on Ubuntu 14.04", "service.version"=>"6.6p1", "openssh.comment"=>"Ubuntu-2ubuntu1", "service.vendor"=>"OpenBSD", "service.family"=>"OpenSSH", "service.product"=>"OpenSSH", "os.vendor"=>"Ubuntu", "os.device"=>"General", "os.family"=>"Linux", "os.product"=>"Linux", "os.version"=>"14.04", "service.protocol"=>"ssh", "fingerprint_db"=>"ssh.banner", "data"=>"OpenSSH_6.6p1 Ubuntu-2ubuntu1"}
+```shell
+$ echo 'OpenSSH_6.6p1 Ubuntu-2ubuntu1' | bin/recog_match xml/ssh_banners.xml -
+MATCH: {"matched"=>"OpenSSH running on Ubuntu 14.04", "service.version"=>"6.6p1", "openssh.comment"=>"Ubuntu-2ubuntu1", "service.vendor"=>"OpenBSD", "service.family"=>"OpenSSH", "service.product"=>"OpenSSH", "os.vendor"=>"Ubuntu", "os.device"=>"General", "os.family"=>"Linux", "os.product"=>"Linux", "os.version"=>"14.04", "service.protocol"=>"ssh", "fingerprint_db"=>"ssh.banner", "data"=>"OpenSSH_6.6p1 Ubuntu-2ubuntu1"}
 ```
 
 ### Best Practices

data/bin/recog_standardize

@@ -0,0 +1,118 @@
+#!/usr/bin/env ruby
+
+$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), "..", "lib")))
+require 'optparse'
+require 'ostruct'
+require 'recog'
+
+def load_identifiers(path)
+  res = {}
+  File.readlines(path).map{|line| line.strip}.each do |ident|
+    res[ident] = true
+  end
+  return res
+end
+
+def write_identifiers(vals, path)
+  res = []
+  vals.each_pair do |k,v|
+    res = res.push(k)
+  end
+  res = res.sort.uniq
+  File.write(path, res.join("\n") + "\n")
+end
+
+bdir = File.expand_path(File.join(File.dirname(__FILE__), "..", "identifiers"))
+
+options = OpenStruct.new(write: false)
+option_parser = OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options] XML_FINGERPRINT_FILE1 ..."
+  opts.separator "Verifies that each fingerprint asserts known identifiers."
+  opts.separator ""
+  opts.separator "Options"
+
+  opts.on("-w", "--write") do
+      options.write = true
+  end
+
+  opts.on("-h", "--help", "Show this message.") do
+    puts opts
+    exit
+  end
+end
+option_parser.parse!(ARGV)
+
+if ARGV.empty?
+  $stderr.puts 'Missing XML fingerprint files'
+  puts option_parser
+  exit(1)
+end
+
+# Load the unique identifiers
+vendors = load_identifiers(File.join(bdir, "vendor.txt"))
+os_arch = load_identifiers(File.join(bdir, "os_architecture.txt"))
+os_prod = load_identifiers(File.join(bdir, "os_product.txt"))
+os_family = load_identifiers(File.join(bdir, "os_family.txt"))
+os_device = load_identifiers(File.join(bdir, "os_device.txt"))
+svc_prod = load_identifiers(File.join(bdir, "service_product.txt"))
+svc_family = load_identifiers(File.join(bdir, "service_family.txt"))
+
+ARGV.each do |arg|
+  Dir.glob(arg).each do |file|
+    ndb = Recog::DB.new(file)
+    ndb.fingerprints.each do |f|
+      f.params.each do |k,v|
+        paramIndex, val = v
+        next if paramIndex != 0
+        case k
+        when "os.vendor", "service.vendor", "service.component.vendor", "hw.vendor"
+          if ! vendors[val]
+            puts "VENDOR MISSING: #{val}"
+            vendors[val] = true
+          end
+        when "os.product"
+          if ! os_prod[val]
+            puts "OS PRODUCT MISSING: #{val}"
+            os_prod[val] = true
+          end
+        when "os.arch"
+          if ! os_arch[val]
+            puts "OS ARCH MISSING: #{val}"
+            os_arch[val] = true
+          end
+        when "os.family"
+          if ! os_family[val]
+            puts "OS FAMILY MISSING: #{val}"
+            os_family[val] = true
+          end
+        when "os.device"
+          if ! os_device[val]
+            puts "OS DEVICE MISSING: #{val}"
+            os_device[val] = true
+          end
+        when "service.product"
+          if ! svc_prod[val]
+            puts "SERVICE PRODUCT MISSING: #{val}"
+            svc_prod[val] = true
+          end            
+        when "service.family"
+          if ! svc_family[val]
+            puts "SERVICE FAMILY MISSING: #{val}"
+            svc_family[val] = true
+          end          
+        end
+      end
+    end
+  end
+end
+
+exit if ! options.write
+
+# Write back the unique identifiers
+write_identifiers(vendors, File.join(bdir, "vendor.txt"))
+write_identifiers(os_arch, File.join(bdir, "os_architecture.txt"))
+write_identifiers(os_prod, File.join(bdir, "os_product.txt"))
+write_identifiers(os_family, File.join(bdir, "os_family.txt"))
+write_identifiers(os_device, File.join(bdir, "os_device.txt"))
+write_identifiers(svc_prod, File.join(bdir, "service_product.txt"))
+write_identifiers(svc_family, File.join(bdir, "service_family.txt"))

data/cpe-remap.yaml

@@ -1,4 +1,6 @@
 mappings:
+  alpine:
+    vendor: alpinelinux
   apache:
     vendor: apache
     products:
@@ -45,10 +47,17 @@ mappings:
     vendor: ibm
     products:
       lotus_domino: lotus_domino_server
+      os/400: os_400
+      z/os: z\/os
+  jamf:
+    products:
+      jamf_pro: jamf
   juniper:
     vendor: juniper
     products:
       junos_os: junos
+  kibana:
+    vendor: elasticsearch
   linux:
     vendor: linux
     products:
@@ -94,6 +103,11 @@ mappings:
     vendor: paloaltonetworks
     products:
       pa_firewall: pan-os
+  parallels:
+    products:
+      plesk: parallels_plesk_panel
+  plesk:
+    vendor: parallels
   proftpd_project:
     vendor: proftpd
   realvnc_ltd.:
@@ -113,6 +127,13 @@ mappings:
     vendor: sun
     products:
       solaris: sunos
+  tandberg:
+    vendor: cisco
+  tightvnc:
+    products:
+      desktop: tightvnc
+  ubiquiti:
+    vendor: ui
   ubuntu:
     vendor: canonical
     products:

data/features/data/successful_tests.xml

@@ -2,7 +2,7 @@
 <fingerprints>
    <fingerprint pattern="^Cisco-SIPGateway/IOS-([\d\.x]+)$">
       <description>Cisco SIPGateway</description>
-      <example>Cisco-SIPGateway/IOS-12.x</example>
+      <example os.version="12.x">Cisco-SIPGateway/IOS-12.x</example>
       <param pos="0" name="os.vendor" value="Cisco"/>
       <param pos="0" name="os.product" value="IOS"/>
       <param pos="1" name="os.version"/>

data/features/data/tests_with_warnings.xml

@@ -1,7 +1,7 @@
 <?xml version="1.0"?>
 <fingerprints>
   <fingerprint pattern="^-{10} Welcome to Pure-FTPd (.*)-{10}$">
-    <example>---------- Welcome to Pure-FTPd ----------</example>
+    <example pureftpd.config="">---------- Welcome to Pure-FTPd ----------</example>
     <description>Pure-FTPd</description>
     <param pos="1" name="pureftpd.config"/>
     <param pos="0" name="service.family" value="Pure-FTPd"/>

data/features/match.feature

@@ -1,4 +1,5 @@
 Feature: Match
+  @no-clobber
   Scenario: Finds matches
     When I run `recog_match matching_banners_fingerprints.xml sample_banner.txt`
     Then it should pass with:
@@ -7,6 +8,7 @@ Feature: Match
       MATCH: {"matched"=>"SunOS/Solaris", "os.vendor"=>"Sun", "os.family"=>"Solaris", "os.product"=>"Solaris", "os.device"=>"General", "host.name"=>"polaris", "os.version"=>"5.8", "service.protocol"=>"ftp", "fingerprint_db"=>"matching_banners_fingerprints", "data"=>"polaris FTP server (SunOS 5.8) ready."}
       """
 
+  @no-clobber
   Scenario: Fails at finding matches
     When I run `recog_match failing_banners_fingerprints.xml sample_banner.txt`
     Then it should pass with:
@@ -15,6 +17,7 @@ Feature: Match
       FAIL: polaris FTP server (SunOS 5.8) ready
       """
 
+  @no-clobber
   Scenario: Finds multiple matches
     When I run `recog_match multiple_banners_fingerprints.xml sample_banner.txt --multi-match`
     Then it should pass with:
@@ -23,6 +26,7 @@ Feature: Match
       MATCHES: {"matched"=>"Generic FTP, Checks for the existence of the word FTP in the line", "service.protocol"=>"", "fingerprint_db"=>"multiple_banners_fingerprints", "data"=>"polaris FTP server (SunOS 5.8) ready."},{"matched"=>"SunOS/Solaris", "service.protocol"=>"ftp", "os.vendor"=>"Sun", "os.family"=>"Solaris", "os.product"=>"Solaris", "os.device"=>"General", "host.name"=>"polaris", "os.version"=>"5.8", "fingerprint_db"=>"multiple_banners_fingerprints", "data"=>"polaris FTP server (SunOS 5.8) ready."}
       """
 
+  @no-clobber
   Scenario: Finds first matches using no-multi-match flag
     When I run `recog_match multiple_banners_fingerprints.xml sample_banner.txt --no-multi-match`
     Then it should pass with:

data/features/support/aruba.rb

@@ -0,0 +1,3 @@
+Aruba.configure do |config|
+  config.working_directory = 'features/data'
+end

data/features/verify.feature

@@ -1,4 +1,5 @@
 Feature: Verify
+  @no-clobber
   Scenario: No tests
     When I run `recog_verify no_tests.xml`
     Then it should pass with:
@@ -6,6 +7,7 @@ Feature: Verify
       SUMMARY: Test completed with 0 successful, 0 warnings, and 0 failures
       """
 
+  @no-clobber
   Scenario: Successful tests
     When I run `recog_verify successful_tests.xml`
     Then it should pass with:
@@ -13,15 +15,18 @@ Feature: Verify
       SUMMARY: Test completed with 4 successful, 0 warnings, and 0 failures
       """
 
+  @no-clobber
   Scenario: Tests with warnings, warnings enabled
     When I run `recog_verify tests_with_warnings.xml`
     Then it should fail with:
       """
       WARN: 'Pure-FTPd' has no test cases
-      SUMMARY: Test completed with 1 successful, 1 warnings, and 0 failures
+      WARN: 'Pure-FTPd' is missing an example that checks for parameter 'pureftpd.config' messsage which is derived from a capture group
+      SUMMARY: Test completed with 1 successful, 2 warnings, and 0 failures
       """
-    And the exit status should be 1
+    And the exit status should be 2
 
+  @no-clobber
   Scenario: Tests with warnings, warnings disabled
     When I run `recog_verify --no-warnings tests_with_warnings.xml`
     Then it should pass with:
@@ -29,6 +34,7 @@ Feature: Verify
       SUMMARY: Test completed with 1 successful, 0 warnings, and 0 failures
       """
 
+  @no-clobber
   Scenario: Tests with failures
     When I run `recog_verify tests_with_failures.xml`
     Then it should fail with:
@@ -40,5 +46,3 @@ Feature: Verify
       SUMMARY: Test completed with 0 successful, 0 warnings, and 4 failures
       """
     And the exit status should be 4
-
-

data/identifiers/README.md

@@ -0,0 +1,47 @@
+# Recog: Identifiers
+
+This directory contains lists of standard identifiers for mapping Recog matches. The goal is define a standard set of constants to represent known software, hardware, vendors, and categories.
+
+This is currently incomplete and will be updated as standardization work moves forward.
+
+Fingerprints should use these identifiers whenever possible; if a different name or syntax for a given identifier is preferred, this should be implemented in the application through a mapping function.
+
+## Lists
+
+### Vendors
+
+`vendor.txt` defines known vendor names, covering services, operating systems, and hardware.
+
+### Operating Systems
+
+`os_architecture.txt` defines known CPU types.
+
+`os_product.txt` defines known operating system names.
+
+`os_family.txt` defines known operating system families.
+
+`os_device.txt` defines known types of devices by function or purpose.
+
+### Services
+
+`service_product.txt` defines known service product names.
+
+`service_family.txt` defines known service product families.
+
+### Software
+
+`software_product.txt` defines known software product names.
+
+`software_family.txt` defines known software product families.
+
+`software_class.txt` defines known types of software by function or purpose.
+
+## Pending Work
+
+  * All existing fingerprints should be correlated against these lists to identify mismatches and updated accordingly.
+
+  * All net new identifiers from the existing fingerprints should be merged into these lists.
+
+  * All fingerprint assertions should be enumerated, documented, and standardized where possible (`host.mac`, etc).
+
+  * Hardware identifiers should be enumerated, consolidated, and standardized.