recog 2.3.22 → 3.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5e33013be558344280c798bb77321b809289a604f55a9f8447bba1cdd2b93151
-  data.tar.gz: 4088c7af5a4dc2250562a610f2bbc6307436fefabe1203f1bd00fd2a9f9c0e3d
+  metadata.gz: fb6f2415bc93d48b60c359793b6aee1e30e513b16a35de7c7bda4373a99cb996
+  data.tar.gz: 106fdf45bfe575ad27e6d9bc71ef2192967d573d00e7eda9a51c1ac50a40b964
 SHA512:
-  metadata.gz: abca30fbb5b218e69a2178c0b9b7337af15f1611d67076031ee5d489140505c558fe8981513d8d446bddc91b4166f304b88fed9eb23e22de0d4a5e10f2bfb668
-  data.tar.gz: 57c7e248435b5d52860cd2117a814176673c0e8cb193f854880b3244c91d9b745ee67ca1bc8f2f67f1f3046d26857ca198c81a0e129f9d50b8cc28349070d250
+  metadata.gz: fd7943c87258799dde4b700bc0a121f5f6973e371f9904d5442e2fd3b4615bc4012b97f66426a7f7a628409c94515f2490273eb83a6cce4d57ad2071b8df58ff
+  data.tar.gz: d8ccfa23a65f4aa5c2d1638647042c6e47e8e9c2ce409a3fb01bf825fd28341c133948b6fff2a3e0dddb7d978460e45a113b883b774ae62cd84b0ce378fb8fe7

checksums.yaml.gz.sig

@@ -0,0 +1,2 @@
+?K��RF|m��/�E�b��ζ�A�}"�RDRz��b�C:��yi�S]l�=�}!��a�B�e�dN���i�s��}��z�gbD;�m8@��+�D�A;�,�+t�z���ĀR��cS��K�[��I�!��}qϞ[�AqeXR� E��C���ނKg�{2>VL'�|{��!k_i�Bb>̚O�Y�W\�J�O���^�m�vh���I��U��mAr3�	E?p����
+�ܦ`T���2tI6SX

data/LICENSE

@@ -1,5 +1,5 @@
 Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
-Source: https://github.com/rapid7/recog
+Source: https://github.com/rapid7/recog-ruby
 
 Files: *
 Copyright: 2014, Rapid7, Inc.

data/README.md

@@ -1,41 +1,50 @@
-# Recog: A Recognition Framework
+# Recog-Ruby: A Recognition Framework
 
 [![Gem Version](https://badge.fury.io/rb/recog.svg)](http://badge.fury.io/rb/recog)
-[![Build Status](https://travis-ci.org/rapid7/recog.svg?branch=master)](https://travis-ci.org/rapid7/recog)
+[![CI Workflow](https://github.com/rapid7/recog-ruby/actions/workflows/ci.yml/badge.svg)](https://github.com/rapid7/recog-ruby/actions/workflows/ci.yml)
 
 
-Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, snmp system description fields, and a whole lot more.
+Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, SNMP system description fields, and a whole lot more.
 
-Recog is open source, please see the [LICENSE](https://raw.githubusercontent.com/rapid7/recog/master/LICENSE) file for more information.
+The Recog-Ruby repository contains the Ruby language implementation of the Recog recognition framework library and the [Recog](https://github.com/rapid7/recog) content, XML fingerprint files, as a git submodule. That makes it easy to develop, test, and use the contained fingerprints.
+
+Recog-Ruby is open source, please see the [LICENSE](LICENSE) file for more information.
 
 ## Table of Contents
 
+1. [Repository split](#repository-split)
 1. [Installation](#installation)
 1. [Maturity](#maturity)
 1. [Fingerprints](#fingerprints)
 1. [Contributing](#contributing)
 
+## Repository split
+
+On March 31, 2022, the Recog content - XML fingerprint files and utilities - were split from the Recog framework library implementation. The original [Recog](https://github.com/rapid7/recog) repository now contains the Recog content and the [Recog-Ruby](https://github.com/rapid7/recog-ruby) repository contains the Ruby language implementation. The Recog content is included in Recog-Ruby as a git submodule and is nested under the `recog` directory. All post-split Recog gem versions equal or greater than 3.0.0 will: 1. contain the XML fingerprint directory under the `recog` directory, and 2. only include the `recog_match` tool since the other tools are focused on fingerprint management.
+
+[^back to top](#recog-ruby-a-recognition-framework)
+
 ## Installation
 
-Recog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
+In order to use the included Ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
 
 ```shell
-$ git clone git@github.com:rapid7/recog.git
-$ cd recog
+$ git clone --recurse-submodules git@github.com:rapid7/recog-ruby.git
+$ cd recog-ruby
 $ bundle install
 ```
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)
 
 ## Maturity
 
-Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase in Recog is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.
+Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)
 
 ## Fingerprints
 
-The fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/rapid7/recog/blob/master/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
+The fingerprints within [Recog](https://github.com/rapid7/recog) are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/rapid7/recog/blob/master/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
 
 A fingerprint file consists of an XML document like the following:
 
@@ -69,12 +78,12 @@ The `param` elements contain a `pos` attribute, which indicates what capture fie
 
 The `example` string can be base64 encoded to permit the use of unprintable characters.  To signal this to Recog an `_encoding` attribute with the value of `base64` is added to the `example` element.  Based64 encoded text that is longer than 80 characters may be wrapped with newlines as shown below to aid in readability.
 
-````xml
+```xml
 <example _encoding="base64">
   dGllczGEAAAAlQQWMS4yLjg0MC4xMTM1NTYuMS40LjgwMAQuZGF0YS5yZW1vdmVkLjCEAAAAK
   AQdZG9tYWluQ29udHJvbGxlckZ1bmN0aW9uYWxpdHkxhAAAAAMEATc=
 </example>
-````
+```
 
 Additionally, examples can be placed in a directory with the same base name as the XML file, in the same directory as the XML file:
 
@@ -93,12 +102,12 @@ They can then be loaded using the `_filename` attribute:
 
 This is useful for long examples.
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)
 
 ## Contributing
 
-The users and maintainers of Recog would greatly appreciate any contributions
+The users and maintainers of Recog-Ruby would greatly appreciate any contributions
 you can make to the project. For guidelines and instructions please see
 [CONTRIBUTING.MD](CONTRIBUTING.md)
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)

data/Rakefile

@@ -2,7 +2,7 @@ require "bundler/gem_tasks"
 
 require 'rspec/core/rake_task'
 RSpec::Core::RakeTask.new do |t|
-    t.pattern = "spec/**/*_spec.rb"
+    t.pattern = ['spec/**/*_spec.rb', 'recog/spec/**/*_spec.rb']
 end
 
 require 'yard'
@@ -11,12 +11,5 @@ YARD::Rake::YardocTask.new do |t|
     t.files = ['lib/**/*.rb', '-', 'README.md']
 end
 
-require 'cucumber'
-require 'cucumber/rake/task'
-
-Cucumber::Rake::Task.new(:features) do |t|
-    t.cucumber_opts = "features --format pretty"
-end
-
 task :default => [ :tests, :yard ]
-task :tests => [ :spec, :features ]
+task :tests => [ :spec ]

data/lib/recog/db_manager.rb

@@ -5,7 +5,7 @@ class DBManager
 
   attr_accessor :path, :databases
 
-  DefaultDatabasePath = File.expand_path( File.join( File.dirname(__FILE__), "..", "..", "xml") )
+  DefaultDatabasePath = File.expand_path(File.join(File.expand_path(__dir__), ["..", "..", "recog", "xml"]))
 
   def initialize(path = DefaultDatabasePath)
     self.path = path

data/lib/recog/fingerprint.rb

@@ -5,6 +5,7 @@ module Recog
 class Fingerprint
   require 'set'
 
+  require 'recog/fingerprint_parse_error'
   require 'recog/fingerprint/regexp_factory'
   require 'recog/fingerprint/test'
 
@@ -28,20 +29,27 @@ class Fingerprint
   # @return (see #parse_examples)
   attr_reader :tests
 
+  # The line number of the XML entity in the source file for this
+  # fingerprint.
+  #
+  # @return [Integer] The line number of this entity.
+  attr_reader :line
+
   # @param xml [Nokogiri::XML::Element]
   # @param match_key [String] See Recog::DB
   # @param protocol [String] Protocol such as ftp, mssql, http, etc.
-  # @param filepath [String] Directory path for fingerprint example files
-  def initialize(xml, match_key=nil, protocol=nil, filepath=nil)
+  # @param example_path [String] Directory path for fingerprint example files
+  def initialize(xml, match_key=nil, protocol=nil, example_path=nil)
     @match_key = match_key
     @protocol = protocol
     @name   = parse_description(xml)
     @regex  = create_regexp(xml)
+    @line   = xml.line
     @params = {}
     @tests = []
 
     @protocol.downcase! if @protocol
-    parse_examples(xml, filepath)
+    parse_examples(xml, example_path)
     parse_params(xml)
   end
 
@@ -161,6 +169,7 @@ class Fingerprint
     # look for the presence of test cases
     if tests.size == 0
       yield :warn, "'#{@name}' has no test cases"
+      return
     end
 
     # make sure each test case passes
@@ -226,7 +235,7 @@ class Fingerprint
       if !param_used
         message = "'#{@name}' is missing an example that checks for parameter '#{param_name}' " +
                   "which is derived from a capture group"
-        yield :warn, message
+        yield :fail, message
       end
     end
   end
@@ -249,9 +258,9 @@ class Fingerprint
   end
 
   # @param xml [Nokogiri::XML::Element]
-  # @param filepath [String] Directory path for fingerprint example files
+  # @param example_path [String] Directory path for fingerprint example files
   # @return [void]
-  def parse_examples(xml, filepath)
+  def parse_examples(xml, example_path)
     elements = xml.xpath('example')
 
     elements.each do |elem|
@@ -259,7 +268,12 @@ class Fingerprint
       attrs = elem.attributes.values.reduce({}) { |a,e| a.merge(e.name => e.value) }
       if attrs["_filename"]
         contents = ""
-        fn = File.join(filepath, attrs["_filename"])
+        filename = attrs["_filename"]
+        fn = File.expand_path(File.join(example_path, filename))
+        unless fn.start_with?(File.expand_path(example_path) + File::Separator)
+          raise FingerprintParseError.new("an example specifies an illegal file path '#{filename}'", line_number = @line)
+        end
+
         File.open(fn, "rb") do |file|
           contents = file.read
           contents.force_encoding(Encoding::ASCII_8BIT)

data/lib/recog/fingerprint_parse_error.rb

@@ -0,0 +1,10 @@
+module Recog
+  class FingerprintParseError < StandardError
+    attr_reader :line_number
+
+    def initialize(msg, line_number=nil)
+      @line_number = line_number
+      super(msg)
+    end
+  end
+end

data/lib/recog/match_reporter.rb

@@ -1,3 +1,5 @@
+require 'json'
+
 module Recog
 class MatchReporter
   attr_reader :formatter
@@ -24,14 +26,46 @@ class MatchReporter
     @line_count += 1
   end
 
-  def match(text)
+  def match(match_data)
     @match_count += 1
-    formatter.success_message(text)
+    if @options.json_format
+      # remove data field from all matches and promote to a top-level field
+      data_field = match_data[0]["data"]
+      match_data.each { |h| h.delete("data") }
+      new_object = {
+        'data' => data_field,
+      }
+
+      if @options.multi_match
+        new_object['matches'] = match_data
+      else
+        new_object['match'] = match_data[0]
+      end
+      msg = new_object.to_json
+    else
+      match_prefix = match_data.size > 1 ? 'MATCHES' : 'MATCH'
+      msg = "#{match_prefix}: #{match_data.map(&:inspect).join(',')}"
+    end
+    formatter.success_message("#{msg}")
   end
 
   def failure(text)
     @fail_count += 1
-    formatter.failure_message(text)
+    if @options.json_format
+      new_object = {
+        'data' => text,
+        'match_failure' => true
+      }
+      if @options.multi_match
+        new_object['matches'] = nil
+      else
+        new_object['match'] = nil
+      end
+      msg = new_object.to_json
+    else
+      msg = "FAIL: #{text}"
+    end
+    formatter.failure_message("#{msg}")
   end
 
   def print_summary

data/lib/recog/matcher.rb

@@ -29,26 +29,21 @@ class Matcher
         line = line.to_s.unpack("C*").pack("C*").strip.gsub(/\\[rn]/, '')
         found_extractions = false
 
-        all_extractions = []
+        extraction_data = []
         fingerprints.each do |fp|
           extractions = fp.match(line)
           if extractions
             found_extractions = true
             extractions['data'] = line
-            if multi_match
-              all_extractions << extractions
-            else
-              reporter.match "MATCH: #{extractions.inspect}"
-              break
-            end
+            extraction_data << extractions
+            break unless multi_match
           end
         end
 
         if found_extractions
-          match_prefix = all_extractions.size > 1 ? 'MATCHES' : 'MATCH'
-          reporter.match "#{match_prefix}: #{all_extractions.map(&:inspect).join(',')}" if multi_match
+          reporter.match extraction_data
         else
-          reporter.failure "FAIL: #{line}"
+          reporter.failure line
         end
 
         if reporter.stop?

data/lib/recog/verifier.rb

@@ -15,9 +15,9 @@ class Verifier
         fp.verify_params do |status, message|
           case status
           when :warn
-            reporter.warning "WARN: #{message}"
+            reporter.warning message, fp.line
           when :fail
-            reporter.failure "FAIL: #{message}"
+            reporter.failure message, fp.line
           when :success
             reporter.success(message)
           end
@@ -25,9 +25,9 @@ class Verifier
         fp.verify_tests do |status, message|
           case status
           when :warn
-            reporter.warning "WARN: #{message}"
+            reporter.warning message, fp.line
           when :fail
-            reporter.failure "FAIL: #{message}"
+            reporter.failure message, fp.line
           when :success
             reporter.success(message)
           end

data/lib/recog/verify_reporter.rb

@@ -24,15 +24,15 @@ class VerifyReporter
     formatter.success_message("#{padding}#{text}") if detail?
   end
 
-  def warning(text)
+  def warning(text, line=nil)
     return unless @options.warnings
     @warning_count += 1
-    formatter.warning_message("#{path_label}#{padding}#{text}")
+    formatter.warning_message("#{path_label(line)}#{padding}WARN: #{text}")
   end
 
-  def failure(text)
+  def failure(text, line=nil)
     @failure_count += 1
-    formatter.failure_message("#{path_label}#{padding}#{text}")
+    formatter.failure_message("#{path_label(line)}#{padding}FAIL: #{text}")
   end
 
   def print_name(fingerprint)
@@ -65,9 +65,10 @@ class VerifyReporter
     @options.detail
   end
 
-  def path_label
+  def path_label(line=nil)
     unless detail?
-      @path.to_s.empty? ? "" : "#{@path}: "
+      line_label = line ? line.to_s + ":" : ""
+      @path.to_s.empty? ? "" : "#{@path}:#{line_label} "
     end
   end
 

data/lib/recog/version.rb

@@ -1,3 +1,3 @@
 module Recog
-  VERSION = '2.3.22'
+  VERSION = '3.0.2'
 end

data/{bin → recog/bin}/recog_match

@@ -1,12 +1,11 @@
 #!/usr/bin/env ruby
 
-$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), "..", "lib")))
 require 'optparse'
 require 'ostruct'
 require 'recog'
 require 'recog/matcher_factory'
 
-options = OpenStruct.new(color: false, detail: false, fail_fast: false, multi_match: false)
+options = OpenStruct.new(color: false, detail: false, json_format: false, fail_fast: false, multi_match: false)
 
 option_parser = OptionParser.new do |opts|
   opts.banner = "Usage: #{$0} [options] XML_FINGERPRINT_FILE [BANNERS_FILE]"
@@ -14,12 +13,18 @@ option_parser = OptionParser.new do |opts|
   opts.separator ""
   opts.separator "Options"
 
-  opts.on("-f", "--format FORMATTER",
+  opts.on("-f", "--format FORMATTER", [:summary, :detail, :json],
           "Choose a formatter.",
           "  [s]ummary (default - failure/match msgs)",
-          "  [d]etail  (msgs with total counts)") do |format|
-    if format.start_with? 'd'
+          "  [d]etail  (msgs with total counts)",
+          "  [j]son    (JSON failure/match msgs)") do |format|
+    if format == :summary
+      options.detail = false
+      options.json_format = false
+    elsif format == :detail
       options.detail = true
+    elsif format == :json
+      options.json_format = true
     end
   end
 
@@ -42,9 +47,17 @@ option_parser = OptionParser.new do |opts|
     exit
   end
 end
-option_parser.parse!(ARGV)
 
-if ARGV.count != 1 && ARGV.count != 2
+
+begin
+  option_parser.parse!(ARGV)
+rescue OptionParser::ParseError => e
+  puts e.message
+  puts option_parser
+  exit(1)
+end
+
+if ARGV.count < 1 || ARGV.count > 2
   puts option_parser
   exit(1)
 end