recog 2.3.21 → 3.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 972b7cc1ae69526b61f221eeefce61d192ccf4b1603342f94195bf6cd2ddba95
-  data.tar.gz: 807831da5cdfd3160bca893367c92f4b817514758b996968829253f548d19709
+  metadata.gz: ca2e3909b1b60418e80721625975ab8025e8dcaaaee59754dccb128b25661f00
+  data.tar.gz: 68be49a8c47fc1bb2b6a0f64d02c18a8fc29b67ee60b62060542be2216979e5c
 SHA512:
-  metadata.gz: a923e57f1f34fb74358756372fc3d3d08c20f0a0b9b1088905f57f4b09a2a56b2d9d4940d39e609ff50c164d15e285c9e1707032864d55f969a7ed4d72e68de5
-  data.tar.gz: 62d17cd2cdf9c3a6d35b36e4ace9c20744d42cfd99a2e90f65e463540fceb551f35f7572179af261c8881116aeadc51986e371631dc451df31081d52d79a58c1
+  metadata.gz: 2e0f749e0a8b313a8ba9845c39aa72ce08108e4c5b4e3785d689ae9e726aeb72b31b23da6e1e4610fff4dcd1caad966ee9578ab887a425cf2f414ee377b7b001
+  data.tar.gz: 7714d9bca0d70f27efd756460f0f167a46c69ee45c38da8a70bdfb5271256decb92375c8281efe67529e5b7e3a897b3e3e2fc20b7a62075b12db9abf940635a6

data/LICENSE

@@ -1,5 +1,5 @@
 Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
-Source: https://github.com/rapid7/recog
+Source: https://github.com/rapid7/recog-ruby
 
 Files: *
 Copyright: 2014, Rapid7, Inc.

data/README.md

@@ -1,41 +1,50 @@
-# Recog: A Recognition Framework
+# Recog-Ruby: A Recognition Framework
 
 [![Gem Version](https://badge.fury.io/rb/recog.svg)](http://badge.fury.io/rb/recog)
-[![Build Status](https://travis-ci.org/rapid7/recog.svg?branch=master)](https://travis-ci.org/rapid7/recog)
+[![CI Workflow](https://github.com/rapid7/recog-ruby/actions/workflows/ci.yml/badge.svg)](https://github.com/rapid7/recog-ruby/actions/workflows/ci.yml)
 
 
-Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, snmp system description fields, and a whole lot more.
+Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, SNMP system description fields, and a whole lot more.
 
-Recog is open source, please see the [LICENSE](https://raw.githubusercontent.com/rapid7/recog/master/LICENSE) file for more information.
+The Recog-Ruby repository contains the Ruby language implementation of the Recog recognition framework library and the [Recog](https://github.com/rapid7/recog) content, XML fingerprint files, as a git submodule. That makes it easy to develop, test, and use the contained fingerprints.
+
+Recog-Ruby is open source, please see the [LICENSE](LICENSE) file for more information.
 
 ## Table of Contents
 
+1. [Repository split](#repository-split)
 1. [Installation](#installation)
 1. [Maturity](#maturity)
 1. [Fingerprints](#fingerprints)
 1. [Contributing](#contributing)
 
+## Repository split
+
+On March 31, 2022, the Recog content - XML fingerprint files and utilities - were split from the Recog framework library implementation. The original [Recog](https://github.com/rapid7/recog) repository now contains the Recog content and the [Recog-Ruby](https://github.com/rapid7/recog-ruby) repository contains the Ruby language implementation. The Recog content is included in Recog-Ruby as a git submodule and is nested under the `recog` directory. All post-split Recog gem versions equal or greater than 3.0.0 will: 1. contain the XML fingerprint directory under the `recog` directory, and 2. only include the `recog_match` tool since the other tools are focused on fingerprint management.
+
+[^back to top](#recog-ruby-a-recognition-framework)
+
 ## Installation
 
-Recog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
+In order to use the included Ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
 
 ```shell
-$ git clone git@github.com:rapid7/recog.git
-$ cd recog
+$ git clone --recurse-submodules git@github.com:rapid7/recog-ruby.git
+$ cd recog-ruby
 $ bundle install
 ```
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)
 
 ## Maturity
 
-Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase in Recog is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.
+Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)
 
 ## Fingerprints
 
-The fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/rapid7/recog/blob/master/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
+The fingerprints within [Recog](https://github.com/rapid7/recog) are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/rapid7/recog/blob/master/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
 
 A fingerprint file consists of an XML document like the following:
 
@@ -69,19 +78,36 @@ The `param` elements contain a `pos` attribute, which indicates what capture fie
 
 The `example` string can be base64 encoded to permit the use of unprintable characters.  To signal this to Recog an `_encoding` attribute with the value of `base64` is added to the `example` element.  Based64 encoded text that is longer than 80 characters may be wrapped with newlines as shown below to aid in readability.
 
-````xml
+```xml
 <example _encoding="base64">
   dGllczGEAAAAlQQWMS4yLjg0MC4xMTM1NTYuMS40LjgwMAQuZGF0YS5yZW1vdmVkLjCEAAAAK
   AQdZG9tYWluQ29udHJvbGxlckZ1bmN0aW9uYWxpdHkxhAAAAAMEATc=
 </example>
-````
+```
+
+Additionally, examples can be placed in a directory with the same base name as the XML file, in the same directory as the XML file:
+
+```
+xml/services.xml
+xml/services/file1
+xml/services/file2
+...
+```
+
+They can then be loaded using the `_filename` attribute:
+
+```xml
+<example _filename="file1"/>
+```
+
+This is useful for long examples.
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)
 
 ## Contributing
 
-The users and maintainers of Recog would greatly appreciate any contributions
+The users and maintainers of Recog-Ruby would greatly appreciate any contributions
 you can make to the project. For guidelines and instructions please see
 [CONTRIBUTING.MD](CONTRIBUTING.md)
 
-[^back to top](#recog-a-recognition-framework)
+[^back to top](#recog-ruby-a-recognition-framework)

data/Rakefile

@@ -2,7 +2,7 @@ require "bundler/gem_tasks"
 
 require 'rspec/core/rake_task'
 RSpec::Core::RakeTask.new do |t|
-    t.pattern = "spec/**/*_spec.rb"
+    t.pattern = ['spec/**/*_spec.rb', 'recog/spec/**/*_spec.rb']
 end
 
 require 'yard'
@@ -11,12 +11,5 @@ YARD::Rake::YardocTask.new do |t|
     t.files = ['lib/**/*.rb', '-', 'README.md']
 end
 
-require 'cucumber'
-require 'cucumber/rake/task'
-
-Cucumber::Rake::Task.new(:features) do |t|
-    t.cucumber_opts = "features --format pretty"
-end
-
 task :default => [ :tests, :yard ]
-task :tests => [ :spec, :features ]
+task :tests => [ :spec ]

data/lib/recog/db.rb

@@ -66,10 +66,11 @@ class DB
 
     end
 
+    filepath =  self.path.sub(/\.xml$/, '')
     @match_key = File.basename(self.path).sub(/\.xml$/, '') unless @match_key
 
     xml.xpath('/fingerprints/fingerprint').each do |fprint|
-      @fingerprints << Fingerprint.new(fprint, @match_key, @protocol)
+      @fingerprints << Fingerprint.new(fprint, @match_key, @protocol, filepath)
     end
 
     xml = nil

data/lib/recog/db_manager.rb

@@ -5,7 +5,7 @@ class DBManager
 
   attr_accessor :path, :databases
 
-  DefaultDatabasePath = File.expand_path( File.join( File.dirname(__FILE__), "..", "..", "xml") )
+  DefaultDatabasePath = File.expand_path(File.join(File.expand_path(__dir__), ["..", "..", "recog", "xml"]))
 
   def initialize(path = DefaultDatabasePath)
     self.path = path

data/lib/recog/fingerprint.rb

@@ -5,6 +5,7 @@ module Recog
 class Fingerprint
   require 'set'
 
+  require 'recog/fingerprint_parse_error'
   require 'recog/fingerprint/regexp_factory'
   require 'recog/fingerprint/test'
 
@@ -28,19 +29,27 @@ class Fingerprint
   # @return (see #parse_examples)
   attr_reader :tests
 
+  # The line number of the XML entity in the source file for this
+  # fingerprint.
+  #
+  # @return [Integer] The line number of this entity.
+  attr_reader :line
+
   # @param xml [Nokogiri::XML::Element]
   # @param match_key [String] See Recog::DB
   # @param protocol [String] Protocol such as ftp, mssql, http, etc.
-  def initialize(xml, match_key=nil, protocol=nil)
+  # @param example_path [String] Directory path for fingerprint example files
+  def initialize(xml, match_key=nil, protocol=nil, example_path=nil)
     @match_key = match_key
     @protocol = protocol
     @name   = parse_description(xml)
     @regex  = create_regexp(xml)
+    @line   = xml.line
     @params = {}
     @tests = []
 
     @protocol.downcase! if @protocol
-    parse_examples(xml)
+    parse_examples(xml, example_path)
     parse_params(xml)
   end
 
@@ -160,6 +169,7 @@ class Fingerprint
     # look for the presence of test cases
     if tests.size == 0
       yield :warn, "'#{@name}' has no test cases"
+      return
     end
 
     # make sure each test case passes
@@ -176,6 +186,7 @@ class Fingerprint
       # out correctly and match the capture group values we expect.
       test.attributes.each do |k, v|
         next if k == '_encoding'
+        next if k == '_filename'
         if !result.has_key?(k) || result[k] != v
           message = "'#{@name}' failed to find expected capture group #{k} '#{v}'. Result was #{result[k]}"
           status = :fail
@@ -223,8 +234,8 @@ class Fingerprint
     capture_group_used.each do |param_name, param_used|
       if !param_used
         message = "'#{@name}' is missing an example that checks for parameter '#{param_name}' " +
-                  "messsage which is derived from a capture group"
-        yield :warn, message
+                  "which is derived from a capture group"
+        yield :fail, message
       end
     end
   end
@@ -247,14 +258,30 @@ class Fingerprint
   end
 
   # @param xml [Nokogiri::XML::Element]
+  # @param example_path [String] Directory path for fingerprint example files
   # @return [void]
-  def parse_examples(xml)
+  def parse_examples(xml, example_path)
     elements = xml.xpath('example')
 
     elements.each do |elem|
       # convert nokogiri Attributes into a hash of name => value
       attrs = elem.attributes.values.reduce({}) { |a,e| a.merge(e.name => e.value) }
-      @tests << Test.new(elem.content, attrs)
+      if attrs["_filename"]
+        contents = ""
+        filename = attrs["_filename"]
+        fn = File.expand_path(File.join(example_path, filename))
+        unless fn.start_with?(File.expand_path(example_path) + File::Separator)
+          raise FingerprintParseError.new("an example specifies an illegal file path '#{filename}'", line_number = @line)
+        end
+
+        File.open(fn, "rb") do |file|
+          contents = file.read
+          contents.force_encoding(Encoding::ASCII_8BIT)
+        end
+        @tests << Test.new(contents, attrs)
+      else
+        @tests << Test.new(elem.content, attrs)
+      end
     end
 
     nil

data/lib/recog/fingerprint_parse_error.rb

@@ -0,0 +1,10 @@
+module Recog
+  class FingerprintParseError < StandardError
+    attr_reader :line_number
+
+    def initialize(msg, line_number=nil)
+      @line_number = line_number
+      super(msg)
+    end
+  end
+end

data/lib/recog/verifier.rb

@@ -1,23 +1,23 @@
 module Recog
 class Verifier
-  attr_reader :fingerprints, :reporter
+  attr_reader :db, :reporter
 
-  def initialize(fingerprints, reporter)
-    @fingerprints = fingerprints
+  def initialize(db, reporter)
+    @db = db
     @reporter = reporter
   end
 
   def verify
-    reporter.report(fingerprints.count) do
-      fingerprints.each do |fp|
+    reporter.report(db.fingerprints.count) do
+      db.fingerprints.each do |fp|
         reporter.print_name fp
 
         fp.verify_params do |status, message|
           case status
           when :warn
-            reporter.warning "WARN: #{message}"
+            reporter.warning message, fp.line
           when :fail
-            reporter.failure "FAIL: #{message}"
+            reporter.failure message, fp.line
           when :success
             reporter.success(message)
           end
@@ -25,9 +25,9 @@ class Verifier
         fp.verify_tests do |status, message|
           case status
           when :warn
-            reporter.warning "WARN: #{message}"
+            reporter.warning message, fp.line
           when :fail
-            reporter.failure "FAIL: #{message}"
+            reporter.failure message, fp.line
           when :success
             reporter.success(message)
           end

data/lib/recog/verify_reporter.rb

@@ -3,14 +3,18 @@ class VerifyReporter
   attr_reader :formatter
   attr_reader :success_count, :warning_count, :failure_count
 
-  def initialize(options, formatter)
+  def initialize(options, formatter, path=nil)
     @options = options
     @formatter = formatter
+    @path = path
     reset_counts
   end
 
   def report(fingerprint_count)
     reset_counts
+    if detail? and !@path.to_s.empty?
+      formatter.status_message("\n#{@path}:\n")
+    end
     yield self
     summarize(fingerprint_count) unless @options.quiet
   end
@@ -20,15 +24,15 @@ class VerifyReporter
     formatter.success_message("#{padding}#{text}") if detail?
   end
 
-  def warning(text)
+  def warning(text, line=nil)
     return unless @options.warnings
     @warning_count += 1
-    formatter.warning_message("#{padding}#{text}")
+    formatter.warning_message("#{path_label(line)}#{padding}WARN: #{text}")
   end
 
-  def failure(text)
+  def failure(text, line=nil)
     @failure_count += 1
-    formatter.failure_message("#{padding}#{text}")
+    formatter.failure_message("#{path_label(line)}#{padding}FAIL: #{text}")
   end
 
   def print_name(fingerprint)
@@ -61,12 +65,19 @@ class VerifyReporter
     @options.detail
   end
 
+  def path_label(line=nil)
+    unless detail?
+      line_label = line ? line.to_s + ":" : ""
+      @path.to_s.empty? ? "" : "#{@path}:#{line_label} "
+    end
+  end
+
   def padding
     '   ' if @options.detail
   end
 
   def summary_line
-    summary = "SUMMARY: Test completed with "
+    summary = "#{path_label}SUMMARY: Test completed with "
     summary << "#{@success_count} successful"
     summary << ", #{@warning_count} warnings"
     summary << ", and #{@failure_count} failures"

data/lib/recog/version.rb

@@ -1,3 +1,3 @@
 module Recog
-  VERSION = '2.3.21'
+  VERSION = '3.0.1'
 end

data/{bin → recog/bin}/recog_match

@@ -1,6 +1,5 @@
 #!/usr/bin/env ruby
 
-$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), "..", "lib")))
 require 'optparse'
 require 'ostruct'
 require 'recog'