recog-intrigue 2.3.7

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8ce6e9357161c0d9ccd4a78a1b29c32ad316d5ebdbc36d1c14c83e7642a4ee87
+  data.tar.gz: 55a033985a85110154d9eb1fda16a4a860ee6f4490cb162ed8e5ed5bdb8d8c8f
+SHA512:
+  metadata.gz: 625729df058c4a85767187b2244edc45a6b7f038f5cd620d7f986adb62744479cf9a4592b827dae962158b32dab3cd97463d237b25425016d87263e596dad304
+  data.tar.gz: ccfdfed40b6691f9477f29d311a1e41273a85cbb481693f4377ab0547ba0fed8ddf765d00c5a6348b9bef673205bf4afbdbc8a27b32fd25b01e82b11ff5a412c

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,37 @@
+---
+name: Bug report
+about: Create a report to help us improve
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+1. Do this...
+2. Do that...
+3. Then something happens...
+
+Code that reproduces the behavior: <!-- This is optional -->
+```ruby
+# paste code here, or create a gist, or link to public code snippet
+```
+
+Matcher that reproduces the behavior: <!-- This is optional -->
+```xml
+<!-- paste matcher xml here, or create a gist, or link to public code snippet -->
+```
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+
+**Environment (please complete the following information):**
+ - Operating System: 
+ - Ruby Version: 
+ - Recog Version: 
+
+**Additional context**
+Add any other context about the problem here.

data/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

data/.github/ISSUE_TEMPLATE/fingerprint_request.md

@@ -0,0 +1,27 @@
+---
+name: Fingerprint request
+about: Request new fingerprint coverage
+
+---
+
+**Request for new fingerprint(s) for a product**  
+Briefly describe the product to be fingerprinted, including vendor and version information.  
+<!-- Include links to relevant public documentation from the vendor or other sources, if available -->
+
+**Provide details about the product**  
+What protocol(s) can be used to retrieve identifiable information about the product?  
+<!-- Examples: Telnet, SSH, SNMP, SMTP, HTTP(S), Vendor Proprietary Protocol -->
+
+What information can be collected on each protocol?  
+<!-- Example: HTTP Server banner: nginx/0.8.53 -->
+
+What request, command, and/or payload can be used to retrieve information on each protocol?  
+<!-- Use code fences like the below example to preserve formatting -->
+```
+HTTP HEAD /
+
+HTTP GET /info.php
+```
+
+**Example banner(s) with specific version info**  
+<!-- Paste raw text here, using code fences to preserve formatting if needed-->

data/.github/PULL_REQUEST_TEMPLATE

@@ -0,0 +1,24 @@
+## Description
+A detailed description of your changes.
+
+
+## Motivation and Context
+Explanation of why these changes are being proposed, including any links to other relevant issues or pull requests.
+
+
+## How Has This Been Tested?
+A clear and concise description of your changes were tested.
+
+
+## Types of changes
+<!--- What types of changes does your code introduce? Remove any that do not apply: -->
+- Bug fix (non-breaking change which fixes an issue)
+- New feature (non-breaking change which adds functionality)
+- Breaking change (fix or feature that would cause existing functionality to change)
+
+
+## Checklist:
+<!--- After submitting the PR, check all of the boxes that apply. -->
+- [ ] I have updated the documentation accordingly (or changes are not required).
+- [ ] I have added tests to cover my changes (or new tests are not required).
+- [ ] All new and existing tests passed.

data/.gitignore

@@ -0,0 +1,14 @@
+# Ruby and tooling specific
+.yardoc
+coverage/
+doc/
+pkg/
+
+/Gemfile.lock
+
+# IDE specific
+.vscode/
+.idea
+
+# Misc
+**/.DS_Store

data/.rbenv-gemset

@@ -0,0 +1 @@
+recog

data/.rspec

@@ -0,0 +1,3 @@
+--color
+--warnings
+--require spec_helper

data/.ruby-gemset

@@ -0,0 +1 @@
+recog

data/.ruby-version

@@ -0,0 +1 @@
+2.6.6

data/.travis.yml

@@ -0,0 +1,25 @@
+language: ruby
+sudo: false
+cache: bundler
+rvm:
+  - '2.5.8'
+  - '2.6.6'
+  - 'jruby-9.1.9.0'
+jdk:
+  - openjdk8
+matrix:
+  allow_failures:
+    - rvm: 'jruby-9.1.9.0'
+before_install:
+  - "echo 'gem: --no-ri --no-rdoc' > ~/.gemrc"
+  - rake --version
+before_script:
+  - bundle exec rake --version
+script: bundle exec rake tests
+env:
+  global:
+    - CI="travis"
+    - JRUBY_OPTS="--server -J-Xms512m -J-Xmx2G"
+branches:
+  only:
+    - master

data/.yardopts

@@ -0,0 +1 @@
+--markup markdown

data/CONTRIBUTING.md

@@ -0,0 +1,171 @@
+# Contributing to Recog
+
+The users and maintainers of Recog would greatly appreciate any contributions
+you can make to the project.  These contributions typically come in the form of
+filed bugs/issues or pull requests (PRs).  These contributions routinely result
+in new versions of the [recog gem](https://rubygems.org/gems/recog) to be
+released.  The process for everything is described below.
+
+## Contributing Issues / Bug Reports
+
+If you encounter any bugs or problems with Recog, please file them
+[here](https://github.com/rapid7/recog/issues/new), providing as much detail as
+possible.  If the bug is straight-forward enough and you understand the fix for
+the bug well enough, you may take the simpler, less-paperwork route and simply
+fill a PR with the fix and the necessary details.
+
+## Contributing Code
+
+Recog uses a model nearly identical to that of
+[Metasploit](https://github.com/rapid7/metasploit-framework) as outlined
+[here](https://github.com/rapid7/metasploit-framework/wiki/Setting-Up-a-Metasploit-Development-Environment),
+at least from a ```git``` perspective.  If you've been through that process
+(or, even better, you've been through it many times with many people), you can
+do exactly what you did for Metasploit but with Recog and ignore the rest of
+this document.
+
+On the other hand, if you haven't, read on!
+
+### Fork and Clone
+
+Generally, this should only need to be done once, or if you need to start over.
+
+1. Fork Recog: Visit https://github.com/rapid7/recog and click Fork,
+   selecting your github account if prompted
+2.  Clone ```git@github.com:<your-github-username>/recog.git```, replacing
+```<your-github-username>``` with, you guessed it, your Github username.
+3.  Add the master Recog repository as your upstream:
+
+ ```
+   git remote add upstream git://github.com/rapid7/recog.git
+ ```
+4. Update your `.git/config` to ensure that the `remote ["upstream"]` section is configured to pull both branches and PRs from upstream.  It should look something like the following, in particular the second `fetch` option:
+
+    ```
+     [remote "upstream"]
+      url = git@github.com:rapid7/recog.git
+      fetch = +refs/heads/*:refs/remotes/upstream/*
+      fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*
+     ```
+5. Fetch the latest revisions, including PRs:
+
+    ```
+    git fetch --all
+    ```
+
+### Branch and Improve
+
+If you have a contribution to make, first create a branch to contain your
+work.  The name is yours to choose, however generally it should roughly
+describe what you are doing.  In this example, and from here on out, the
+branch will be FOO, but you should obviously change this:
+
+```
+git fetch --all
+git checkout master
+git rebase upstream/master
+git checkout -b FOO
+```
+
+Now, make your changes, commit as necessary with useful commit messages.
+
+Please note that changes to [lib/recog/version.rb](https://github.com/rapid7/recog/blob/master/lib/recog/version.rb) in PRs are almost never necessary.
+
+Now push your changes to your fork:
+
+```
+git push origin FOO
+```
+
+Finally, submit the PR.  Navigate to ```https://github.com/<your-github-username>/recog/compare/FOO```, fill in the details and submit.
+
+### Testing
+
+When your PR is submitted, it will be automatically subjected to the full run of tests in [Travis](https://travis-ci.org/rapid7/recog/), however you are encourage to perform testing _before_ submitting the PR.  To do this, simply run `rake tests`.
+
+## Updating CPEs
+
+There exists some automation to update the CPEs that might be asserted with
+some recog fingerprints.  This should be run periodically to ensure that all
+fingerprints that could have CPEs do, etc.
+
+First, setup a python3 venv:
+
+  ```
+  python3 -m venv venv
+  source venv/bin/activate
+  pip install -r requirements.txt
+  ```
+
+Download the latest CPE 2.3 dictionary:
+
+  ```
+  wget https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz
+  ````
+
+Run the CPE automation against every XML file, using GNU `parallel` to speed things up:
+
+  ```
+  ls xml/*.xml | parallel --gnu "./update_cpes.py {} official-cpe-dictionary_v2.3.xml cpe-remap.yaml && xmllint --format --noblanks {} > {}.bak &&  mv {}.bak {} || echo {}" 2> errors.txt
+  ```
+
+Any mismatched fingerprints will be listed in `errors.txt` for eventual
+maintenance.  The `cpe-remap.yaml` file can be used to map between
+vendor/product/etc differences between Recog and CPE, or to work around bugs in
+either.
+
+## Landing PRs
+
+(Note: this portion is a work-in-progress.  Please update it as things change)
+
+Much like with the process of submitting PRs, Recog's process for landing PRs
+is very similar to [Metasploit's process for landing
+PRs](https://github.com/rapid7/metasploit-framework/wiki/Landing-Pull-Requests).
+In short:
+
+1. Follow the "Fork and Clone" steps from above
+2. Update your `.git/config` to ensure that the `remote ["upstream"]` section is configured to pull both branches and PRs from upstream.  It should look something like the following, in particular the second `fetch` option:
+
+    ```
+     [remote "upstream"]
+      url = git@github.com:rapid7/recog.git
+      fetch = +refs/heads/*:refs/remotes/upstream/*
+      fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*
+     ```
+3. Fetch the latest revisions, including PRs:
+
+    ```
+    git fetch --all
+    ```
+4. Checkout and branch the PR for testing.  Replace ```PR``` below with the actual PR # in question:
+
+    ```
+    git checkout -b landing-PR upstream/pr/PR
+    ```
+5. Test the PR (see the Testing section above)
+6. Merge with master, re-test, validate and push:
+
+    ```
+    git checkout -b upstream-master --track upstream/master
+    git merge -S --no-ff --edit landing-PR # merge the PR into upstream-master
+    # re-test if/as necessary
+    git push upstream upstream-master:master --dry-run # confirm you are pushing what you expect
+    git push upstream upstream-master:master # push upstream-master to upstream:master
+    ```
+7. If applicable, release a new version (see next section)
+
+## Releasing New Versions
+
+When Recog's critical parts are modified, for example its fingerprints or underlying supporting code, a new version _must_ eventually be released.  These new releases can then be optionally included in projects such as Metasploit or products such as Rapid7's Nexpose in a controlled manner.  Releases for non-functional updates such as updates to documentation are not necessary.
+
+When a new version of Recog is to be released, you _must_ follow the instructions below.
+
+1. If are not already a Recog project contributor for the Recog gem (you'd be listed [here under OWNERS](https://rubygems.org/gems/recog)), become one:
+  1. Get an account on [Rubygems](https://rubygems.org)
+  2. Contact one of the Recog project contributors (listed [here under OWNERS](https://rubygems.org/gems/recog) and have them add you to the Recog gem.  They'll need to run:
+    ```
+      gem owner recog -a EMAIL
+    ```
+2. Edit [lib/recog/version.rb](https://github.com/rapid7/recog/blob/master/lib/recog/version.rb) and increment ```VERSION```.  Commit and push to rapid7/recog master.
+3. Run `rake release`.  Among other things, this creates the new gem, uploads it to Rubygems and tags the release with a tag like `v<VERSION>`, where `<VERSION>` is replaced with the version from `version.rb`.  For example, if you release version 1.2.3 of the gem, the tag will be `v1.2.3`.
+4. If your default remote repository is not `rapid7/recog`, you must ensure that the tags created in the previous step are also pushed to the right location(s).  For example, if `origin` is your fork of recog and `upstream` is `rapid7/master`, you should run `git push --tags --dry-run upstream` to confirm what tags will be pushed and then `git push --tags upstream` to push the tags.

data/COPYING

@@ -0,0 +1,23 @@
+Copyright (c) 2014-2015, Rapid7
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this
+  list of conditions and the following disclaimer in the documentation and/or
+  other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/Gemfile

@@ -0,0 +1,10 @@
+source 'https://rubygems.org'
+
+gemspec name: 'recog'
+
+gem 'nokogiri'
+
+group :test do
+  gem 'rake'
+  gem 'regexp_parser'
+end

data/LICENSE

@@ -0,0 +1,7 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Source: https://github.com/rapid7/recog
+
+Files: *
+Copyright: 2014-2015, Rapid7, Inc.
+License: BSD-2-clause
+

data/README.md

@@ -0,0 +1,85 @@
+# Recog: A Recognition Framework
+[![Gem Version](https://badge.fury.io/rb/recog.svg)](http://badge.fury.io/rb/recog)
+[![Build Status](https://travis-ci.org/rapid7/recog.svg?branch=master)](https://travis-ci.org/rapid7/recog)
+
+
+Recog is a framework for identifying products, services, operating systems, and hardware by matching fingerprints against data returned from various network probes. Recog makes it simple to extract useful information from web server banners, snmp system description fields, and a whole lot more.
+
+Recog is open source, please see the [LICENSE](https://raw.githubusercontent.com/rapid7/recog/master/LICENSE) file for more information.
+
+## Installation
+
+Recog consists of both XML fingerprint files and an assortment of code, mostly in Ruby, that makes it easy to develop, test, and use the contained fingerprints. In order to use the included ruby code, a recent version of Ruby (2.31+) is required, along with Rubygems and the `bundler` gem. Once these dependencies are in place, use the following commands to grab the latest source code and install any additional dependencies.
+
+```shell
+$ git clone git@github.com:rapid7/recog.git
+$ cd recog
+$ bundle install
+```
+
+## Maturity
+
+Please note that while the XML fingerprints themselves are quite stable and well-tested, the Ruby codebase in Recog is still fairly new and subject to change quickly. Please contact us (research[at]rapid7.com) before leveraging the Recog code within any production projects.
+
+## Fingerprints
+
+The fingerprints within Recog are stored in XML files, each of which is designed to match a specific protocol response string or field. For example, the file [ssh_banners.xml](https://github.com/rapid7/recog/blob/master/xml/ssh_banners.xml) can determine the os, vendor, and sometimes hardware product by matching the initial SSH daemon banner string.
+
+A fingerprint file consists of an XML document like the following:
+
+```xml
+<fingerprints matches="ssh.banner">
+  <fingerprint pattern="^RomSShell_([\d\.]+)$">
+    <description>Allegro RomSShell SSH</description>
+    <example service.version="4.62">RomSShell_4.62</example>
+    <param pos="0" name="service.vendor" value="Allegro"/>
+    <param pos="0" name="service.product" value="RomSShell"/>
+    <param pos="1" name="service.version"/>
+  </fingerprint>
+</fingerprints>
+```
+
+The first line should always consist of the XML version declaration. The first element should always be a `fingerprints` block with a `matches` attribute indicating what data this fingerprint file is supposed to match. The `matches` attribute is normally in the form of `protocol.field`.
+
+Inside of the `fingerprints` element there should be one or more `fingerprint` elements. Every `fingerprint` must contain a `pattern` attribute, which contains the regular expression to be used to match against the data.  An optional `flags` attribute can be specified to control how the regular expression is to be interpreted.  See [the Recog documentation for `FLAG_MAP`](http://www.rubydoc.info/gems/recog/Recog/Fingerprint/RegexpFactory#FLAG_MAP-constant) for more information.
+
+Inside of the fingerprint, a `description` element should contain a human-readable string describing this fingerprint.
+
+At least one `example` element should be present, however multiple `example` elements are preferred.  These elements are used as part of the test coverage present in `rspec` which validates that the provided data matches the specified regular expression.  Additionally, if the fingerprint is using the `param` elements to extract field values from the data (described next), you can add these expected extractions as attributes for the `example` elements.  In the example above, this:
+
+```xml
+<example service.version="4.62">RomSShell_4.62</example>
+```
+
+tests that `RomSShell_4.62` matches the provided regular expression and that the value of `service.version` is 4.62.
+
+The `param` elements contain a `pos` attribute, which indicates what capture field from the `pattern` should be extracted, or `0` for a static string. The `name` attribute is the key that will be reported in the case of a successful match and the `value` will either be a static string for `pos` values of `0` or missing and taken from the captured field.
+
+The `example` string can be base64 encoded to permit the use of unprintable characters.  To signal this to Recog an `_encoding` attribute with the value of `base64` is added to the `example` element.  Based64 encoded text that is longer than 80 characters may be wrapped with newlines as shown below to aid in readability.
+
+````xml
+<example _encoding="base64">
+  dGllczGEAAAAlQQWMS4yLjg0MC4xMTM1NTYuMS40LjgwMAQuZGF0YS5yZW1vdmVkLjCEAAAAK
+  AQdZG9tYWluQ29udHJvbGxlckZ1bmN0aW9uYWxpdHkxhAAAAAMEATc=
+</example>
+````
+
+### Testing
+
+Once a fingerprint has been added, the `example` entries can be tested by executing `bin/recog_verify` against the fingerprint file:
+
+```shell
+$ bin/recog_verify xml/ssh_banners.xml
+```
+
+Matches can be tested on the command-line in a similar fashion:
+
+```shell
+$ echo 'OpenSSH_6.6p1 Ubuntu-2ubuntu1' | bin/recog_match xml/ssh_banners.xml -
+MATCH: {"matched"=>"OpenSSH running on Ubuntu 14.04", "service.version"=>"6.6p1", "openssh.comment"=>"Ubuntu-2ubuntu1", "service.vendor"=>"OpenBSD", "service.family"=>"OpenSSH", "service.product"=>"OpenSSH", "os.vendor"=>"Ubuntu", "os.device"=>"General", "os.family"=>"Linux", "os.product"=>"Linux", "os.version"=>"14.04", "service.protocol"=>"ssh", "fingerprint_db"=>"ssh.banner", "data"=>"OpenSSH_6.6p1 Ubuntu-2ubuntu1"}
+```
+
+### Best Practices
+
+* Create a single fingerprint for each product as long as the pattern remains clear and readable. If that is not possible, the pattern should be logically decomposed into additional fingerprints.
+* Create regular expressions that allow for flexible version number matching. This ensures greater probability of matching a product. For example, all known public releases of a product report either `major.minor` or `major.minor.build` format version numbers. If the fingerprint strictly matches this version number format, it would fail to match a modified build of the product that reports only a `major` version number format.