fidius-cvedb 0.0.2

data/Gemfile

@@ -0,0 +1,7 @@
+source "http://rubygems.org"
+
+gem "nokogiri"
+gem "mysql2"
+
+# Specify your gem's dependencies in fidius-cvedb.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,57 @@
+The Simplified BSD License
+
+Copyright (C) 2010-2011  FIDIUS Intrusion Detection with Intelligent
+User Support (FIDIUS). All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY FIDIUS ``AS IS'' AND ANY EXPRESS OR
+IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL FIDIUS OR CONTRIBUTORS BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
+IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+The views and conclusions contained in the software and documentation
+are those of the authors and should not be interpreted as representing
+official policies, either expressed or implied, of FIDIUS.
+
+
+*OR*
+
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 2, June 1991
+
+Copyright (C) 2010-2011  FIDIUS Intrusion Detection with Intelligent
+User Suppport.
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along
+with this program; if not, write to the Free Software Foundation, Inc.,
+51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+A digital copy is also available for download here:
+http://www.gnu.org/licenses/gpl-2.0.txt.

data/README.md

@@ -0,0 +1,106 @@
+# FIDIUS fidius-cvedb
+
+The FIDIUS CVE-DB Gem is used to create and run your own vulnerability database,
+or as an addition to the FIDIUS Command&Control Server to receive vulnerability
+information about target hosts. It uses the National Vulnerability Database
+(NVD [nvd.nist.gov](http://nvd.nist.gov/)) to gather vulnerability entries
+which are based on the Common Vulnerabilities and Exposures (CVE) identifiers.
+
+Therefore it includes rake tasks to download and parse XML files provided by the
+NVD, to store and update them in your personal database. Furthermore it includes
+ActiveRecord models, migrations and example database configuration to store
+Vulnerabilities easily.
+
+This gem is developed in the context of the students project "FIDIUS" at the
+University of Bremen, for more information about FIDIUS visit
+[fidius.me](http://fidius.me/en).
+
+## Installation
+
+Simply install this package with Rubygems:
+
+    $ gem install fidius-cvedb
+
+Then add it to your gemfile (Rails 3)
+
+    gem 'fidius-cvedb'
+    gem 'mysql2' # only required when you use mysql db like in the example below
+
+or environment.rb (prior Rails 3)
+
+    require 'rubygems'
+    require 'active_record'
+    require 'fidius-cvedb'
+
+Please note: The CVE-DB Gem has only been tested with Linux systems and might
+not work with Windows.
+
+## Configuration
+
+The setup depends on the context you want to use the gem. It can be used in the
+context of the FIDIUS Command&Control Server, or in your own Rails-app. It might
+access an already existing database or migrate a new one.
+
+0. Go to your Rails-app folder and run `fidius-cvedb --standalone` or
+   `fidius-cvedb --fidius`, depending on the context you are using it. For Rails
+   versions prior 3 this will create symlinks for the Rake tasks.
+1. Set up a new CVE Database if you need to or configure an existing one, add
+   the CVE Database to your database.yml accordingly. Note that it must be named
+   "cve_db":
+
+          cve_db:
+            adapter: mysql2
+            encoding: utf8
+            database: my_cve_database
+            pool: 5
+            username: my_username
+            password: my_password
+            host: localhost
+
+2. When you created a new database, run `rake nvd:migrate` to create the tables
+   needed.
+3. When you set up your own database initialize it (note that it needs to be
+   migrated before). Go to your Rails-app folder and run
+   `rake nvd:initialize`. This will download all available informations from the
+   NVD, parse and store it in your database. This takes about 3 hours, depending
+   on your machine. To keep your database up-to-date run `rake nvd:update`
+   regularly, e.g. as daily cronjob.
+4. Now you should be able to use the NVD Entries, to test this go to your
+   console (`rails console` | `ruby script/console`) and get an Entry:
+
+        $ FIDIUS::CveDb::NvdEntry.first
+
+
+## Synopsis
+
+This package comes with an executable script. You may invoke it as
+
+    $ fidius-cvedb <option>
+
+where _option_ may be:
+
+* `-f` | `--fidius` Initialize CVE-DB for Usage in FIDIUS C&C-Server
+* `-s` | `--standalone` Initialize CVE-DB standalone version
+* `-h` | `--help` Show help message
+* `-v` | `--version` Shows the gem version
+
+
+## Authors and Contact
+
+fidius-cvedb was written by
+
+* FIDIUS Intrusion Detection with Intelligent User Support
+  <grp-fidius@tzi.de>, <http://fidius.me>
+* in particular:
+ * Andreas Bender <bender+fidius-cvedb@tzi.de>
+ * Jens Färber <jfaerber+fidius-cvedb@tzi.de>
+
+If you have any questions, remarks, suggestion, improvements,
+etc. feel free to drop a line at the addresses given above.
+You might also join `#fidius` on Freenode or use the contact
+form on our [website](http://fidius.me/en/contact).
+
+
+## License
+
+Simplified BSD License and GNU GPLv2. See also the file LICENSE.

data/Rakefile

@@ -0,0 +1,15 @@
+require 'bundler'
+require 'rake/clean'
+require 'rubygems'
+
+Bundler::GemHelper.install_tasks
+
+CLOBBER.include 'pkg'
+
+
+namespace :nvd do
+
+  desc 'Test parsing functionality of the gem.'
+  task :test do
+  end
+end

data/bin/fidius-cvedb

@@ -0,0 +1,64 @@
+#!/usr/bin/env ruby
+require 'optparse'
+require 'fidius-cvedb/version'
+require 'fileutils'
+
+GEM_BASE = File.join(ENV['GEM_HOME'], 'gems',
+                     "fidius-cvedb-#{FIDIUS::CveDb::VERSION}", 'lib')
+
+options = {}
+
+optparse = OptionParser.new do|opts|
+  
+  opts.banner = "Usage: fidius-cvedb-runner [options]"
+  
+  opts.on_tail("-f", "--fidius", "Initialize CVE-DB for Usage in FIDIUS C&C-Server") do
+    if rails_root?
+      rake_tasks
+    end
+    exit
+  end
+  
+  opts.on_tail("-s", "--standalone", "Initialize CVE-DB standalone version") do
+    if rails_root?
+      rake_tasks
+    end
+    exit
+  end
+  
+  opts.on_tail("-h", "--help", "Show this message") do
+    puts opts
+    exit
+  end
+
+  opts.on_tail("-v", "--version", "Show version") do
+    puts "FIDIUS CVE-DB, Version #{FIDIUS::Cvedb::VERSION}"
+    exit
+  end
+end
+
+def rake_tasks
+  if rails_3?
+    puts "It seems like you are using Rails 3. Rake tasks are included via "+
+         "Railties and don't need to be symlinked."
+  else
+    symlink_rake_tasks
+  end
+end
+
+def symlink_rake_tasks
+  Dir.glob(File.join GEM_BASE, 'tasks', '*.rake') do |rake|
+    dest = File.join 'lib', 'tasks', File.basename(rake)
+    FileUtils.ln_s(rake, dest)
+  end
+end
+
+def rails_root?
+  (File.exists?('config/environment.rb') && File.exists?('app/models'))
+end
+
+def rails_3?
+  File.exists? 'Gemfile'
+end
+
+optparse.parse!

data/fidius-cvedb.gemspec

@@ -0,0 +1,22 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "fidius-cvedb/version"
+
+Gem::Specification.new do |s|
+  s.name        = "fidius-cvedb"
+  s.version     = FIDIUS::CveDb::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.add_dependency('nokogiri') 
+  s.authors     = ["Andreas Bender", "Jens Färber"]
+  s.email       = ["bender@tzi.de", "jfaerber@tzi.de"]
+  s.homepage    = "http://fidius.me"
+  s.summary     = %q{Provides a parser and ActiveRecord models for the Common Vulnerability and Exposures (CVE) entries offered by the National Vulnerability Database (http://nvd.nist.gov/). }
+  s.description = %q{This gem provides an opportunity to run a vulnerability database in your own environt. Therefore it comes with a parser for the National Vulnerability Database and ActiveRecord models for storing the entries in a local database and accessing Entries comfortable with Rails. }
+
+  s.rubyforge_project = ""
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+end

data/lib/cveparser/main.rb

@@ -0,0 +1,31 @@
+require "#{FIDIUS::CveDb::GEM_BASE}/cveparser/parser"
+require "#{FIDIUS::CveDb::GEM_BASE}/cveparser/rails_store"
+require "#{FIDIUS::CveDb::GEM_BASE}/cveparser/ms_parser"
+
+include FIDIUS::CveDb
+
+PARAMS = {
+  '-p' => 'Parse new XML file passed as 2nd param.',
+  '-f' => 'Fix duplicate products.',
+  '-u' => 'Updates CVE-Entries, needs modified.xml or recent.xml by nvd.nist.gov as 2nd argument.',
+  '-m' => 'Creates the mapping between CVEs and Microsoft Security Bulletin Notation entries in the database.'
+}
+
+case ARGV[0]
+  when '-p' 
+    entries = FIDIUS::NVDParser.parse_cve_file ARGV[1]  
+    RailsStore.create_new_entries(ARGV[1].split("/").last, entries)
+  when '-f'
+    RailsStore.fix_product_duplicates
+  when '-u'
+    entries = FIDIUS::NVDParser.parse_cve_file ARGV[1]
+    RailsStore.update_cves(entries)
+  when '-m'
+    FIDIUS::MSParser.parse_ms_cve
+  else
+    puts "ERROR: You've passed none or an unknown parameter, available "+
+      "parameters are:"
+    PARAMS.each_key do |param|
+      puts "#{param}\t#{PARAMS[param]}"
+    end
+end

data/lib/cveparser/ms_parser.rb

@@ -0,0 +1,65 @@
+# Author::    FIDIUS (mailto:grp-fidius@tzi.de) 
+# License::   Distributes under the same terms as fidius-cvedb Gem
+
+require 'open-uri'
+require 'nokogiri'
+
+# This module provides a parser for the mapping between CVE-Number
+# and Microsoft Security Bulletin Number. It also stores the 
+# relationships in the database, so you need the models to store 
+# them.
+
+module FIDIUS
+  module MSParser
+  
+    BASE_URL = "http://cve.mitre.org/data/refs/refmap/source-MS.html"
+    
+    # Calls 'parse' and stores the mapping in the database
+    def self.parse_ms_cve
+      entries = parse
+      counter = 0
+      entries.each_pair do |ms,cves|
+        cves.each do |cve|       
+          existing_cve = NvdEntry.find_by_cve(cve.strip)
+          if existing_cve
+            Mscve.find_or_create_by_nvd_entry_id_and_name(existing_cve.id, ms)
+            puts "Found: #{existing_cve.cve}."
+            counter += 1
+          end
+        end
+      end  
+      puts "Added #{counter} items to database."
+    end
+
+    # Print all MS-Notation numbers mapped to CVE-Entries 
+    def self.print_map
+      entries = parse    
+      entries.each_pair do |ms,cves|
+        puts "#{ms}"
+        cves.each {|cve| puts "----#{cve}"}
+      end
+    end 
+
+    private
+    
+    # Parses the page given by BASE_URL and returns
+    # all entries 
+    def self.parse
+      doc = Nokogiri::HTML(open(BASE_URL))
+      entries = Hash.new("")
+      current_ms_entry = ""
+      doc.css('table[border="2"] > tr').each do |entry|
+        entry.css("td").each do |td|
+          if td.content =~ /CVE-\d{4}-\d{4}/
+            entries[current_ms_entry] = td.content.split("\n")
+          else
+            current_ms_entry = td.content.split(":").last
+            entries[current_ms_entry]
+          end
+        end
+      end
+      puts "Parsed #{entries.size} entries."
+      entries
+    end
+  end
+end

data/lib/cveparser/parser.rb

@@ -0,0 +1,138 @@
+# Author::    FIDIUS (mailto:grp-fidius@tzi.de) 
+# License::   Distributes under the same terms as fidius-cvedb Gem
+
+require "#{FIDIUS::CveDb::GEM_BASE}/cveparser/parser_model"
+require 'rubygems'
+require 'nokogiri'
+
+# This module provides a parser for the National Vulnerability Database
+# (nvd.nist.gov). The parser can handle XML files provided by the NVD in 
+# Version 2.0 and stores all entries from the parsed file in objects 
+# from the module 'FIDIUS::NVDParserModel'. 
+
+module FIDIUS
+  module NVDParser
+  
+    include NVDParserModel
+    
+    # Parse Version 2.0 XML-File from nvd.nist.gov.     
+    def self.parse_cve_file file
+      
+      doc = Nokogiri::XML(File.open(file))
+      doc.css("nvd").each do |nvd| 
+        version = nvd.attributes['nvd_xml_version'].value
+        if version != "2.0"
+          puts "Your XML has the wrong version (#{version}). " + 
+               "The CVE-Parser can only handle XML-Feeds in Version 2.0."
+          raise 'Invalid XML Version'
+        end 
+      end
+
+      start_time = Time.now
+      puts "[*] Start parsing \"#{file}\""
+    
+      entries = []
+      entry_count = 0
+      doc.css('nvd > entry').each do |entry|
+        entries << single_entry(entry)
+        entry_count += 1
+        if entry_count % 100 == 0 and entry_count > 0
+          puts "Parsed #{entry_count} CVE Entries."
+        end
+      end
+      end_time = Time.now
+      puts "[*] Finished parsing, parsed #{entries.size} entries in " +
+           "#{(end_time-start_time).round} seconds."
+      entries
+    end
+    
+    private  
+
+    # Parse single NVD-Entry and store it in an NVDEntry Object
+    def self.single_entry entry
+      params = {}
+      params[:cve] = entry.attributes['id'].value
+      params[:vulnerable_configurations] = vulnerable_configurations(entry)
+      params[:cvss] = cvss(entry)
+      params[:vulnerable_software] = vulnerable_software(entry)
+      params[:published_datetime] = child_value(entry, 'vuln|published-datetime')
+      params[:last_modified_datetime] = child_value(entry, 'vuln|last-modified-datetime')
+      params[:cwe] = cwe(entry)
+      params[:summary] = child_value(entry, 'vuln|summary')
+      params[:references] = references(entry)
+        
+      NVDParserModel::NVDEntry.new(params)
+    end  
+    
+    # Return CWE number for given CVE-Entry
+    def self.cwe entry
+      cwe = entry.at_css('vuln|cwe')
+      cwe.attributes['id'].value if cwe
+    end
+    
+    # Returns an array of all references which belong to the given CVE-Entry 
+    def self.references entry
+      ref_array = []
+      entry.css('vuln|references').each do |references|
+        ref_params = {}
+        ref_params[:source] = child_value(references, 'vuln|source')
+        ref = references.at_css('vuln|reference')
+        ref_params[:link] = ref.attributes['href'].value if ref
+        ref_params[:name] = child_value(references, 'vuln|reference')
+        ref_array << NVDParserModel::Reference.new(ref_params)
+      end
+      ref_array
+    end
+      
+    # Returns an array of all vulnerable products which belong to the given CVE-Entry
+    def self.vulnerable_software entry
+      vuln_products = []
+      entry.css('vuln|vulnerable-software-list > vuln|product').each do |product|
+        vuln_products << product.children.to_s if product
+      end
+      vuln_products
+    end
+    
+    # Returns the CVSS-Data from the given CVE-Entry
+    def self.cvss entry
+      
+      metrics = entry.css('vuln|cvss > cvss|base_metrics')
+      unless metrics.empty?
+        cvss_params = {}
+        {
+          :score                  => 'score',
+          :source                 => 'source',
+          :access_vector          => 'access-vector',
+          :authentication         => 'authentication',
+          :access_complexity      => 'access-complexity',
+          :confidentiality_impact => 'confidentiality-impact',
+          :integrity_impact       => 'integrity-impact',
+          :availability_impact    => 'availability-impact',
+          :generated_on_datetime  => 'generated-on-datetime'
+        
+        }.each_pair do |hash_key, xml_name|
+          elem = metrics.at_css("cvss|#{xml_name}")
+          value = elem ? elem.children.to_s : nil
+          cvss_params[hash_key] = value
+        end
+        NVDParserModel::Cvss_.new(cvss_params)
+      end
+    end
+    
+    # Returns an array of all vulnerable configurations which belong to the given CVE-Entry
+    def self.vulnerable_configurations entry
+      v_confs = []
+      entry.css('vuln|vulnerable-configuration > cpe-lang|logical-test'+
+                      ' > cpe-lang|fact-ref').each do |conf|
+          v_confs << conf.attributes['name'].value
+      end
+      v_confs
+    end
+    
+    # Helper method to retrieve the value of a child node in a XML file
+    def self.child_value(node, xml)
+      val = node.at_css(xml)
+      val.children.to_s if val
+    end
+  end
+end