yavdb 0.1.0.pre.alpha.2

data/README.md

@@ -0,0 +1,87 @@
+# Yet Another Vulnerability Database
+
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/00298529610b41f4a6ec380550ea45de)](https://www.codacy.com/app/rtfpessoa/yavdb?utm_source=github.com&utm_medium=referral&utm_content=rtfpessoa/yavdb&utm_campaign=Badge_Grade)
+[![Codacy Badge](https://api.codacy.com/project/badge/Coverage/00298529610b41f4a6ec380550ea45de)](https://www.codacy.com/app/rtfpessoa/yavdb?utm_source=github.com&utm_medium=referral&utm_content=rtfpessoa/yavdb&utm_campaign=Badge_Coverage)
+[![CircleCI](https://circleci.com/gh/rtfpessoa/yavdb.svg?style=svg)](https://circleci.com/gh/rtfpessoa/yavdb)
+
+The Free and Open Source vulnerability database.
+
+This database aims to aggregate multiple sources of vulnerabilities for the most common package managers helping 
+developers identify and fix know vulnerabilities in their apps.
+
+The sources for this database include 
+[Rubysec](https://rubysec.com/),
+[snyk](https://snyk.io/),
+[OSSIndex](https://ossindex.net/),
+[NodeSecurity](https://nodesecurity.io/),
+[Friends of PHP](https://github.com/FriendsOfPHP/security-advisories),
+[Magento Related Security Advisories](https://github.com/victims/victims-cve-db),
+[Victims CVE Database](https://github.com/victims/victims-cve-db)
+
+## Prerequisites
+
+* Ruby 2.3 or newer
+
+## Installation
+
+```sh
+gem install yavdb --pre
+```
+
+> Notice the `--pre` in the end
+
+## TODO:
+
+#### Tests
+    
+* Sources
+    - [ ] [Rubysec](lib/yavdb/sources/ruby_advisory.rb)
+    - [X] [snyk](lib/yavdb/sources/snyk_io.rb)
+    - [ ] [OSSIndex](lib/yavdb/sources/ossindex.rb)
+    - [X] [NodeSecurity](lib/yavdb/sources/nodesecurity_io.rb)
+    - [ ] [Friends of PHP and Magento Related Security Advisories](lib/yavdb/sources/friends_of_php.rb)
+    - [ ] [Victims CVE Database](lib/yavdb/sources/victims.rb)
+* Others
+    - [ ] [Advisory](lib/yavdb/dtos/advisory.rb)
+
+#### Features/Improvements
+
+- [ ] Merge  duplicates
+- [ ] Scrape [NVD](https://nvd.nist.gov/) for other package manager vulnerabilities
+- [ ] Find more sources
+
+### Help
+
+    Commands:
+      yavdb download                                                            # Download a previously generated database from the official yavdb repository into yavdb-path.
+        Options: p, [--yavdb-path=YAVDB-PATH]  # Default: <HOME>/.yavdb/yavdb
+      yavdb generate                                                            # Crawl several sources and generate a local database in database-path.
+        Options: p, [--database-path=DATABASE-PATH]  # Default: <PWD>/database
+      yavdb help [COMMAND]                                                      # Describe available commands or one specific command
+      yavdb list --package-manager=PACKAGE-MANAGER --package-name=PACKAGE-NAME  # List vulnerabilities from database-path of package-name for package-manager.   
+        Options: p, [--database-path=DATABASE-PATH]  # Default: <HOME>/.yavdb/yavdb/database
+    
+    Options:
+      [--verbose], [--no-verbose]
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies.
+Then, run `bundle exec rake spec` to run the tests.
+You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`,
+which will create a git tag for the version,
+push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/rtfpessoa/yavdb.
+This project is intended to be a safe, welcoming space for collaboration,
+and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+
+## Copyright
+
+Copyright (c) 2017-present Rodrigo Fernandes.
+See [LICENSE](https://github.com/rtfpessoa/yavdb/blob/master/LICENSE) for details.

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'yavdb'
+
+require 'irb'
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+IFS=$'\n\t'
+set -vx
+
+bundle install

data/bin/vulndb

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+load File.expand_path('../yavdb', __FILE__)

data/bin/vulnerabilitydb

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+
+load File.expand_path('../yavdb', __FILE__)

data/bin/yavdb

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'yavdb/cli'
+
+YAVDB::CLI.start(ARGV)

data/lib/yavdb.rb

@@ -0,0 +1,68 @@
+# yavdb - The Free and Open Source vulnerability database
+# Copyright (C) 2017-present Rodrigo Fernandes
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+require 'net/http'
+require 'socket'
+require 'yaml'
+require 'semantic_interval'
+require 'oga'
+require 'oga/xml/entities'
+
+require_relative 'yavdb/constants'
+require_relative 'yavdb/crawler'
+require_relative 'yavdb/database'
+require_relative 'yavdb/utils/exec'
+require_relative 'yavdb/utils/http'
+require_relative 'yavdb/utils/git'
+
+module YAVDB
+  class API
+
+    # List vulnerabilities from database_path of package_name for package_manager.
+    #
+    # @param package_manager [String] the package manager.
+    # @param package_name [String] the package_name.
+    # @param database_path [String] the local path to the database.
+    # @return [Array<Advisory>] the array of vulnerabilities.
+    def self.list_vulnerabilities(package_manager,
+      package_name,
+      database_path = YAVDB::Constants::DEFAULT_YAVDB_PATH)
+      YAVDB::Database.search(database_path, package_manager, package_name)
+    end
+
+    # Crawl several sources and generate a local database in database_path.
+    #
+    # @param database_path [String] the local path to the database.
+    def self.generate_database(database_path = YAVDB::Constants::DEFAULT_GENERATE_DATABASE_PATH)
+      vulnerabilities = YAVDB::Crawler.vulnerabilities
+      YAVDB::Database.save(database_path, vulnerabilities)
+    end
+
+    # Download a previously generated database from the official yavdb repository into yavdb_path.
+    #
+    # @param force_update [Boolean] force an update of the database if it already exists but is in a previous version.
+    # @param yavdb_path [String] the local path to the yavdb repository with the database.
+    # @param yavdb_url [String] the yavdb url to clone the database repository from.
+    # @param yavdb_branch [String] the yavdb branch with the database.
+    def self.download_database(force_update = false,
+      yavdb_path = YAVDB::Constants::DEFAULT_YAVDB_PATH,
+      yavdb_url = YAVDB::Constants::YAVDB_DB_URL,
+      yavdb_branch = YAVDB::Constants::YAVDB_DB_BRANCH)
+      YAVDB::Utils::Git.download_or_update(yavdb_path, yavdb_url, yavdb_branch, force_update)
+    end
+
+  end
+end

data/lib/yavdb/cli.rb

@@ -0,0 +1,60 @@
+# yavdb - The Free and Open Source vulnerability database
+# Copyright (C) 2017-present Rodrigo Fernandes
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+require 'thor'
+
+require_relative 'constants'
+require_relative '../yavdb'
+
+module YAVDB
+  class CLI < Thor
+
+    map '--version' => :version
+
+    class_option('verbose', :type => :boolean, :default => false)
+
+    method_option('package-manager', :alias => :m, :type => :string, :required => true)
+    method_option('package-name', :alias => :n, :type => :string, :required => true)
+    method_option('database-path', :type => :string, :aliases => :p, :default => YAVDB::Constants::DEFAULT_YAVDB_DATABASE_PATH)
+    desc('list', 'List vulnerabilities from database-path of package-name for package-manager.')
+
+    def list
+      package_manager = options['package-manager']
+
+      unless YAVDB::Constants::POSSIBLE_PACKAGE_MANAGERS.include?(package_manager)
+        puts "Package manager #{package_manager} is not supported yet."
+        exit(1)
+      end
+
+      API.list_vulnerabilities(package_manager, options['package-name'], options['database-path'])
+    end
+
+    method_option('database-path', :type => :string, :aliases => :p, :default => YAVDB::Constants::DEFAULT_GENERATE_DATABASE_PATH)
+    desc('generate', 'Crawl several sources and generate a local database in database-path.')
+
+    def generate
+      API.generate_database(options['database-path'])
+    end
+
+    method_option('yavdb-path', :type => :string, :aliases => :p, :default => YAVDB::Constants::DEFAULT_YAVDB_PATH)
+    desc('download', 'Download a previously generated database from the official yavdb repository into yavdb-path.')
+
+    def download
+      API.download_database(options['yavdb-path'])
+    end
+
+  end
+end

data/lib/yavdb/constants.rb

@@ -0,0 +1,34 @@
+# yavdb - The Free and Open Source vulnerability database
+# Copyright (C) 2017-present Rodrigo Fernandes
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+module YAVDB
+  module Constants
+
+    DEBUG = ENV['debug'].freeze
+
+    YAVDB_DB_URL    = 'https://github.com/rtfpessoa/yavdb.git'
+    YAVDB_DB_BRANCH = 'database'
+
+    DEFAULT_GENERATE_DATABASE_PATH = File.expand_path(File.join(Dir.pwd, ['database'])).freeze
+
+    DEFAULT_YAVDB_PATH          = File.expand_path(File.join(ENV['HOME'], '.yavdb', 'yavdb')).freeze
+    DEFAULT_YAVDB_DATABASE_PATH = File.expand_path(File.join(DEFAULT_YAVDB_PATH, 'database')).freeze
+    DEFAULT_CACHE_PATH          = File.expand_path(File.join(ENV['HOME'], '.yavdb', 'cache')).freeze
+
+    POSSIBLE_PACKAGE_MANAGERS = ['npm', 'rubygems', 'maven', 'nuget', 'packagist', 'pypi', 'go'].freeze
+
+  end
+end

data/lib/yavdb/crawler.rb

@@ -0,0 +1,52 @@
+# yavdb - The Free and Open Source vulnerability database
+# Copyright (C) 2017-present Rodrigo Fernandes
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Dir[File.expand_path('sources/*.rb', __dir__)].each do |file|
+  require file
+end
+
+module YAVDB
+  class Crawler
+
+    def self.sources
+      YAVDB::Sources.constants
+        .map { |c| YAVDB::Sources.const_get(c) }
+        .sort_by { |c| c.to_s.downcase }
+    end
+
+    def self.vulnerabilities
+      vulns = sources.map { |src| src::Client.advisories }.flatten
+      clean_vulnerability_versions(vulns)
+    end
+
+    class << self
+
+      private
+
+      def clean_vulnerability_versions(vulnerabilities)
+        vulnerabilities
+          .map do |vln|
+          vln.vulnerable_versions = YAVDB::Utils::SemVer.clean_versions(vln.vulnerable_versions)
+          vln.unaffected_versions = YAVDB::Utils::SemVer.clean_versions(vln.unaffected_versions)
+          vln.patched_versions    = YAVDB::Utils::SemVer.clean_versions(vln.patched_versions)
+          vln
+        end
+      end
+
+    end
+
+  end
+end

data/lib/yavdb/database.rb

@@ -0,0 +1,95 @@
+# yavdb - The Free and Open Source vulnerability database
+# Copyright (C) 2017-present Rodrigo Fernandes
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+require_relative 'constants'
+require_relative 'utils/semver'
+
+module YAVDB
+  class Database
+
+    def self.save(database_path, vulns)
+      vulns_grouped_by_package_manager = group_by_package_manager(vulns)
+      save_to_file(database_path, vulns_grouped_by_package_manager)
+    end
+
+    def self.search(database_path, package_manager, package_name)
+      package_file_path = package_path(database_path, package_manager, package_name)
+
+      if File.exist?(package_file_path)
+        YAVDB::Advisory.load(package_file_path)
+      else
+        []
+      end
+    end
+
+    class << self
+
+      private
+
+      def group_by_package_manager(vulns)
+        vulns
+          .group_by(&:package_manager)
+          .map do |package_manager, vunerabilities_by_pm|
+
+          puts "#{package_manager}: #{vunerabilities_by_pm.length}"
+
+          vunerabilities_by_pm =
+            vunerabilities_by_pm
+              .group_by(&:affected_package)
+              .map do |package, vunerabilities_by_p|
+              [package, vunerabilities_by_p]
+            end.to_h
+
+          [package_manager, vunerabilities_by_pm]
+        end.to_h
+      end
+
+      def save_to_file(database_path, vulns)
+        vulns.map do |package_manager, vunerabilities_by_pm|
+          vunerabilities_by_pm.map do |package, vunerabilities_by_p|
+            package_path           = package_path(database_path, package_manager, package)
+            package_path_directory = File.dirname(package_path)
+            FileUtils.mkdir_p(package_path_directory) unless File.exist?(package_path_directory)
+
+            File.open(package_path, 'wb') do |file|
+              package_vulns_yml_str = vunerabilities_by_p
+                                        .map(&:to_map)
+                                        .to_yaml(
+                                          :Indent        => 4,
+                                          :SortKeys      => true,
+                                          :UseHeader     => true,
+                                          :UseVersion    => true,
+                                          :ExplicitTypes => true,
+                                          :BestWidth     => 80,
+                                          :UseFold       => true,
+                                          :UseBlock      => true,
+                                          :Encoding      => :Utf8
+                                        )
+
+              file.puts(package_vulns_yml_str)
+            end
+          end
+        end
+      end
+
+      def package_path(database_path, package_manager, package_name)
+        File.expand_path(File.join(database_path, package_manager, "#{package_name}.yml"))
+      end
+
+    end
+
+  end
+end