spandx 0.11.0 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: da243ee84b54d710a15eeb5182f7b70084cbc69cb86beaa93c25c13dc650fe2a
-  data.tar.gz: 904eff3844f540311828386dde54ad70054f5c4fe90e456bb0af702efc9f7cc3
+  metadata.gz: 1f54b3c9f0a58ce0038a5ad242c02d7aad3b11090bdecb9d84e67a9f74d5dfd6
+  data.tar.gz: b4ed659d450474bff291b4ee0f723f3342fe9369b690894c1731badb797a4b6b
 SHA512:
-  metadata.gz: 67135d0cff701f454ce5d127a7f4ec7fd061d63bc832d5eb7df20bb26efa6e45bc90e28bc22eceaa2c135437858aa63337db8ff898245d870fe7acfcae14413d
-  data.tar.gz: a4086a92383a8c334c3488004b6be02a0ce80a79b9c7386eea5948d3d52d031d46e742d9e290afa45e306c4d3d1e4bbb7c2849cdd1d2d3dc98b5aba74fd13ed8
+  metadata.gz: e94f49bff3619f876aa2157f15f8efde4b585720bd95417b1d4c6a48ce24208e18d836597604e4837ea37aee424124404825f8887311ab6697b2efee8474ea42
+  data.tar.gz: cc3553ebc60ff24667c0b182e9beb319bed075e837ef34b348b6b8c15ff16ebccfc56e4cb80c75743baa7862b07e7331e5c924ee2305e6008f5ff1f772e78a71

data/CHANGELOG.md

@@ -1,4 +1,4 @@
-Version 0.11.0
+Version 0.12.0
 
 # Changelog
 
@@ -9,6 +9,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.12.0] - 2020-04-14
+### Added
+- Add `--format csv` option to scan command.
+- Add `--format table` option to scan command.
+- Add `--index` option to `build` command.
+- Add pypi index.
+- Add maven index.
+- Add support for parsing `yarn.lock` files.
+- Add support for parsing `package-lock.json` files.
+- Add `--pull` option to fetch latest cache before scan.
+- Add support for parsing `composer.lock` files.
+- Add support for loading custom plugins via the `--require` option.
+
+### Changed
+- Change the default `--format` to `table` for the scan command.
+
 ## [0.11.0] - 2020-03-20
 ### Added
 - Add `--format` option to scan command.
@@ -115,7 +131,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 - Provide ruby API to the latest SPDX catalogue.
 
-[Unreleased]: https://github.com/mokhan/spandx/compare/v0.10.1...HEAD
+[Unreleased]: https://github.com/mokhan/spandx/compare/v0.12.0...HEAD
+[0.12.0]: https://github.com/mokhan/spandx/compare/v0.11.0...v0.12.0
+[0.11.0]: https://github.com/mokhan/spandx/compare/v0.10.1...v0.11.0
 [0.10.1]: https://github.com/mokhan/spandx/compare/v0.10.0...v0.10.1
 [0.10.0]: https://github.com/mokhan/spandx/compare/v0.9.0...v0.10.0
 [0.9.0]: https://github.com/mokhan/spandx/compare/v0.8.0...v0.9.0

data/README.md

@@ -1,8 +1,26 @@
-# Spandx
+# Spandx ![badge](https://github.com/mokhan/spandx/workflows/ci/badge.svg)
 
 A ruby API for interacting with the https://spdx.org software license catalogue.
+This gem includes a command line interface to scan a software project for the
+software licenses that are associated with each dependency in the project.
+`spandx` leverages an offline cache of software licenses for known dependencies.
+The offline cache allows spandx to perform a truly airgap friendly scan of software
+projects.
+
+### Supported project types
+
+Spandx can work with following language's package managers. It utilises lock files generated by package managers to find dependencies.
+
+| Language  | Package Manager    | Tested in   |
+| ------------ | --------------- | -------:    |
+| Ruby         | bundler         | 1.17.3      |
+| Js           | Npm             | 6.13.4      |
+| Js           | Yarn            | 1.19.1      |
+| Python       | Pypi(pipenv)    | v2018.11.26 |
+| C#           | nuget           | <>          |
+| Java         | Maven           | 3.6.3       |
+| Php          | Composer        | 1.10.5      |
 
-![badge](https://github.com/mokhan/spandx/workflows/ci/badge.svg)
 
 ## Installation
 
@@ -22,6 +40,45 @@ Or install it yourself as:
 
 ## Usage
 
+### Command line interface
+
+The command line interface supports operations to build and fetch the latest offline index.
+See the help for each subcommand for more information on how to use the command.
+
+```bash
+モ spandx
+Commands:
+  spandx help [COMMAND]      # Describe available commands or one specific command
+  spandx scan LOCKFILE       # Scan a lockfile and list dependencies/licenses
+  spandx version             # spandx version
+```
+
+To scan a specific project file use the `scan` command:
+
+```bash
+モ spandx scan dotnet/application.sln
+モ spandx scan java/pom.xml
+モ spandx scan python/Pipfile.lock
+モ spandx scan ruby/Gemfile.lock
+```
+
+To activate airgap mode use the `--airgap` option:
+
+```bash
+モ spandx scan dotnet/application.sln --airgap
+モ spandx scan ruby/Gemfile.lock --airgap
+```
+
+Airgap mode assumes that an offline cache has been placed in `$HOME/.local/share/`.
+
+To fetch the latest offline cache:
+
+```bash
+モ spandx index update
+```
+
+### Ruby API
+
 To fetch the latest version of the catalogue data from [SPDX](https://spdx.org/licenses/licenses.json).
 
 ```ruby

data/exe/spandx

@@ -2,8 +2,7 @@
 # frozen_string_literal: true
 
 lib_path = File.expand_path('../lib', __dir__)
-$LOAD_PATH.unshift(lib_path) unless $LOAD_PATH.include?(lib_path)
-require 'spandx/cli'
+require "#{lib_path}/spandx"
 
 Signal.trap('INT') do
   warn("\n#{caller.join("\n")}: interrupted")
@@ -11,8 +10,8 @@ Signal.trap('INT') do
 end
 
 begin
-  Spandx::CLI.start
-rescue Spandx::CLI::Error => error
+  Spandx::Cli::Main.start
+rescue Spandx::Cli::Error => error
   puts "ERROR: #{error.message}"
   exit 1
 end

data/lib/spandx.rb

@@ -9,38 +9,15 @@ require 'logger'
 require 'net/hippie'
 require 'nokogiri'
 require 'pathname'
+require 'yaml'
+require 'zeitwerk'
 
-require 'spandx/core/cache'
-require 'spandx/core/content'
-require 'spandx/core/database'
-require 'spandx/core/dependency'
-require 'spandx/core/guess'
-require 'spandx/core/http'
-require 'spandx/core/parser'
-require 'spandx/core/report'
-require 'spandx/core/score'
-require 'spandx/dotnet/index'
-require 'spandx/dotnet/nuget_gateway'
-require 'spandx/dotnet/package_reference'
-require 'spandx/dotnet/parsers/csproj'
-require 'spandx/dotnet/parsers/packages_config'
-require 'spandx/dotnet/parsers/sln'
-require 'spandx/dotnet/project_file'
-require 'spandx/java/index'
-require 'spandx/java/metadata'
-require 'spandx/java/parsers/maven'
-require 'spandx/python/parsers/pipfile_lock'
-require 'spandx/python/pypi'
-require 'spandx/python/source'
-require 'spandx/rubygems/gateway'
-require 'spandx/rubygems/parsers/gemfile_lock'
-require 'spandx/spdx/catalogue'
-require 'spandx/spdx/gateway'
-require 'spandx/spdx/license'
-require 'spandx/version'
+loader = Zeitwerk::Loader.for_gem
+loader.setup # ready!
 
 module Spandx
   class Error < StandardError; end
+  Rubygems = Ruby
 
   class << self
     attr_writer :airgap, :logger
@@ -61,10 +38,14 @@ module Spandx
       @logger ||= Logger.new('/dev/null')
     end
 
-    def spdx_db
-      @spdx_db ||= Spandx::Core::Database
-        .new(url: 'https://github.com/spdx/license-list-data.git')
-        .tap(&:update!)
+    def git
+      @git ||= {
+        cache: ::Spandx::Core::Git.new(url: 'https://github.com/mokhan/spandx-index.git'),
+        rubygems: ::Spandx::Core::Git.new(url: 'https://github.com/mokhan/spandx-rubygems.git'),
+        spdx: ::Spandx::Core::Git.new(url: 'https://github.com/spdx/license-list-data.git'),
+      }
     end
   end
 end
+
+loader.eager_load

data/lib/spandx/cli.rb

@@ -1,38 +1,9 @@
 # frozen_string_literal: true
 
 require 'thor'
-require 'spandx'
-require 'spandx/cli/command'
-require 'spandx/cli/commands/index'
-require 'spandx/cli/commands/scan'
 
 module Spandx
-  class CLI < Thor
+  module Cli
     Error = Class.new(StandardError)
-
-    desc 'version', 'spandx version'
-    def version
-      puts "v#{Spandx::VERSION}"
-    end
-    map %w[--version -v] => :version
-
-    register Spandx::Cli::Commands::Index, 'index', 'index [SUBCOMMAND]', 'Command description...'
-
-    desc 'scan LOCKFILE', 'Scan a lockfile and list dependencies/licenses'
-    method_option :help, aliases: '-h', type: :boolean, desc: 'Display usage information'
-    method_option :recursive, aliases: '-r', type: :boolean, desc: 'Perform recursive scan', default: false
-    method_option :airgap, aliases: '-a', type: :boolean, desc: 'Disable network connections', default: false
-    method_option :logfile, aliases: '-l', type: :string, desc: 'Path to a logfile', default: '/dev/null'
-    method_option :format, aliases: '-f', type: :string, desc: 'Format of report', default: 'json'
-    def scan(lockfile)
-      Spandx.airgap = options[:airgap]
-      Spandx.logger = Logger.new(options[:logfile])
-
-      if options[:help]
-        invoke :help, ['scan']
-      else
-        Spandx::Cli::Commands::Scan.new(lockfile, options).execute
-      end
-    end
   end
 end

data/lib/spandx/cli/commands/build.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Spandx
+  module Cli
+    module Commands
+      class Build
+        INDEXES = {
+          maven: Spandx::Java::Index,
+          nuget: Spandx::Dotnet::Index,
+          dotnet: Spandx::Dotnet::Index,
+          pypi: Spandx::Python::Index,
+        }.freeze
+
+        def initialize(options)
+          @options = options
+        end
+
+        def execute(output: $stdout)
+          catalogue = Spandx::Spdx::Catalogue.from_git
+          indexes.each do |index|
+            output.puts index.name
+            index.update!(catalogue: catalogue, output: output)
+          end
+          output.puts 'OK'
+        end
+
+        private
+
+        def indexes
+          index = INDEXES[@options[:index]&.to_sym]
+
+          if index.nil?
+            INDEXES.values.uniq.map { |x| x.new(directory: @options[:directory]) }
+          else
+            [index.new(directory: @options[:directory])]
+          end
+        end
+      end
+    end
+  end
+end

data/lib/spandx/cli/commands/pull.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Spandx
+  module Cli
+    module Commands
+      class Pull
+        def initialize(options)
+          @options = options
+        end
+
+        def execute(output: $stdout)
+          Spandx.git.each_value do |db|
+            output.puts "Updating #{db.url}..."
+            db.update!
+          end
+          output.puts 'OK'
+        end
+      end
+    end
+  end
+end

data/lib/spandx/cli/commands/scan.rb

@@ -3,12 +3,13 @@
 module Spandx
   module Cli
     module Commands
-      class Scan < Spandx::Cli::Command
+      class Scan
         attr_reader :scan_path
 
         def initialize(scan_path, options)
           @scan_path = ::Pathname.new(scan_path)
           @options = options
+          require(options[:require]) if options[:require]
         end
 
         def execute(output: $stdout)
@@ -18,7 +19,7 @@ module Spandx
               report.add(dependency)
             end
           end
-          output.puts report.to(@options[:format])
+          output.puts(format(report.to(@options[:format])))
         end
 
         private
@@ -33,6 +34,7 @@ module Spandx
             if File.directory?(file)
               each_file_in(file, &block) if recursive?
             else
+              Spandx.logger.debug(file)
               block.call(file)
             end
           end
@@ -42,7 +44,20 @@ module Spandx
           ::Spandx::Core::Parser
             .for(file)
             .parse(file)
+            .map { |dependency| enhance(dependency) }
             .each { |dependency| yield dependency }
+        rescue StandardError => error
+          Spandx.logger.error(error)
+        end
+
+        def format(output)
+          Array(output).map(&:to_s)
+        end
+
+        def enhance(dependency)
+          ::Spandx::Core::Plugin
+            .all
+            .inject(dependency) { |memo, plugin| plugin.enhance(memo) }
         end
       end
     end

data/lib/spandx/cli/main.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module Spandx
+  module Cli
+    class Main < Thor
+      desc 'scan LOCKFILE', 'Scan a lockfile and list dependencies/licenses'
+      method_option :help, aliases: '-h', type: :boolean, desc: 'Display usage information'
+      method_option :recursive, aliases: '-R', type: :boolean, desc: 'Perform recursive scan', default: false
+      method_option :airgap, aliases: '-a', type: :boolean, desc: 'Disable network connections', default: false
+      method_option :logfile, aliases: '-l', type: :string, desc: 'Path to a logfile', default: '/dev/null'
+      method_option :format, aliases: '-f', type: :string, desc: 'Format of report', default: 'table'
+      method_option :pull, aliases: '-p', type: :boolean, desc: 'Pull the latest cache before the scan', default: false
+      method_option :require, aliases: '-r', type: :string, desc: 'Causes spandx to load the library using require.', default: nil
+      def scan(lockfile)
+        if options[:help]
+          invoke :help, ['scan']
+        else
+          Spandx.airgap = options[:airgap]
+          Spandx.logger = Logger.new(options[:logfile])
+          pull if options[:pull]
+          Spandx::Cli::Commands::Scan.new(lockfile, options).execute
+        end
+      end
+
+      desc 'pull', 'Pull the latest offline cache'
+      method_option :help, aliases: '-h', type: :boolean, desc: 'Display usage information'
+      def pull(*)
+        if options[:help]
+          invoke :help, ['pull']
+        else
+          Commands::Pull.new(options).execute
+        end
+      end
+
+      desc 'build', 'Build a package index'
+      method_option :help, aliases: '-h', type: :boolean, desc: 'Display usage information'
+      method_option :directory, aliases: '-d', type: :string, desc: 'Directory to build index in', default: '.index'
+      method_option :index, aliases: '-i', type: :string, desc: 'The specific index to build', default: :all
+      def build(*)
+        if options[:help]
+          invoke :help, ['build']
+        else
+          Commands::Build.new(options).execute
+        end
+      end
+
+      desc 'version', 'spandx version'
+      def version
+        puts "v#{Spandx::VERSION}"
+      end
+      map %w[--version -v] => :version
+    end
+  end
+end

data/lib/spandx/core/cache.rb

@@ -5,14 +5,14 @@ module Spandx
     class Cache
       attr_reader :db, :package_manager
 
-      def initialize(package_manager, url:)
+      def initialize(package_manager, db: Spandx.git[:cache])
         @package_manager = package_manager
-        @db = ::Spandx::Core::Database.new(url: url)
+        @db = db
         @cache = {}
         @lines = {}
       end
 
-      def licenses_for(name:, version:)
+      def licenses_for(name, version)
         found = search(name: name, version: version)
         Spandx.logger.debug("Cache miss: #{name}-#{version}") if found.nil?
         found ? found[2].split('-|-') : []

data/lib/spandx/core/circuit.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Spandx
+  module Core
+    class Circuit
+      attr_reader :state
+
+      def initialize(state: :closed)
+        @state = state
+      end
+
+      def attempt
+        return if open?
+
+        open!
+        result = yield
+        close!
+        result
+      end
+
+      def open!
+        @state = :open
+      end
+
+      def close!
+        @state = :closed
+      end
+
+      def open?
+        state == :open
+      end
+    end
+  end
+end