licensed 3.2.0 → 3.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 46db33bf2c824a144fbe5a85acfef469c35faeec69c3afd15a6df0c363025174
-  data.tar.gz: 73e300eaeebd28afed3ded55f60fc24b0fae9d20795ac150322c1b1975052215
+  metadata.gz: f452bd7c6a58fdaa9a56cf7085b20fe4ff3a8f3eb214835ba82a52b2ed1ac71c
+  data.tar.gz: 8b3aff33c001623780455c68d23c014746e988b82a44db0fa243829c2be34cd5
 SHA512:
-  metadata.gz: 7d487c920e977198ac91f7eeac4fbea8c4c49a326c6d449532a06e206e9472d75276879a6e3247fee7f6e64d87f595300b3c7ee995e8d1d595fb53401888ccec
-  data.tar.gz: 77ac80e1833b1c02cbb67aac8a79422e2af24958f89a577f07a0885fb7c3cbbc71dc61e0d5fb1bd453b58a1e6d8d0c5b47bf65fdf669edec4347012af83363b9
+  metadata.gz: e0bb95e3496257986e52294a7788824043697d8f99d2745c65e30e3a5c255843bc1471cf47ab3f3cd407d597c658b2d82e1bc27a76e6f985b45af6803d0e98a5
+  data.tar.gz: 93eb593c4389bff724a0a41be7c583e96541bfc308a9c331bf5d34c35217c98160e026733a49cc07b93b654b23e4507a447dbd5ab9ef8f1596a0e38139187757

data/CHANGELOG.md

@@ -6,6 +6,49 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 3.3.0
+
+2021-09-18
+
+### Added
+
+- New cargo source enumerates rust dependencies (https://github.com/github/licensed/pull/404)
+
+### Changed
+
+- Removed non-functional files from gem builds (https://github.com/github/licensed/pull/405)
+
+## 3.2.3
+
+2021-09-14
+
+### Fixed
+
+- Bundler source will no longer infinitely recurse when enumerating specifications (https://github.com/github/licensed/pull/402)
+- Using the `--sources` command line option will no longer delete skipped sources' cached files (https://github.com/github/licensed/pull/401)
+
+## 3.2.2
+
+2021-09-09
+
+### Fixed
+
+- Bundler source works properly again when used outside of `bundle exec` (https://github.com/github/licensed/pull/397)
+
+## 3.2.1
+
+2021-09-06
+
+### Changed
+
+- Updated multiple dependency versions (:tada: @mmorel-35 https://github.com/github/licensed/pull/385, https://github.com/github/licensed/pull/389)
+- Go homepage links use pkg.go.dev instead of godoc.org (:tada: @mmorel-35 https://github.com/github/licensed/commit/73cfbbe954a3e8c8cbaf8b68253053b157e01b79)
+- Local development ruby version changed to 2.7.4 (https://github.com/github/licensed/pull/393)
+
+### Fixed
+
+- Bundler source correctly finds platform specific dependencies (https://github.com/github/licensed/pull/392)
+
 ## 3.2.0
 
 2021-08-19
@@ -466,4 +509,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/3.1.0...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/3.3.0...HEAD

data/README.md

@@ -84,7 +84,7 @@ A configuration file is required for most commands.  See the [configuration file
 
 ### Available dependency sources
 
-Licensed can enumerate dependency for many languages, package managers, and frameworks.  See the [sources documentation](./docs/sources) for the list of currently available sources.  Sources can be explicitly enabled and disabled as a [configuration option](./docs/configuration/sources.md).
+Licensed can enumerate dependency for many languages, package managers, and frameworks.  See the [sources documentation](./docs/sources) for the list of currently available sources.  Sources can be explicitly enabled and disabled as a [configuration option](./docs/configuration/dependency_source_enumerators.md.md).
 
 ### Automation
 

data/docs/commands/status.md

@@ -65,6 +65,7 @@ If the dependency does not include license text but does specify that it uses a
 **Resolution:** Review the dependency's usage and specified license with someone familiar with OSS licensing and compliance rules to determine whether the dependency is allowable.  Some common resolutions:
 
 1. The dependency's specified license text differed enough from the standard license text that it was not recognized and classified as `other`.  If, with human review, the license text is recognizable then update the `license: other` value in the cached metadata file to the correct license.
+   - An updated classification will persist through version upgrades until the detected license contents have changed.  The determination is made by [licensee/licensee](https://github.com/licensee/licensee), the library which this tool uses to detect and classify license contents.
 1. The dependency might need to be marked as [ignored] or [reviewed] if either of those scenarios are applicable.
 1. If the used license should be allowable without review (if your entity has a legal team, they may want to review this assessment), ensure the license SPDX is set as [allowed] in the licensed configuration file.
 

data/docs/sources/cargo.md

@@ -0,0 +1,19 @@
+# Cargo
+
+The cargo source will detect dependencies when `Cargo.toml` is found at an apps `source_path`.  The source uses the `cargo metadata` CLI and reports on all dependencies that are listed in the output in `resolve.nodes`, excluding packages that are listed in `workspace_members`.
+
+## Metadata CLI options
+
+Licensed by default runs `cargo metadata --format-version=1`.  You can specify additional CLI options by specifying them in your licensed configuration file under `cargo.metadata_options`.  The configuration can be set as a string, or as an array of strings for multiple options.
+
+```yml
+cargo:
+  metadata_options: '--all-features'
+```
+
+```yml
+cargo:
+  metadata_options:
+    - '--all-features'
+    - '--filter-platform x86_64-pc-windows-msvc'
+```

data/lib/licensed/commands/cache.rb

@@ -39,11 +39,13 @@ module Licensed
       #
       # Returns whether the command succeeded for the dependency source enumerator
       def run_source(app, source, report)
+        result = super
+
         # add the full cache path to the list of cache paths
         # that should be cleaned up after the command run
-        cache_paths << app.cache_path.join(source.class.type)
+        cache_paths << app.cache_path.join(source.class.type) unless result == :skipped
 
-        super
+        result
       end
 
       # Cache dependency record data.

data/lib/licensed/commands/command.rb

@@ -121,13 +121,16 @@ module Licensed
       # source - A dependency source enumerator
       # report - A report object for this source
       #
-      # Returns whether the command succeeded for the dependency source enumerator
+      # Returns whether the command succeeded, failed, or was skipped for the dependency source enumerator
       def run_source(app, source, report)
         reporter.begin_report_source(source, report)
 
         if !sources_overrides.empty? && !sources_overrides.include?(source.class.type)
           report.warnings << "skipped source"
-          return true
+
+          # return a symbol to speficy the source was skipped.
+          # This is truthy and will result in the source being considered successful
+          return :skipped
         end
 
         dependencies = source.dependencies.sort_by { |dependency| dependency.name }

data/lib/licensed/reporters/status_reporter.rb

@@ -48,7 +48,7 @@ module Licensed
 
         errored_reports = all_reports.select { |r| r.errors.any? }.to_a
 
-        dependency_count = all_reports.select { |r| r.target.is_a?(Licensed::Dependency) }.size
+        dependency_count = all_reports.count { |r| r.target.is_a?(Licensed::Dependency) }
         error_count = errored_reports.sum { |r| r.errors.size }
 
         if error_count > 0

data/lib/licensed/sources/bundler/definition.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module Licensed
+  module Bundler
+    module DefinitionExtensions
+      attr_accessor :force_exclude_groups
+
+      # Override specs to avoid logic that would raise Gem::NotFound
+      # which is handled in this ./missing_specification.rb, and to not add
+      # bundler as a dependency if it's not a user-requested gem.
+      #
+      # Newer versions of Bundler have changed the implementation of specs_for
+      # as well which no longer calls this function.  Overriding this function
+      # gives a stable access point for licensed
+      def specs
+        @specs ||= begin
+          specs = resolve.materialize(requested_dependencies)
+
+          all_dependencies = requested_dependencies.concat(specs.flat_map(&:dependencies))
+          if all_dependencies.any? { |d| d.name == "bundler" } && !specs["bundler"].any?
+            bundler = sources.metadata_source.specs.search(Gem::Dependency.new("bundler", ::Bundler::VERSION)).last
+            specs["bundler"] = bundler
+          end
+
+          specs
+        end
+      end
+
+      # Override requested_groups to also exclude any groups that are
+      # in the "bundler.without" section of the licensed configuration file.
+      def requested_groups
+        super - Array(force_exclude_groups)
+      end
+    end
+  end
+end

data/lib/licensed/sources/bundler/missing_specification.rb

@@ -38,17 +38,20 @@ module Licensed
         "could not find #{name} (#{version}) in any sources"
       end
     end
+
+    module LazySpecification
+      def __materialize__
+        spec = super
+        return spec if spec
+
+        Licensed::Bundler::MissingSpecification.new(name: name, version: version, platform: platform, source: source)
+      end
+    end
   end
 end
 
 module Bundler
   class LazySpecification
-    alias_method :orig_materialize, :__materialize__
-    def __materialize__
-      spec = orig_materialize
-      return spec if spec
-
-      Licensed::Bundler::MissingSpecification.new(name: name, version: version, platform: platform, source: source)
-    end
+    prepend ::Licensed::Bundler::LazySpecification
   end
 end

data/lib/licensed/sources/bundler.rb

@@ -3,6 +3,7 @@ require "delegate"
 begin
   require "bundler"
   require "licensed/sources/bundler/missing_specification"
+  require "licensed/sources/bundler/definition"
 rescue LoadError
 end
 
@@ -37,7 +38,6 @@ module Licensed
         end
       end
 
-      GEMFILES = { "Gemfile" => "Gemfile.lock", "gems.rb" => "gems.locked" }
       DEFAULT_WITHOUT_GROUPS = %i{development test}
       RUBY_PACKER_ERROR = "The bundler source cannot be used from the executable built with ruby-packer.  Please install licensed using `gem install` or using bundler."
 
@@ -45,15 +45,20 @@ module Licensed
         # running a ruby-packer-built licensed exe when ruby isn't available
         # could lead to errors if the host ruby doesn't exist
         return false if ruby_packer? && !Licensed::Shell.tool_available?("ruby")
-        defined?(::Bundler) && lockfile_path && lockfile_path.exist?
+
+        # if Bundler isn't loaded, this enumerator won't work!
+        return false unless defined?(::Bundler)
+
+        with_application_environment { ::Bundler.default_lockfile&.exist? }
+      rescue ::Bundler::GemfileNotFound
+        false
       end
 
       def enumerate_dependencies
         raise Licensed::Sources::Source::Error.new(RUBY_PACKER_ERROR) if ruby_packer?
 
-        with_local_configuration do
-          specs.map do |spec|
-            next if spec.name == "bundler" && !include_bundler?
+        with_application_environment do
+          definition.specs.map do |spec|
             next if spec.name == config["name"]
 
             error = spec.error if spec.respond_to?(:error)
@@ -73,41 +78,13 @@ module Licensed
         end
       end
 
-      # Returns an array of Gem::Specifications for all gem dependencies
-      def specs
-        @specs ||= definition.specs_for(groups)
-      end
-
-      # Returns whether to include bundler as a listed dependency of the project
-      def include_bundler?
-        @include_bundler ||= begin
-          # include if bundler is listed as a direct dependency that should be included
-          requested_dependencies = definition.dependencies.select { |d| (d.groups & groups).any? && d.should_include? }
-          return true if requested_dependencies.any? { |d| d.name == "bundler" }
-          # include if bundler is an indirect dependency
-          return true if specs.flat_map(&:dependencies).any? { |d| d.name == "bundler" }
-          false
-        end
-      end
-
-      # Build the bundler definition
       def definition
-        @definition ||= ::Bundler::Definition.build(gemfile_path, lockfile_path, nil)
-      end
-
-      # Returns the bundle definition groups, removing "without" groups,
-      # and including "with" groups
-      def groups
-        @groups ||= definition.groups - bundler_setting_array(:without) + bundler_setting_array(:with) - exclude_groups
-      end
-
-      # Returns a bundler setting as an array.
-      # Depending on the version of bundler, array values are either returned as
-      # a raw string ("a:b:c") or as an array ([:a, :b, :c])
-      def bundler_setting_array(key)
-        setting = ::Bundler.settings[key]
-        setting = setting.split(":").map(&:to_sym) if setting.is_a?(String)
-        Array(setting)
+        @definition ||= begin
+          definition = ::Bundler::Definition.build(::Bundler.default_gemfile, ::Bundler.default_lockfile, nil)
+          definition.extend Licensed::Bundler::DefinitionExtensions
+          definition.force_exclude_groups = exclude_groups
+          definition
+        end
       end
 
       # Returns any groups to exclude specified from both licensed configuration
@@ -121,46 +98,33 @@ module Licensed
         end
       end
 
-      # Returns the path to the Bundler Gemfile
-      def gemfile_path
-        @gemfile_path ||= GEMFILES.keys
-                                  .map { |g| config.pwd.join g }
-                                  .find { |f| f.exist? }
-      end
+      # helper to clear all bundler environment around a yielded block
+      def with_application_environment
+        backup = nil
 
-      # Returns the path to the Bundler Gemfile.lock
-      def lockfile_path
-        return unless gemfile_path
-        @lockfile_path ||= gemfile_path.dirname.join(GEMFILES[gemfile_path.basename.to_s])
-      end
+        ::Bundler.ui.silence do
+          if ::Bundler.root != config.source_path
+            backup = ENV.to_hash
+            ENV.replace(::Bundler.original_env)
 
-      # helper to clear all bundler environment around a yielded block
-      def with_local_configuration
-        # silence any bundler warnings while running licensed
-        bundler_ui, ::Bundler.ui = ::Bundler.ui, ::Bundler::UI::Silent.new
+            # reset bundler to load from the current app's source path
+            ::Bundler.reset!
+          end
 
-        original_bundle_gemfile = nil
-        if gemfile_path.to_s != ENV["BUNDLE_GEMFILE"]
-          # force bundler to use the local gem file
-          original_bundle_gemfile, ENV["BUNDLE_GEMFILE"] = ENV["BUNDLE_GEMFILE"], gemfile_path.to_s
+          # ensure the bundler environment is loaded before enumeration
+          ::Bundler.load
 
-          # reset all bundler configuration
-          ::Bundler.reset!
-          # and re-configure with settings for current directory
-          ::Bundler.configure
+          yield
         end
-
-        yield
       ensure
-        if original_bundle_gemfile
-          ENV["BUNDLE_GEMFILE"] = original_bundle_gemfile
-
+        if backup
           # restore bundler configuration
+          ENV.replace(backup)
           ::Bundler.reset!
-          ::Bundler.configure
         end
 
-        ::Bundler.ui = bundler_ui
+        # reload the bundler environment after enumeration
+        ::Bundler.load
       end
 
       # Returns whether the current licensed execution is running ruby-packer

data/lib/licensed/sources/cargo.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Licensed
+  module Sources
+    class Cargo < Source
+      # Source is enabled when the cargo tool and Cargo.toml manifest file are available
+      def enabled?
+        return false unless Licensed::Shell.tool_available?("cargo")
+        config.pwd.join("Cargo.toml").exist?
+      end
+
+      def enumerate_dependencies
+        packages.map do |package|
+          Dependency.new(
+            name: "#{package["name"]}-#{package["version"]}",
+            version: package["version"],
+            path: File.dirname(package["manifest_path"]),
+            metadata: {
+              "name" => package["name"],
+              "type" => Cargo.type,
+              "summary" => package["description"],
+              "homepage" => package["homepage"]
+            }
+          )
+        end
+      end
+
+      # Returns the package data for all dependencies used to build the current package
+      def packages
+        cargo_metadata_resolved_node_ids.map { |id| cargo_metadata_packages[id] }
+      end
+
+      # Returns the ids of all resolved nodes used to build the current package
+      def cargo_metadata_resolved_node_ids
+        cargo_metadata.dig("resolve", "nodes")
+                      .map { |node| node["id"] }
+                      .reject { |id| cargo_metadata_workspace_members.include?(id) }
+
+      end
+
+      # Returns a hash of id => package pairs sourced from the "packages" cargo metadata property
+      def cargo_metadata_packages
+        @cargo_metadata_packages ||= cargo_metadata["packages"].each_with_object({}) do |package, hsh|
+          hsh[package["id"]] = package
+        end
+      end
+
+      # Returns a set of the ids of packages in the current workspace
+      def cargo_metadata_workspace_members
+        @cargo_metadata_workspace_members ||= Set.new(Array(cargo_metadata["workspace_members"]))
+      end
+
+      # Returns parsed JSON metadata returned from the cargo CLI
+      def cargo_metadata
+        @cargo_metadata ||= JSON.parse(cargo_metadata_command)
+      rescue JSON::ParserError => e
+        message = "Licensed was unable to parse the output from 'cargo metadata'. JSON Error: #{e.message}"
+        raise Licensed::Sources::Source::Error, message
+      end
+
+      # Runs a command to get cargo metadata for the current package
+      def cargo_metadata_command
+        options = Array(config.dig("cargo", "metadata_options")).flat_map(&:split)
+        Licensed::Shell.execute("cargo", "metadata", "--format-version=1", *options)
+      end
+    end
+  end
+end

data/lib/licensed/sources/dep.rb

@@ -40,10 +40,10 @@ module Licensed
         end
       end
 
-      # Returns the godoc.org page for a package.
+      # Returns the pkg.go.dev page for a package.
       def homepage(import_path)
         return unless import_path
-        "https://godoc.org/#{import_path}"
+        "https://pkg.go.dev/#{import_path}"
       end
 
       # Returns whether the package is part of the go std list.  Replaces

data/lib/licensed/sources/go.rb

@@ -98,7 +98,7 @@ module Licensed
       # Returns whether the package is local to the current project
       def local_package?(package)
         return false unless package && package["Dir"]
-        return false unless File.fnmatch?("#{config.root.to_s}*", package["Dir"], File::FNM_CASEFOLD)
+        return false unless File.fnmatch?("#{config.root}*", package["Dir"], File::FNM_CASEFOLD)
         vendored_path_parts(package).nil?
       end
 
@@ -132,10 +132,10 @@ module Licensed
         end
       end
 
-      # Returns the godoc.org page for a package.
+      # Returns the pkg.go.dev page for a package.
       def homepage(import_path)
         return unless import_path
-        "https://godoc.org/#{import_path}"
+        "https://pkg.go.dev/#{import_path}"
       end
 
       # Returns the root directory to search for a package license

data/lib/licensed/sources/helpers/content_versioning.rb

@@ -61,11 +61,12 @@ module Licensed
 
         paths = paths.compact.select { |path| File.file?(path) }
         return if paths.empty?
-
+        # rubocop:disable GitHub/InsecureHashAlgorithm
         paths.sort
              .reduce(Digest::XXHash64.new, :file)
              .digest
              .to_s(16) # convert to hex
+        # rubocop:enable GitHub/InsecureHashAlgorithm
       end
     end
   end

data/lib/licensed/sources/nuget.rb

@@ -234,8 +234,7 @@ module Licensed
         ].compact
 
         nuget_package_dirs.map { |dir| File.join(dir, dependency_path) }
-                          .select { |path| File.directory?(path) }
-                          .first
+                          .find { |path| File.directory?(path) }
       end
     end
   end

data/lib/licensed/sources.rb

@@ -5,6 +5,7 @@ module Licensed
     require "licensed/sources/bower"
     require "licensed/sources/bundler"
     require "licensed/sources/cabal"
+    require "licensed/sources/cargo"
     require "licensed/sources/composer"
     require "licensed/sources/dep"
     require "licensed/sources/git_submodule"

data/lib/licensed/version.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 module Licensed
-  VERSION = "3.2.0".freeze
+  VERSION = "3.3.0".freeze
 
   def self.previous_major_versions
     major_version = Gem::Version.new(Licensed::VERSION).segments.first

data/licensed.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |spec|
   spec.homepage      = "https://github.com/github/licensed"
   spec.license       = "MIT"
 
-  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test/|script/|docker/|\..+)}) }
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
@@ -26,16 +26,16 @@ Gem::Specification.new do |spec|
   spec.add_dependency "licensee", ">= 9.14.0", "< 10.0.0"
   spec.add_dependency "thor", ">= 0.19"
   spec.add_dependency "pathname-common_prefix", "~> 0.0.1"
-  spec.add_dependency "tomlrb", "~> 1.2"
+  spec.add_dependency "tomlrb", ">= 1.2", "< 3.0"
   spec.add_dependency "bundler", ">= 1.10"
   spec.add_dependency "ruby-xxHash", "~> 0.4"
   spec.add_dependency "parallel", ">= 0.18.0"
-  spec.add_dependency "reverse_markdown", "~> 1.0"
+  spec.add_dependency "reverse_markdown", ">= 1", "< 3"
 
   spec.add_development_dependency "rake", ">= 12.3.3"
   spec.add_development_dependency "minitest", "~> 5.8"
   spec.add_development_dependency "mocha", "~> 1.0"
-  spec.add_development_dependency "rubocop", "~> 0.49", "< 0.67"
+  spec.add_development_dependency "rubocop", "~> 0.49", "< 1.20"
   spec.add_development_dependency "rubocop-github", "~> 0.6"
-  spec.add_development_dependency "byebug", "~> 10.0.0"
+  spec.add_development_dependency "byebug", "~> 11.0.1"
 end