licensed 3.2.2 → 3.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7ed29bbc7c7c643bf2eda9aacca1d64bd5ccb6e9211e4eeda006c33ec1fe3900
-  data.tar.gz: eeefafd3638220b419d57353947ab74a17bc5504fca50562a2d63baf0562e485
+  metadata.gz: a3cb5bf8c20bf1177466536fbc30bc29135a9e76933e225af8353ba8eb89205d
+  data.tar.gz: 26acda6f4b9d90c2457725ecda8f0b00bc7fc793a4993a1e6aa9a6810a995d04
 SHA512:
-  metadata.gz: 9d54389571ce81aadee0c3a46871f38c8c87768d86abd8c1ef50978378d23d4a17524c4188de8f306f4156a2b0ae670a5e437279aa458113b89e311919296332
-  data.tar.gz: 3a93838012493a51fd430815d42fa7b3b52ba85ea2513a879517a437e00af05d47f7325cfdeac2440adf299f05af0b0fbf2b0a16bb4b34caa81db4becc632808
+  metadata.gz: 5d71df74f6fca5c309230b18a7423f608cdf61ca0a5ddf9f45116b75681196a15ecee70d4d56877841b82c1d75d43fa573182ec300f1976a1e815f169de1598f
+  data.tar.gz: 645c4bf3cbb162e46061b730f947f0299336506fa425640f215e0e96ccb89124ef0e8a9725e6b9044fcd1c8e38c8809b7f5a911479d6583cc097b15d1f4a48c3

data/CHANGELOG.md

@@ -6,6 +6,57 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 3.4.0
+
+2021-12-14
+
+### Added
+
+- New Yarn enumerator with support for berry versions (https://github.com/github/licensed/pull/423)
+
+### Fixed
+
+- Error handling cases return correct values in the Yarn enumerator (https://github.com/github/licensed/pull/425)
+- Fixed link in command documentation (:tada: @chibicco https://github.com/github/licensed/pull/416)
+- Fixed minor backwards compatibility issue for Ruby 2.3 support (:tada: @dzunk https://github.com/github/licensed/pull/414)
+
+### Changed
+
+- Licensed's own dependencies are cached in the repository and kept up to date with GitHub Actions (https://github.com/github/licensed/pull/421)
+
+## 3.3.1
+
+2021-10-07
+
+### Fixed
+
+- Fix evaluation of peer dependencies with npm 7 (:tada: @manuelpuyol https://github.com/github/licensed/pull/411)
+
+### Changed
+
+- Manifest source evaluation performance improvements (https://github.com/github/licensed/pull/407)
+
+## 3.3.0
+
+2021-09-18
+
+### Added
+
+- New cargo source enumerates rust dependencies (https://github.com/github/licensed/pull/404)
+
+### Changed
+
+- Removed non-functional files from gem builds (https://github.com/github/licensed/pull/405)
+
+## 3.2.3
+
+2021-09-14
+
+### Fixed
+
+- Bundler source will no longer infinitely recurse when enumerating specifications (https://github.com/github/licensed/pull/402)
+- Using the `--sources` command line option will no longer delete skipped sources' cached files (https://github.com/github/licensed/pull/401)
+
 ## 3.2.2
 
 2021-09-09
@@ -488,4 +539,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/3.2.2...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/3.4.0...HEAD

data/Rakefile

@@ -2,13 +2,16 @@
 require "bundler/gem_tasks"
 require "rake/testtask"
 require "rubocop/rake_task"
+require "licensed"
 
 desc "Run source setup scripts"
 task :setup, [:arguments] do |task, args|
   arguments = args[:arguments].to_s.split
   force = arguments.include?("-f") ? "-f" : ""
 
-  Dir["script/source-setup/*"].each do |script|
+  Dir["script/source-setup/**/*"].each do |script|
+    next if File.directory?(script)
+
     # green
     puts "\033[32mRunning #{script}.\e[0m"
 
@@ -27,8 +30,7 @@ task :setup, [:arguments] do |task, args|
   end
 end
 
-sources_search = File.expand_path("lib/licensed/sources/*.rb", __dir__)
-sources = Dir[sources_search].map { |f| File.basename(f, ".*") }
+sources = Licensed::Sources::Source.sources.map { |source| source.full_type }
 
 namespace :test do
   sources.each do |source|

data/docs/adding_a_new_source.md

@@ -13,8 +13,37 @@ Dependency enumerators inherit and override the [`Licensed::Sources::Source`](..
 
 ### Optional method overrides
 
-1. `Licensed::Sources::Source.type`
-   - Returns the name of the current dependency enumerator as it is found in a licensed configuration file.
+1. `Licensed::Sources::Source.type_and_version`
+   - Returns the name, and optionally a version, of the current dependency enumerator as it is found in a licensed configuration file.  See [the method description](../lib/licensed/sources/source.rb#L38-L41) for more details
+
+### Implementing an enumerator for a new version of an existing source
+
+If a package manager introduces breaking changes, it can be easier to build a new implementation rather than making a single class work for all cases.  To enable seamless migration between source versions, the implementation for each version of the source enumerator should return the same `.type` and determine whether the version implementation should run in `#enabled?`.
+
+The sections below describe what was done when adding a new version for the `yarn` source.  Following these steps will make sure that the new version implementation follows the expected patterns for local development and test scenarios.
+
+#### Migrating the file structure for a single source enumerator to enable multiple source enumerator versions
+
+The following steps will migrate the source to the pattern expected for multi-version source enumerators.  
+
+The enumerators source code file is likely named to closely match the source enumerator, e.g. `lib/licensed/sources/yarn.rb`
+
+1. Create a new directory matching the name of the source and move the existing enumerator into the new folder with a version descriptive name, e.g. `lib/licensed/sources/yarn/v1.rb`
+1. Update the source enumerator class name to include a version identifier, e.g. `Licensed::Sources::Yarn::V1`
+1. Make similar changes for the source's [unit test fixtures](../test/fixtures), [unit test file](../test/sources) and [setup script](../scripts/source-setup), moving these files into subfolders and renaming the files to match the change in (1)
+   - Also be sure to update any references to old paths or class names
+1. If needed, update the source's `#type_and_version` to include a version value as a second array value
+   - If this isn't already set, the default implementation will return the type and version as the last two part names of the class name, snake cased and with a `/` delimeter,  e.g. `yarn/v1`
+1. Update the source's `#enabled?` method, adding a version check to ensure that the source only runs in the expected scenario
+1. Add a new generic source file in `lib/licensed/sources` that `require`s the new file, e.g. `lib/licensed/sources/yarn.rb`
+   - This is also an ideal spot to put shared code in a module that can be included in one or more versions of the source enumerator
+1. Update any references to the source in scripting and GitHub Actions automation to use the new versioned identifier, e.g. `yarn/v1` instead of the unversioned identifier.
+
+#### Adding a new implementation for the new version of the source
+
+1. Add the new implementation to the source's `lib/licensed/sources` subfolder.
+1. If there is shared code that can be reused between multiple source enumerator versions, put it in a module in the source's base file, e.g. `lib/licensed/sources/yarn.rb`.  Include the module in the version implementations.
+1. Ensure that the new version implementation checks for the expected source enumerator version in `#enabled?`
 
 ## Determining if dependencies should be enumerated
 
@@ -24,7 +53,7 @@ whether `Licensed::Source::Sources#enumerate_dependencies` should be called on t
 Determining whether dependencies should be enumerated depends on whether all the tools or files needed to find dependencies are present.
 For example, to enumerate `npm` dependencies the `npm` CLI tool must be found with `Licensed::Shell.tool_available?` and a `package.json` file needs to exist in the licensed app's configured [`source_path`](./configuration.md#configuration-paths).
 
-### Gating functionality when required tools are not available.
+### Gating functionality when required tools are not available
 
 When adding new dependency sources, ensure that `script/bootstrap` scripting and tests are only run if the required tooling is available on the development machine.
 

data/docs/commands/README.md

@@ -8,7 +8,7 @@ Run `licensed -h` to see help content for running licensed commands.
 - [migrate](migrate.md)
 - [notices](notices.md)
 - [status](status.md)
-- [version](verison.md)
+- [version](version.md)
 
 Most commands accept a `-c`/`--config` option to specify a path to a configuration file or directory. If a directory is specified, `licensed` will look in that directory for a file named (in order of preference):
 

data/docs/sources/cargo.md

@@ -0,0 +1,19 @@
+# Cargo
+
+The cargo source will detect dependencies when `Cargo.toml` is found at an apps `source_path`.  The source uses the `cargo metadata` CLI and reports on all dependencies that are listed in the output in `resolve.nodes`, excluding packages that are listed in `workspace_members`.
+
+## Metadata CLI options
+
+Licensed by default runs `cargo metadata --format-version=1`.  You can specify additional CLI options by specifying them in your licensed configuration file under `cargo.metadata_options`.  The configuration can be set as a string, or as an array of strings for multiple options.
+
+```yml
+cargo:
+  metadata_options: '--all-features'
+```
+
+```yml
+cargo:
+  metadata_options:
+    - '--all-features'
+    - '--filter-platform x86_64-pc-windows-msvc'
+```

data/docs/sources/yarn.md

@@ -2,13 +2,14 @@
 
 The yarn source will detect dependencies when `package.json` and `yarn.lock` are found at an app's `source_path`.
 
-It uses `yarn list` to enumerate dependencies and `yarn info` to get metadata on each package.
+It uses the `yarn` CLI commands to enumerate dependencies and gather metadata on each package.
 
-### Including development dependencies
+## Including development dependencies
 
-Yarn versions < 1.3.0 will always include non-production dependencies due to a bug in those yarn versions.
+**Note** Yarn versions < 1.3.0 will always include non-production dependencies due to a bug in those versions of yarn.
+**Note** Yarn versions > 2.0 will always include non-production dependencies due to lack of filtering of production vs non-production dependencies in the yarn CLI.
 
-Starting with yarn version >= 1.3.0, the yarn source excludes non-production dependencies by default.  To include development and test dependencies, set `production_only: false` in `.licensed.yml`.
+For yarn versions between 1.3.0 and 2.0, the yarn source excludes non-production dependencies by default.  To include development and test dependencies in these versions, set `production_only: false` in `.licensed.yml`.
 
 ```yml
 yarn:

data/lib/licensed/commands/cache.rb

@@ -39,11 +39,13 @@ module Licensed
       #
       # Returns whether the command succeeded for the dependency source enumerator
       def run_source(app, source, report)
+        result = super
+
         # add the full cache path to the list of cache paths
         # that should be cleaned up after the command run
-        cache_paths << app.cache_path.join(source.class.type)
+        cache_paths << app.cache_path.join(source.class.type) unless result == :skipped
 
-        super
+        result
       end
 
       # Cache dependency record data.

data/lib/licensed/commands/command.rb

@@ -121,13 +121,16 @@ module Licensed
       # source - A dependency source enumerator
       # report - A report object for this source
       #
-      # Returns whether the command succeeded for the dependency source enumerator
+      # Returns whether the command succeeded, failed, or was skipped for the dependency source enumerator
       def run_source(app, source, report)
         reporter.begin_report_source(source, report)
 
         if !sources_overrides.empty? && !sources_overrides.include?(source.class.type)
           report.warnings << "skipped source"
-          return true
+
+          # return a symbol to speficy the source was skipped.
+          # This is truthy and will result in the source being considered successful
+          return :skipped
         end
 
         dependencies = source.dependencies.sort_by { |dependency| dependency.name }

data/lib/licensed/reporters/status_reporter.rb

@@ -49,7 +49,7 @@ module Licensed
         errored_reports = all_reports.select { |r| r.errors.any? }.to_a
 
         dependency_count = all_reports.count { |r| r.target.is_a?(Licensed::Dependency) }
-        error_count = errored_reports.sum { |r| r.errors.size }
+        error_count = errored_reports.reduce(0) { |count, r| count + r.errors.size }
 
         if error_count > 0
           shell.newline

data/lib/licensed/sources/bundler/missing_specification.rb

@@ -38,17 +38,20 @@ module Licensed
         "could not find #{name} (#{version}) in any sources"
       end
     end
+
+    module LazySpecification
+      def __materialize__
+        spec = super
+        return spec if spec
+
+        Licensed::Bundler::MissingSpecification.new(name: name, version: version, platform: platform, source: source)
+      end
+    end
   end
 end
 
 module Bundler
   class LazySpecification
-    alias_method :orig_materialize, :__materialize__
-    def __materialize__
-      spec = orig_materialize
-      return spec if spec
-
-      Licensed::Bundler::MissingSpecification.new(name: name, version: version, platform: platform, source: source)
-    end
+    prepend ::Licensed::Bundler::LazySpecification
   end
 end

data/lib/licensed/sources/cargo.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Licensed
+  module Sources
+    class Cargo < Source
+      # Source is enabled when the cargo tool and Cargo.toml manifest file are available
+      def enabled?
+        return false unless Licensed::Shell.tool_available?("cargo")
+        config.pwd.join("Cargo.toml").exist?
+      end
+
+      def enumerate_dependencies
+        packages.map do |package|
+          Dependency.new(
+            name: "#{package["name"]}-#{package["version"]}",
+            version: package["version"],
+            path: File.dirname(package["manifest_path"]),
+            metadata: {
+              "name" => package["name"],
+              "type" => Cargo.type,
+              "summary" => package["description"],
+              "homepage" => package["homepage"]
+            }
+          )
+        end
+      end
+
+      # Returns the package data for all dependencies used to build the current package
+      def packages
+        cargo_metadata_resolved_node_ids.map { |id| cargo_metadata_packages[id] }
+      end
+
+      # Returns the ids of all resolved nodes used to build the current package
+      def cargo_metadata_resolved_node_ids
+        cargo_metadata.dig("resolve", "nodes")
+                      .map { |node| node["id"] }
+                      .reject { |id| cargo_metadata_workspace_members.include?(id) }
+
+      end
+
+      # Returns a hash of id => package pairs sourced from the "packages" cargo metadata property
+      def cargo_metadata_packages
+        @cargo_metadata_packages ||= cargo_metadata["packages"].each_with_object({}) do |package, hsh|
+          hsh[package["id"]] = package
+        end
+      end
+
+      # Returns a set of the ids of packages in the current workspace
+      def cargo_metadata_workspace_members
+        @cargo_metadata_workspace_members ||= Set.new(Array(cargo_metadata["workspace_members"]))
+      end
+
+      # Returns parsed JSON metadata returned from the cargo CLI
+      def cargo_metadata
+        @cargo_metadata ||= JSON.parse(cargo_metadata_command)
+      rescue JSON::ParserError => e
+        message = "Licensed was unable to parse the output from 'cargo metadata'. JSON Error: #{e.message}"
+        raise Licensed::Sources::Source::Error, message
+      end
+
+      # Runs a command to get cargo metadata for the current package
+      def cargo_metadata_command
+        options = Array(config.dig("cargo", "metadata_options")).flat_map(&:split)
+        Licensed::Shell.execute("cargo", "metadata", "--format-version=1", *options)
+      end
+    end
+  end
+end

data/lib/licensed/sources/manifest.rb

@@ -61,7 +61,7 @@ module Licensed
         manifest.each_with_object({}) do |(src, package_name), hsh|
           next if src.nil? || src.empty?
           hsh[package_name] ||= []
-          hsh[package_name] << File.join(config.root, src)
+          hsh[package_name] << File.absolute_path(src, config.root)
         end
       end
 
@@ -130,19 +130,17 @@ module Licensed
         @configured_dependencies ||= begin
           dependencies = config.dig("manifest", "dependencies")&.dup || {}
 
-          dependencies.each do |name, patterns|
+          dependencies.each_with_object({}) do |(name, patterns), hsh|
             # map glob pattern(s) listed for the dependency to a listing
             # of files that match the patterns and are not excluded
-            dependencies[name] = files_from_pattern_list(patterns) & included_files
+            hsh[name] = files_from_pattern_list(patterns) & included_files
           end
-
-          dependencies
         end
       end
 
       # Returns the set of project files that are included in dependency evaluation
       def included_files
-        @sources ||= all_files - files_from_pattern_list(config.dig("manifest", "exclude"))
+        @included_files ||= tracked_files - files_from_pattern_list(config.dig("manifest", "exclude"))
       end
 
       # Finds and returns all files in the project that match
@@ -151,26 +149,23 @@ module Licensed
         return Set.new if patterns.nil? || patterns.empty?
 
         # evaluate all patterns from the project root
-        Dir.chdir config.root do
-          Array(patterns).reduce(Set.new) do |files, pattern|
-            if pattern.start_with?("!")
-              # if the pattern is an exclusion, remove all matching files
-              # from the result
-              files - Dir.glob(pattern[1..-1], File::FNM_DOTMATCH)
-            else
-              # if the pattern is an inclusion, add all matching files
-              # to the result
-              files + Dir.glob(pattern, File::FNM_DOTMATCH)
-            end
+        Array(patterns).each_with_object(Set.new) do |pattern, files|
+          if pattern.start_with?("!")
+            # if the pattern is an exclusion, remove all matching files
+            # from the result
+            files.subtract(Dir.glob(pattern[1..-1], File::FNM_DOTMATCH, base: config.root))
+          else
+            # if the pattern is an inclusion, add all matching files
+            # to the result
+            files.merge(Dir.glob(pattern, File::FNM_DOTMATCH, base: config.root))
           end
         end
       end
 
-      # Returns all tracked files in the project
-      def all_files
-        # remove files if they are tracked but don't exist on the file system
-        @all_files ||= Set.new(Licensed::Git.files || [])
-                          .delete_if { |f| !File.exist?(File.join(Licensed::Git.repository_root, f)) }
+      # Returns all tracked files in the project as the intersection of what git tracks and the files in the project
+      def tracked_files
+        @tracked_files ||= Set.new(Array(Licensed::Git.files)) &
+                           Set.new(Dir.glob("**/*", File::FNM_DOTMATCH, base: config.root))
       end
 
       class Dependency < Licensed::Dependency

data/lib/licensed/sources/npm.rb

@@ -23,10 +23,6 @@ module Licensed
         end
       end
 
-      def self.type
-        "npm"
-      end
-
       def enabled?
         Licensed::Shell.tool_available?("npm") && File.exist?(config.pwd.join("package.json"))
       end
@@ -66,15 +62,17 @@ module Licensed
 
       # Recursively parse dependency JSON data.  Returns a hash mapping the
       # package name to it's metadata
-      def recursive_dependencies(dependencies, result = {})
+      def recursive_dependencies(dependencies, result = {}, parent = nil)
         dependencies.each do |name, dependency|
-          next if dependency["peerMissing"]
+          next if missing_peer?(parent, dependency, name)
           next if yarn_lock_present && dependency["missing"]
           next if dependency["extraneous"] && dependency["missing"]
 
           dependency["name"] = name
+          dependency["version"] ||= extract_version(parent, name) if dependency["missing"]
+
           (result[name] ||= []) << dependency
-          recursive_dependencies(dependency["dependencies"] || {}, result)
+          recursive_dependencies(dependency["dependencies"] || {}, result, dependency)
         end
         result
       end
@@ -135,6 +133,18 @@ module Licensed
       def include_non_production?
         config.dig("npm", "production_only") == false
       end
+
+      def missing_peer?(parent, dependency, name)
+        dependency["peerMissing"] || (dependency["missing"] && peer_dependency(parent, name))
+      end
+
+      def peer_dependency(parent, name)
+        parent&.dig("peerDependencies", name)
+      end
+
+      def extract_version(parent, name)
+        parent&.dig("_dependencies", name) || peer_dependency(parent, name)
+      end
     end
   end
 end

data/lib/licensed/sources/nuget.rb

@@ -7,8 +7,8 @@ module Licensed
     # Only supports ProjectReference (project.assets.json) style restore used in .NET Core.
     # Does not currently support packages.config style restore.
     class NuGet < Source
-      def self.type
-        "nuget"
+      def self.type_and_version
+        ["nuget"]
       end
 
       class NuGetDependency < Licensed::Dependency

data/lib/licensed/sources/source.rb

@@ -19,13 +19,32 @@ module Licensed
           (@sources ||= []) << klass
         end
 
-        # Returns the source name as the snake cased class name
+        # Returns the source name as the first snake cased class or module name
+        # following "Licensed::Sources::".  This is the type that is included
+        # in metadata files and cache paths.
+        # e.g. for `Licensed::Sources::Yarn::V1`, this returns "yarn"
         def type
-          self.name.split(/::/)
-                   .last
+          type_and_version[0]
+        end
+
+        # Returns the source name as a "/" delimited string of all the module and
+        # class names following "Licensed::Sources::".  This is the type that is
+        # used to distinguish multiple versions of a sources from each other.
+        # e.g. for `Licensed::Sources::Yarn::V1`, this returns `yarn/v1`
+        def full_type
+          type_and_version.join("/")
+        end
+
+        # Returns an array that includes the source's type name at the first index, and
+        # optionally a version string for the source as the second index.
+        # Callers should override this function and not `type` or `full_type` when
+        # needing to adjust the default type and version parsing logic
+        def type_and_version
+          self.name.gsub("#{Licensed::Sources.name}::", "")
                    .gsub(/([A-Z\d]+)([A-Z][a-z])/, "\\1_\\2".freeze)
                    .gsub(/([a-z\d])([A-Z])/, "\\1_\\2".freeze)
                    .downcase
+                   .split("::")
         end
       end
 

data/lib/licensed/sources/yarn/berry.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+require "json"
+
+module Licensed
+  module Sources
+    class Yarn::Berry < Source
+      include Licensed::Sources::Yarn
+
+      def self.version_requirement
+        Gem::Requirement.new(">= 2.0")
+      end
+
+      def enumerate_dependencies
+        packages.map do |name, package|
+          Dependency.new(
+            name: name,
+            version: package["version"],
+            path: package["path"],
+            metadata: {
+              "type"     => self.class.type,
+              "name"     => package["name"],
+              "homepage" => package["homepage"]
+            }
+          )
+        end
+      end
+
+      # Finds packages that the current project relies on based on the output from `yarn info`
+      def packages
+        # parse all lines of output to json and find one that is "type": "tree"
+        yarn_info = JSON.parse("[#{yarn_info_command.lines.join(",")}]")
+        mapped_packages = yarn_info.reduce({}) do |accum, package|
+          name, _ = package["value"].rpartition("@")
+          version = package.dig("children", "Version")
+          id = "#{name}-#{version}"
+
+          accum[name] ||= []
+          accum[name] << {
+            "id" => id,
+            "name" => name,
+            "version" => version,
+            "homepage" => package.dig("children", "Manifest", "Homepage"),
+            "path" => dependency_paths[id]
+          }
+          accum
+        end
+
+        mapped_packages.each_with_object({}) do |(name, results), hsh|
+          results.uniq! { |package| package["version"] }
+          if results.size == 1
+            # if there is only one package for a name, reference it by name
+            hsh[name] = results[0]
+          else
+            # if there is more than one package for a name, reference each by id
+            results.each do |package|
+              hsh[package["id"]] = package
+            end
+          end
+        end
+      end
+
+      # Returns a hash that maps all dependency names to their location on disk
+      # by parsing every package.json file under node_modules.
+      def dependency_paths
+        @dependency_paths ||= Dir.glob(config.pwd.join("node_modules/**/package.json")).each_with_object({}) do |file, hsh|
+          dirname = File.dirname(file)
+          json = JSON.parse(File.read(file))
+          hsh["#{json["name"]}-#{json["version"]}"] = dirname
+        end
+      end
+
+      # Returns the output from running `yarn list` to get project dependencies
+      def yarn_info_command
+        args = %w(--json --manifest --recursive --all)
+        Licensed::Shell.execute("yarn", "info", *args)
+      end
+    end
+  end
+end