licensed 2.8.0 → 2.11.0

data/.gitignore

@@ -45,6 +45,8 @@ test/fixtures/mix/mix.lock
 test/fixtures/yarn/*
 !test/fixtures/yarn/package.json
 
+test/fixtures/nuget/obj/*
+
 vendor/licenses
 .licenses
 *.gem

data/CHANGELOG.md

@@ -6,6 +6,53 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 2.11.0
+2020-06-02
+
+### Added
+- `notices` command to create a `NOTICE` file for each configured app (https://github.com/github/licensed/pull/277)
+
+### Fixed
+- NuGet source no longer crashes on a non-existent dependency path (https://github.com/github/licensed/pull/280)
+- Go source no longer crashes on a non-existent dependency package path (https://github.com/github/licensed/pull/274)
+
+## 2.10.0
+2020-05-15
+
+### Changed
+- NPM source ignores missing peer dependencies (https://github.com/github/licensed/pull/267)
+
+### Added
+- NuGet source (:tada: @zarenner https://github.com/github/licensed/pull/261)
+- Multiple apps can share a single cache location (https://github.com/github/licensed/pull/263)
+
+## 2.9.2
+2020-04-28
+
+### Changed
+- `licensee` minimum version bumped to 9.13.2 (https://github.com/github/licensed/pull/256)
+
+## 2.9.1
+2020-03-24
+
+### Changed
+- relaxed gem version restrictions on Thor (:tada: @eileencodes https://github.com/github/licensed/pull/254)
+
+## 2.9.0
+2020-03-19
+
+### Added
+- Source paths use glob pattern matching (https://github.com/github/licensed/pull/245)
+
+### Fixed
+- Mix source supports updates to mix.lock format (:tada: @bruce https://github.com/github/licensed/pull/242)
+- Go source supports `go list` format changes in go 1.14 (https://github.com/github/licensed/pull/247)
+
+### Changed
+- `licensed cache` will flag dependencies for re-review when license text changes (https://github.com/github/licensed/pull/248)
+- `licensed status` will raise errors on dependencies that need re-review (https://github.com/github/licensed/pull/248)
+- `licensee` minimum version bumped to 9.13.1 (https://github.com/github/licensed/pull/251)
+
 ## 2.8.0
 2020-01-03
 
@@ -265,4 +312,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/2.8.0...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/2.11.0...HEAD

data/README.md

@@ -24,13 +24,13 @@ See the [migration documentation](./docs/migrating_to_newer_versions.md) for mor
 ### Dependencies
 
 Licensed uses the `libgit2` bindings for Ruby provided by `rugged`. `rugged` requires `cmake` and `pkg-config` which you may need to install before you can install Licensed.
-   
+
    >  Ubuntu
-    
+
     sudo apt-get install cmake pkg-config
-    
+
    >  OS X
-       
+
     brew install cmake pkg-config
 
 ### With a Gemfile
@@ -64,8 +64,10 @@ For system wide usage, install licensed to a location on `$PATH`, e.g. `/usr/loc
 
 - `licensed list`: Output enumerated dependencies only.
 - `licensed cache`: Cache licenses and metadata.
-- `licensed status`: Check status of dependencies' cached licenses. For example:
+- `licensed status`: Check status of dependencies' cached licenses.
+- `licensed notices`: Write a `NOTICE` file for each application configuration.
 - `licensed version`: Show current installed version of Licensed. Aliases: `-v|--version`
+- `licensed env`: Output environment information from the licensed configuration.
 
 See the [commands documentation](./docs/commands.md) for additional documentation, or run `licensed -h` to see all of the current available commands.
 
@@ -102,14 +104,16 @@ Dependencies will be automatically detected for all of the following sources by
 1. [Bundler](./docs/sources/bundler.md)
 1. [Cabal](./docs/sources/cabal.md)
 1. [Composer](./docs/sources/composer.md)
+1. [Git Submodules (git_submodule)](./docs/sources/git_submodule.md)
 1. [Go](./docs/sources/go.md)
 1. [Go Dep (dep)](./docs/sources/dep.md)
+1. [Gradle](./docs/sources/gradle.md)
 1. [Manifest lists (manifests)](./docs/sources/manifests.md)
+1. [Mix](./docs/sources/mix.md)
 1. [NPM](./docs/sources/npm.md)
+1. [NuGet](./docs/sources/nuget.md)
 1. [Pip](./docs/sources/pip.md)
 1. [Pipenv](./docs/sources/pipenv.md)
-1. [Git Submodules (git_submodule)](./docs/sources/git_submodule.md)
-1. [Mix](./docs/sources/mix.md)
 1. [Yarn](./docs/sources/yarn.md)
 
 You can disable any of them in the configuration file:

data/docs/commands.md

@@ -28,6 +28,14 @@ A dependency will fail the status checks if:
 3. The cached record's `licenses` data is empty
 4. The cached record's `license` metadata doesn't match an `allowed` license from the dependency's application configuration.
    - If `license: other` is specified and all of the `licenses` entries match an `allowed` license a failure will not be logged
+5. The cached record is flagged for re-review.
+   - This occurs when the record's license text has changed since the record was reviewed.
+
+## `notices`
+
+Outputs license and notice text for all dependencies in each app into a `NOTICE` file in the app's `cache_path`.  If an app uses a shared cache path, the file name will contain the app name as well, e.g. `NOTICE.my_app`.
+
+The `NOTICE` file contents are retrieved from cached records, with the assumption that cached records have already been reviewed in a compliance workflow.
 
 ## `env`
 

data/docs/configuration.md

@@ -19,6 +19,22 @@ If a root path is not specified, it will default to using the following, in orde
 1. the root of the local git repository, if run inside a git repository
 2. the current directory
 
+### Source path glob patterns
+
+The `source_path` property can use a glob path to share configuration properties across multiple application entrypoints.
+
+For example, there is a common pattern in go projects to include multiple executable entrypoints under folders in `cmd`.  Using a glob pattern allows users to avoid manually configuring and maintaining multiple licensed application `source_path`s.  Using a glob pattern will also ensure that any new entrypoints matching the pattern are automatically picked up by licensed commands as they are added.
+
+```yml
+sources:
+  go: true
+
+# treat all directories under `cmd` as separate apps
+source_path: cmd/*
+```
+
+Glob patterns are syntactic sugar for, and provide the same functionality as, manually specifying multiple `source_path` values. See the instructions on [specifying multiple apps](./#specifying-multiple-apps) below for additional considerations when using multiple apps.
+
 ## Restricting sources
 
 The `sources` configuration property specifies which sources `licensed` will use to enumerate dependencies.
@@ -193,6 +209,26 @@ apps:
 
 In this example, the root configuration will contain a default cache path of `.licenses`.  `app1` will inherit this value and append it's name, resulting in a cache path of `.licenses/app1`.
 
+### Sharing caches between apps
+
+Dependency caches can be shared between apps by setting the same cache path on each app.
+
+```yaml
+apps:
+  - source_path: "path/to/app1"
+    cache_path: ".licenses/apps"
+  - source_path: "path/to/app2"
+    cache_path: ".licenses/apps"
+```
+
+When using a source path with a glob pattern, the apps created from the glob pattern can share a dependency by setting an explicit cache path and setting `shared_cache` to true.
+
+```yaml
+source_path: "path/to/apps/*"
+cache_path: ".licenses/apps"
+shared_cache: true
+```
+
 ## Source specific configuration
 
 See the [source documentation](./sources) for details on any source specific configuration.

data/docs/sources/nuget.md

@@ -0,0 +1,14 @@
+# NuGet
+
+The NuGet source will detect ProjectReference-style restored packages by inspecting `project.assets.json` files for dependencies. It requires that `dotnet restore` has already ran on the project.
+
+The source currently expects that `source_path` is set to the `obj` directory containing the `project.assets.json`.
+For example, if your project lives at `foo/foo.proj`, you likely want to set `source_path` to `foo/obj`.
+If in MSBuild you have customized your `obj` paths (e.g. to live outside your source tree), you may need to set `source_path` to something different such as `../obj/foo`.
+
+### Search strategy
+This source looks for licenses:
+1. Specified by SPDX expression via `<license type="expression">` in a package's `.nuspec` (via licensee)
+2. In license files such as `LICENSE.txt`, even if not specified in the `.nuspec` (via licensee)
+3. Specified by filepath via `<license type="file">` in a package's `.nuspec`, even if not a standard license filename.
+4. By downloading and inspecting the contents of `<licenseUrl>` in a package's `.nuspec`, if not found otherwise.

data/lib/licensed/cli.rb

@@ -28,6 +28,13 @@ module Licensed
       run Licensed::Commands::List.new(config: config)
     end
 
+    desc "notices", "Generate a NOTICE file from cached records"
+    method_option :config, aliases: "-c", type: :string,
+      desc: "Path to licensed configuration file"
+    def notices
+      run Licensed::Commands::Notices.new(config: config)
+    end
+
     map "-v" => :version
     map "--version" => :version
     desc "version", "Show Installed Version of Licensed, [-v, --version]"

data/lib/licensed/commands.rb

@@ -6,5 +6,6 @@ module Licensed
     require "licensed/commands/status"
     require "licensed/commands/list"
     require "licensed/commands/environment"
+    require "licensed/commands/notices"
   end
 end

data/lib/licensed/commands/cache.rb

@@ -11,20 +11,39 @@ module Licensed
         Licensed::Reporters::CacheReporter.new
       end
 
+      # Run the command.
+      # Removes any cached records that don't match a current application
+      # dependency.
+      #
+      # options - Options to run the command with
+      #
+      # Returns whether the command was a success
+      def run(**options)
+        begin
+          result = super
+          clear_stale_cached_records if result
+
+          result
+        ensure
+          cache_paths.clear
+          files.clear
+        end
+      end
+
       protected
 
-      # Run the command for all enumerated dependencies found in a dependency source,
+      # Run the command for all enabled sources for an application configuration,
       # recording results in a report.
-      # Removes any cached records that don't match a current application
-      # dependency.
       #
-      # app - The application configuration for the source
-      # source - A dependency source enumerator
+      # app - An application configuration
       #
-      # Returns whether the command succeeded for the dependency source enumerator
-      def run_source(app, source)
+      # Returns whether the command succeeded for the application.
+      def run_app(app)
         result = super
-        clear_stale_cached_records(app, source) if result
+
+        # add the full cache path to the list of cache paths evaluted during this run
+        cache_paths << app.cache_path
+
         result
       end
 
@@ -45,8 +64,15 @@ module Licensed
         filename = app.cache_path.join(source.class.type, "#{dependency.name}.#{DependencyRecord::EXTENSION}")
         cached_record = Licensed::DependencyRecord.read(filename)
         if options[:force] || save_dependency_record?(dependency, cached_record)
-          # use the cached license value if the license text wasn't updated
-          dependency.record["license"] = cached_record["license"] if dependency.record.matches?(cached_record)
+          if dependency.record.matches?(cached_record)
+            # use the cached license value if the license text wasn't updated
+            dependency.record["license"] = cached_record["license"]
+          elsif cached_record && app.reviewed?(dependency.record)
+            # if the license text changed and the dependency is set as reviewed
+            # force a re-review of the dependency
+            dependency.record["review_changed_license"] = true
+          end
+
           dependency.record.save(filename)
           report["cached"] = true
         end
@@ -55,6 +81,9 @@ module Licensed
           report.warnings << "expected dependency path #{dependency.path} does not exist"
         end
 
+        # add the absolute dependency file path to the list of files seen during this licensed run
+        files << filename.to_s
+
         true
       end
 
@@ -79,18 +108,26 @@ module Licensed
 
       # Clean up cached files that dont match current dependencies
       #
-      # app - An application configuration
-      # source - A dependency source enumerator
-      #
       # Returns nothing
-      def clear_stale_cached_records(app, source)
-        names = source.dependencies.map { |dependency| File.join(source.class.type, dependency.name) }
-        Dir.glob(app.cache_path.join(source.class.type, "**/*.#{DependencyRecord::EXTENSION}")).each do |file|
-          file_path = Pathname.new(file)
-          relative_path = file_path.relative_path_from(app.cache_path).to_s
-          FileUtils.rm(file) unless names.include?(relative_path.chomp(".#{DependencyRecord::EXTENSION}"))
+      def clear_stale_cached_records
+        cache_paths.each do |cache_path|
+          Dir.glob(cache_path.join("**/*.#{DependencyRecord::EXTENSION}")).each do |file|
+            next if files.include?(file)
+
+            FileUtils.rm(file)
+          end
         end
       end
+
+      # Set of unique cache paths that are evaluted during the run
+      def cache_paths
+        @cache_paths ||= Set.new
+      end
+
+      # Set of unique absolute file paths of cached records evaluted during the run
+      def files
+        @files ||= Set.new
+      end
     end
   end
 end

data/lib/licensed/commands/environment.rb

@@ -23,14 +23,14 @@ module Licensed
             "allowed" => config["allowed"],
             "ignored" => config["ignored"],
             "reviewed" => config["reviewed"],
-            "version_strategy" => self.version_strategy
+            "version_strategy" => self.version_strategy,
+            "root" => config.root
           }
         end
       end
 
       def run(**options)
         super do |report|
-          report["root"] = config.root
           report["git_repo"] = Licensed::Git.git_repo?
         end
       end

data/lib/licensed/commands/notices.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+module Licensed
+  module Commands
+    class Notices < Command
+      # Create a reporter to use during a command run
+      #
+      # options - The options the command was run with
+      #
+      # Raises a Licensed::Reporters::CacheReporter
+      def create_reporter(options)
+        Licensed::Reporters::NoticesReporter.new
+      end
+
+      protected
+
+      # Load stored dependency record data to add to the notices report.
+      #
+      # app - The application configuration for the dependency
+      # source - The dependency source enumerator for the dependency
+      # dependency - An application dependency
+      # report - A report hash for the command to provide extra data for the report output.
+      #
+      # Returns true.
+      def evaluate_dependency(app, source, dependency, report)
+        filename = app.cache_path.join(source.class.type, "#{dependency.name}.#{DependencyRecord::EXTENSION}")
+        report["cached_record"] = Licensed::DependencyRecord.read(filename)
+        if !report["cached_record"]
+          report["warning"] = "expected cached record not found at #{filename}"
+        end
+
+        true
+      end
+    end
+  end
+end

data/lib/licensed/commands/status.rb

@@ -35,7 +35,11 @@ module Licensed
         else
           report.errors << "cached dependency record out of date" if cached_record["version"] != dependency.version
           report.errors << "missing license text" if cached_record.licenses.empty?
-          report.errors << "license needs review: #{cached_record["license"]}" if license_needs_review?(app, cached_record)
+          if cached_record["review_changed_license"]
+            report.errors << "license text has changed and needs re-review. if the new text is ok, remove the `review_changed_license` flag from the cached record"
+          elsif license_needs_review?(app, cached_record)
+            report.errors << "license needs review: #{cached_record["license"]}"
+          end
         end
 
         report.errors.empty?

data/lib/licensed/configuration.rb

@@ -4,40 +4,39 @@ require "pathname"
 module Licensed
   class AppConfiguration < Hash
     DEFAULT_CACHE_PATH = ".licenses".freeze
-    DEFAULT_CONFIG_FILES = [
-      ".licensed.yml".freeze,
-      ".licensed.yaml".freeze,
-      ".licensed.json".freeze
-    ].freeze
+
+    # Returns the root for a configuration in following order of precendence:
+    # 1. explicitly configured "root" property
+    # 2. a found git repository root
+    # 3. the current directory
+    def self.root_for(configuration)
+      configuration["root"] || Licensed::Git.repository_root || Dir.pwd
+    end
 
     def initialize(options = {}, inherited_options = {})
       super()
 
       # update order:
       # 1. anything inherited from root config
-      # 2. app defaults
-      # 3. explicitly configured app settings
+      # 2. explicitly configured app settings
       update(inherited_options)
-      update(defaults_for(options, inherited_options))
       update(options)
+      verify_arg "source_path"
 
       self["sources"] ||= {}
       self["reviewed"] ||= {}
       self["ignored"] ||= {}
       self["allowed"] ||= []
-
-      # default the root to the git repository root,
-      # or the current directory if no other options are available
-      self["root"] ||= Licensed::Git.repository_root || Dir.pwd
-
-      verify_arg "source_path"
-      verify_arg "cache_path"
+      self["root"] = AppConfiguration.root_for(self)
+      # defaults to the directory name of the source path if not set
+      self["name"] ||= File.basename(self["source_path"])
+      # setting the cache path might need a valid app name
+      self["cache_path"] = detect_cache_path(options, inherited_options)
     end
 
     # Returns the path to the workspace root as a Pathname.
-    # Defaults to Licensed::Git.repository_root if not explicitly set
     def root
-      Pathname.new(self["root"])
+      @root ||= Pathname.new(self["root"])
     end
 
     # Returns the path to the app cache directory as a Pathname
@@ -102,13 +101,20 @@ module Licensed
 
     private
 
-    def defaults_for(options, inherited_options)
-      name = options["name"] || File.basename(options["source_path"])
-      cache_path = inherited_options["cache_path"] || DEFAULT_CACHE_PATH
-      {
-        "name" => name,
-        "cache_path" => File.join(cache_path, name)
-      }
+    # Returns the cache path for the application based on:
+    # 1. An explicitly set cache path for the application, if set
+    # 2. An inherited root cache path joined with the app name
+    # 3. The default cache path joined with the app name
+    def detect_cache_path(options, inherited_options)
+      return options["cache_path"] unless options["cache_path"].to_s.empty?
+
+      # if cache_path and shared_cache are both set in inherited_options,
+      # don't append the app name to the cache path
+      cache_path = inherited_options["cache_path"]
+      return cache_path if cache_path && inherited_options["shared_cache"] == true
+
+      cache_path ||= DEFAULT_CACHE_PATH
+      File.join(cache_path, self["name"])
     end
 
     def verify_arg(property)
@@ -118,9 +124,18 @@ module Licensed
     end
   end
 
-  class Configuration < AppConfiguration
+  class Configuration
+    DEFAULT_CONFIG_FILES = [
+      ".licensed.yml".freeze,
+      ".licensed.yaml".freeze,
+      ".licensed.json".freeze
+    ].freeze
+
     class LoadError < StandardError; end
 
+    # An array of the applications in this licensed configuration.
+    attr_reader :apps
+
     # Loads and returns a Licensed::Configuration object from the given path.
     # The path can be relative or absolute, and can point at a file or directory.
     # If the path given is a directory, the directory will be searched for a
@@ -133,21 +148,41 @@ module Licensed
 
     def initialize(options = {})
       apps = options.delete("apps") || []
-      super(default_options.merge(options))
-
-      self["apps"] = apps.map { |app| AppConfiguration.new(app, options) }
-    end
-
-    # Returns an array of the applications for this licensed configuration.
-    # If the configuration did not explicitly configure any applications,
-    # return self as an application configuration.
-    def apps
-      return [self] if self["apps"].empty?
-      self["apps"]
+      apps << default_options.merge(options) if apps.empty?
+      apps = apps.flat_map { |app| self.class.expand_app_source_path(app) }
+      @apps = apps.map { |app| AppConfiguration.new(app, options) }
     end
 
     private
 
+    def self.expand_app_source_path(app_config)
+      return app_config if app_config["source_path"].to_s.empty?
+
+      source_path = File.expand_path(app_config["source_path"], AppConfiguration.root_for(app_config))
+      expanded_source_paths = Dir.glob(source_path).select { |p| File.directory?(p) }
+      # return the original configuration if glob didn't result in multiple paths
+      return app_config if expanded_source_paths.size <= 1
+
+      # map the expanded paths to new application configurations
+      expanded_source_paths.map do |path|
+        config = app_config.merge("source_path" => path)
+
+        # update configured values for name and cache_path for uniqueness.
+        # this is only needed when values are explicitly set, AppConfiguration
+        # will handle configurations that don't have these explicitly set
+        dir_name = File.basename(path)
+        config["name"] = "#{config["name"]}-#{dir_name}" if config["name"]
+
+        # if a cache_path is set and is not marked as shared, append the app name
+        # to the end of the cache path to make a unique cache path for the app
+        if config["cache_path"] && config["shared_cache"] != true
+          config["cache_path"] = File.join(config["cache_path"], dir_name)
+        end
+
+        config
+      end
+    end
+
     # Find a default configuration file in the given directory.
     # File preference is given by the order of elements in DEFAULT_CONFIG_FILES
     #
@@ -198,7 +233,7 @@ module Licensed
       # manually set a cache path without additional name
       {
         "source_path" => Dir.pwd,
-        "cache_path" => DEFAULT_CACHE_PATH
+        "cache_path" => AppConfiguration::DEFAULT_CACHE_PATH
       }
     end
   end