licensed 2.9.1 → 2.12.0

data/CHANGELOG.md

@@ -6,6 +6,47 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 2.12.0
+2020-06-19
+
+### Added
+- `--sources` argument for cache, list, status and notices commands to filter running sources (https://github.com/github/licensed/pull/287)
+
+### Fixed
+- `cache` command will not remove files outside of enabled source cache paths (https://github.com/github/licensed/pull/287)
+
+## 2.11.1
+2020-06-09
+
+### Fixed
+- `notices` command properly reads cached dependency notices contents (https://github.com/github/licensed/pull/283)
+
+## 2.11.0
+2020-06-02
+
+### Added
+- `notices` command to create a `NOTICE` file for each configured app (https://github.com/github/licensed/pull/277)
+
+### Fixed
+- NuGet source no longer crashes on a non-existent dependency path (https://github.com/github/licensed/pull/280)
+- Go source no longer crashes on a non-existent dependency package path (https://github.com/github/licensed/pull/274)
+
+## 2.10.0
+2020-05-15
+
+### Changed
+- NPM source ignores missing peer dependencies (https://github.com/github/licensed/pull/267)
+
+### Added
+- NuGet source (:tada: @zarenner https://github.com/github/licensed/pull/261)
+- Multiple apps can share a single cache location (https://github.com/github/licensed/pull/263)
+
+## 2.9.2
+2020-04-28
+
+### Changed
+- `licensee` minimum version bumped to 9.13.2 (https://github.com/github/licensed/pull/256)
+
 ## 2.9.1
 2020-03-24
 
@@ -286,4 +327,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/2.9.1...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/2.11.1...HEAD

data/README.md

@@ -24,13 +24,13 @@ See the [migration documentation](./docs/migrating_to_newer_versions.md) for mor
 ### Dependencies
 
 Licensed uses the `libgit2` bindings for Ruby provided by `rugged`. `rugged` requires `cmake` and `pkg-config` which you may need to install before you can install Licensed.
-   
+
    >  Ubuntu
-    
+
     sudo apt-get install cmake pkg-config
-    
+
    >  OS X
-       
+
     brew install cmake pkg-config
 
 ### With a Gemfile
@@ -64,8 +64,10 @@ For system wide usage, install licensed to a location on `$PATH`, e.g. `/usr/loc
 
 - `licensed list`: Output enumerated dependencies only.
 - `licensed cache`: Cache licenses and metadata.
-- `licensed status`: Check status of dependencies' cached licenses. For example:
+- `licensed status`: Check status of dependencies' cached licenses.
+- `licensed notices`: Write a `NOTICE` file for each application configuration.
 - `licensed version`: Show current installed version of Licensed. Aliases: `-v|--version`
+- `licensed env`: Output environment information from the licensed configuration.
 
 See the [commands documentation](./docs/commands.md) for additional documentation, or run `licensed -h` to see all of the current available commands.
 
@@ -102,14 +104,16 @@ Dependencies will be automatically detected for all of the following sources by
 1. [Bundler](./docs/sources/bundler.md)
 1. [Cabal](./docs/sources/cabal.md)
 1. [Composer](./docs/sources/composer.md)
+1. [Git Submodules (git_submodule)](./docs/sources/git_submodule.md)
 1. [Go](./docs/sources/go.md)
 1. [Go Dep (dep)](./docs/sources/dep.md)
+1. [Gradle](./docs/sources/gradle.md)
 1. [Manifest lists (manifests)](./docs/sources/manifests.md)
+1. [Mix](./docs/sources/mix.md)
 1. [NPM](./docs/sources/npm.md)
+1. [NuGet](./docs/sources/nuget.md)
 1. [Pip](./docs/sources/pip.md)
 1. [Pipenv](./docs/sources/pipenv.md)
-1. [Git Submodules (git_submodule)](./docs/sources/git_submodule.md)
-1. [Mix](./docs/sources/mix.md)
 1. [Yarn](./docs/sources/yarn.md)
 
 You can disable any of them in the configuration file:

data/docs/commands.md

@@ -6,10 +6,14 @@ Run `licensed -h` to see help content for running licensed commands.
 
 Running the list command finds the dependencies for all sources in all configured applications.  No additional actions are taken on each dependency.
 
+An optional `--sources` flag can be given to limit which dependency sources are run.  This is a filter over sources that are enabled via the licensed configuration file and cannot be used to run licensed with a disabled source.
+
 ## `cache`
 
 The cache command finds all dependencies and ensures that each dependency has an up-to-date cached record.
 
+An optional `--sources` flag can be given to limit which dependency sources are run.  This is a filter over sources that are enabled via the licensed configuration file and cannot be used to run licensed with a disabled source.
+
 Dependency records will be saved if:
 1. The `force` option is set
 2. No cached record is found
@@ -22,6 +26,8 @@ After the cache command is run, any cached records that don't match up to a curr
 
 The status command finds all dependencies and checks whether each dependency has a valid cached record.
 
+An optional `--sources` flag can be given to limit which dependency sources are run.  This is a filter over sources that are enabled via the licensed configuration file and cannot be used to run licensed with a disabled source.
+
 A dependency will fail the status checks if:
 1. No cached record is found
 2. The cached record's version is different than the current dependency's version
@@ -31,6 +37,14 @@ A dependency will fail the status checks if:
 5. The cached record is flagged for re-review.
    - This occurs when the record's license text has changed since the record was reviewed.
 
+## `notices`
+
+Outputs license and notice text for all dependencies in each app into a `NOTICE` file in the app's `cache_path`.  If an app uses a shared cache path, the file name will contain the app name as well, e.g. `NOTICE.my_app`.
+
+An optional `--sources` flag can be given to limit which dependency sources are run.  This is a filter over sources that are enabled via the licensed configuration file and cannot be used to run licensed with a disabled source.
+
+The `NOTICE` file contents are retrieved from cached records, with the assumption that cached records have already been reviewed in a compliance workflow.
+
 ## `env`
 
 Prints the runtime environment used by licensed after loading a configuration file.  By default the output is in YAML format, but can be output in JSON using the `--json` flag.

data/docs/configuration.md

@@ -209,6 +209,26 @@ apps:
 
 In this example, the root configuration will contain a default cache path of `.licenses`.  `app1` will inherit this value and append it's name, resulting in a cache path of `.licenses/app1`.
 
+### Sharing caches between apps
+
+Dependency caches can be shared between apps by setting the same cache path on each app.
+
+```yaml
+apps:
+  - source_path: "path/to/app1"
+    cache_path: ".licenses/apps"
+  - source_path: "path/to/app2"
+    cache_path: ".licenses/apps"
+```
+
+When using a source path with a glob pattern, the apps created from the glob pattern can share a dependency by setting an explicit cache path and setting `shared_cache` to true.
+
+```yaml
+source_path: "path/to/apps/*"
+cache_path: ".licenses/apps"
+shared_cache: true
+```
+
 ## Source specific configuration
 
 See the [source documentation](./sources) for details on any source specific configuration.

data/docs/sources/nuget.md

@@ -0,0 +1,14 @@
+# NuGet
+
+The NuGet source will detect ProjectReference-style restored packages by inspecting `project.assets.json` files for dependencies. It requires that `dotnet restore` has already ran on the project.
+
+The source currently expects that `source_path` is set to the `obj` directory containing the `project.assets.json`.
+For example, if your project lives at `foo/foo.proj`, you likely want to set `source_path` to `foo/obj`.
+If in MSBuild you have customized your `obj` paths (e.g. to live outside your source tree), you may need to set `source_path` to something different such as `../obj/foo`.
+
+### Search strategy
+This source looks for licenses:
+1. Specified by SPDX expression via `<license type="expression">` in a package's `.nuspec` (via licensee)
+2. In license files such as `LICENSE.txt`, even if not specified in the `.nuspec` (via licensee)
+3. Specified by filepath via `<license type="file">` in a package's `.nuspec`, even if not a standard license filename.
+4. By downloading and inspecting the contents of `<licenseUrl>` in a package's `.nuspec`, if not found otherwise.

data/lib/licensed/cli.rb

@@ -10,22 +10,41 @@ module Licensed
       desc: "Overwrite licenses even if version has not changed."
     method_option :config, aliases: "-c", type: :string,
       desc: "Path to licensed configuration file"
+    method_option :sources, aliases: "-s", type: :array,
+      desc: "Individual source(s) to evaluate.  Must also be enabled via configuration."
     def cache
-      run Licensed::Commands::Cache.new(config: config), force: options[:force]
+      run Licensed::Commands::Cache.new(config: config),
+          { force: options[:force], sources: options[:sources] }
     end
 
     desc "status", "Check status of dependencies' cached licenses"
     method_option :config, aliases: "-c", type: :string,
       desc: "Path to licensed configuration file"
+    method_option :sources, aliases: "-s", type: :array,
+      desc: "Individual source(s) to evaluate.  Must also be enabled via configuration."
     def status
-      run Licensed::Commands::Status.new(config: config)
+      run Licensed::Commands::Status.new(config: config),
+          { sources: options[:sources] }
     end
 
     desc "list", "List dependencies"
     method_option :config, aliases: "-c", type: :string,
       desc: "Path to licensed configuration file"
+    method_option :sources, aliases: "-s", type: :array,
+      desc: "Individual source(s) to evaluate.  Must also be enabled via configuration."
     def list
-      run Licensed::Commands::List.new(config: config)
+      run Licensed::Commands::List.new(config: config),
+          { sources: options[:sources] }
+    end
+
+    desc "notices", "Generate a NOTICE file from cached records"
+    method_option :config, aliases: "-c", type: :string,
+      desc: "Path to licensed configuration file"
+    method_option :sources, aliases: "-s", type: :array,
+      desc: "Individual source(s) to evaluate.  Must also be enabled via configuration."
+    def notices
+      run Licensed::Commands::Notices.new(config: config),
+          { sources: options[:sources] }
     end
 
     map "-v" => :version

data/lib/licensed/commands.rb

@@ -6,5 +6,6 @@ module Licensed
     require "licensed/commands/status"
     require "licensed/commands/list"
     require "licensed/commands/environment"
+    require "licensed/commands/notices"
   end
 end

data/lib/licensed/commands/cache.rb

@@ -11,21 +11,47 @@ module Licensed
         Licensed::Reporters::CacheReporter.new
       end
 
+      # Run the command.
+      # Removes any cached records that don't match a current application
+      # dependency.
+      #
+      # options - Options to run the command with
+      #
+      # Returns whether the command was a success
+      def run(**options)
+        begin
+          result = super
+          clear_stale_cached_records if result
+
+          result
+        ensure
+          cache_paths.clear
+          files.clear
+        end
+      end
+
       protected
 
       # Run the command for all enumerated dependencies found in a dependency source,
       # recording results in a report.
-      # Removes any cached records that don't match a current application
-      # dependency.
+      # Enumerating dependencies in the source is skipped if a :sources option
+      # is provided and the evaluated `source.class.type` is not in the :sources values
       #
       # app - The application configuration for the source
       # source - A dependency source enumerator
       #
       # Returns whether the command succeeded for the dependency source enumerator
       def run_source(app, source)
-        result = super
-        clear_stale_cached_records(app, source) if result
-        result
+        super do |report|
+          if Array(options[:sources]).any? && !options[:sources].include?(source.class.type)
+            report.warnings << "skipped source"
+            next :skip
+          end
+
+          # add the full cache path to the list of cache paths
+          # that should be cleaned up after the command run
+          cache_paths << app.cache_path.join(source.class.type)
+        end
       end
 
       # Cache dependency record data.
@@ -62,6 +88,9 @@ module Licensed
           report.warnings << "expected dependency path #{dependency.path} does not exist"
         end
 
+        # add the absolute dependency file path to the list of files seen during this licensed run
+        files << filename.to_s
+
         true
       end
 
@@ -86,18 +115,26 @@ module Licensed
 
       # Clean up cached files that dont match current dependencies
       #
-      # app - An application configuration
-      # source - A dependency source enumerator
-      #
       # Returns nothing
-      def clear_stale_cached_records(app, source)
-        names = source.dependencies.map { |dependency| File.join(source.class.type, dependency.name) }
-        Dir.glob(app.cache_path.join(source.class.type, "**/*.#{DependencyRecord::EXTENSION}")).each do |file|
-          file_path = Pathname.new(file)
-          relative_path = file_path.relative_path_from(app.cache_path).to_s
-          FileUtils.rm(file) unless names.include?(relative_path.chomp(".#{DependencyRecord::EXTENSION}"))
+      def clear_stale_cached_records
+        cache_paths.each do |cache_path|
+          Dir.glob(cache_path.join("**/*.#{DependencyRecord::EXTENSION}")).each do |file|
+            next if files.include?(file)
+
+            FileUtils.rm(file)
+          end
         end
       end
+
+      # Set of unique cache paths that are evaluted during the run
+      def cache_paths
+        @cache_paths ||= Set.new
+      end
+
+      # Set of unique absolute file paths of cached records evaluted during the run
+      def files
+        @files ||= Set.new
+      end
     end
   end
 end

data/lib/licensed/commands/command.rb

@@ -21,7 +21,9 @@ module Licensed
         begin
           result = reporter.report_run(self) do |report|
             # allow additional report data to be given by commands
-            yield report if block_given?
+            if block_given?
+              next if (yield report) == :skip
+            end
 
             config.apps.sort_by { |app| app["name"] }
                        .map { |app| run_app(app) }
@@ -57,7 +59,9 @@ module Licensed
           Dir.chdir app.source_path do
             begin
               # allow additional report data to be given by commands
-              yield report if block_given?
+              if block_given?
+                next if (yield report) == :skip
+              end
 
               app.sources.select(&:enabled?)
                          .sort_by { |source| source.class.type }
@@ -81,7 +85,9 @@ module Licensed
         reporter.report_source(source) do |report|
           begin
             # allow additional report data to be given by commands
-            yield report if block_given?
+            if block_given?
+              next if (yield report) == :skip
+            end
 
             source.dependencies.sort_by { |dependency| dependency.name }
                                .map { |dependency| run_dependency(app, source, dependency) }
@@ -114,7 +120,9 @@ module Licensed
 
           begin
             # allow additional report data to be given by commands
-            yield report if block_given?
+            if block_given?
+              next if (yield report) == :skip
+            end
 
             evaluate_dependency(app, source, dependency, report)
           rescue Licensed::Shell::Error => err

data/lib/licensed/commands/list.rb

@@ -13,6 +13,25 @@ module Licensed
 
       protected
 
+      # Run the command for all enumerated dependencies found in a dependency source,
+      # recording results in a report.
+      # Enumerating dependencies in the source is skipped if a :sources option
+      # is provided and the evaluated `source.class.type` is not in the :sources values
+      #
+      # app - The application configuration for the source
+      # source - A dependency source enumerator
+      #
+      # Returns whether the command succeeded for the dependency source enumerator
+      def run_source(app, source)
+        super do |report|
+          next if Array(options[:sources]).empty?
+          next if options[:sources].include?(source.class.type)
+
+          report.warnings << "skipped source"
+          :skip
+        end
+      end
+
       # Listing dependencies requires no extra work.
       #
       # app - The application configuration for the dependency

data/lib/licensed/commands/notices.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+module Licensed
+  module Commands
+    class Notices < Command
+      # Create a reporter to use during a command run
+      #
+      # options - The options the command was run with
+      #
+      # Raises a Licensed::Reporters::CacheReporter
+      def create_reporter(options)
+        Licensed::Reporters::NoticesReporter.new
+      end
+
+      protected
+
+      # Run the command for all enumerated dependencies found in a dependency source,
+      # recording results in a report.
+      # Enumerating dependencies in the source is skipped if a :sources option
+      # is provided and the evaluated `source.class.type` is not in the :sources values
+      #
+      # app - The application configuration for the source
+      # source - A dependency source enumerator
+      #
+      # Returns whether the command succeeded for the dependency source enumerator
+      def run_source(app, source)
+        super do |report|
+          next if Array(options[:sources]).empty?
+          next if options[:sources].include?(source.class.type)
+
+          report.warnings << "skipped source"
+          :skip
+        end
+      end
+
+      # Load stored dependency record data to add to the notices report.
+      #
+      # app - The application configuration for the dependency
+      # source - The dependency source enumerator for the dependency
+      # dependency - An application dependency
+      # report - A report hash for the command to provide extra data for the report output.
+      #
+      # Returns true.
+      def evaluate_dependency(app, source, dependency, report)
+        filename = app.cache_path.join(source.class.type, "#{dependency.name}.#{DependencyRecord::EXTENSION}")
+        report["cached_record"] = Licensed::DependencyRecord.read(filename)
+        if !report["cached_record"]
+          report.warnings << "expected cached record not found at #{filename}"
+        end
+
+        true
+      end
+    end
+  end
+end