licensed 3.4.3 → 3.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6ff397378a053e97414fb762f3d06c41f4217f61d9e63130aa3786456e6f2114
-  data.tar.gz: b197f77bc54c68e5bf9462917351ca2dbda1509813315a8207fd31ed4c611de0
+  metadata.gz: a328b5551bdf77593f4bf97f4a846b7792898b6f749c25f5c5f39e68669f2164
+  data.tar.gz: ac9b2013cf25d9dab94aadd2122a41bfa2790d741bd9e1588a270cd122dfaddb
 SHA512:
-  metadata.gz: 68d86169b07dd46f28de9cd31745e705f82e8f49e1968df5d8ff90139d6c60d2b31d861ca79d39657b5af956b3d73540cf492db6f6f84589132dc27ab65e33e5
-  data.tar.gz: '005804bf6877d487434dc74fe9c2fbbe9c57c15bfd7924fc3a3992f7879c878056c871c4f5d7867b5e5254a59751b8839363fb4de60e1d43cdfeb6e2e1ae67f1'
+  metadata.gz: 846cadb01c2045ea258a785767ebfc8df3cee9d3a05648c93291c94cc21b7e84fb83146476c4afb64f9bc137a530cb84ba523ec41e2c4938396629b5b8901795
+  data.tar.gz: 4aa7028294894b9f0c1781b558032d80f87669e959e71aa1b635c7a8687f77b4cf11be6431f280cfcbc06ea25f605da061a9af2fb480ea134a4172f989060451

data/CHANGELOG.md

@@ -6,6 +6,40 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 3.6.0
+
+2022-03-17
+
+### Added
+
+- Composer dev dependencies can optionally be included in enumerated PHP dependencies (:tada: @digilist https://github.com/github/licensed/pull/486)
+- Getting started usage documentation (https://github.com/github/licensed/pull/483)
+- Initial support for NPM workspaces (https://github.com/github/licensed/pull/485)
+
+### Changed
+
+- Transitive dependencies are now enumerated by the `pip` source (https://github.com/github/licensed/pull/480)
+
+### Fixed
+
+- `licensed cache --force` will now correctly overwrite existing license classifications (https://github.com/github/licensed/pull/473)
+
+## 3.5.0
+
+2022-02-24
+
+### Added
+
+- [Licensee](https://github.com/licensee/licensee) confidence thresholds can be configured in the licensed configuration file (https://github.com/github/licensed/pull/455)
+
+## 3.4.4
+
+2022-02-07
+
+### Fixed
+
+- The npm and pip sources have better protection from strings causing crashes in `Hash#dig` (https://github.com/github/licensed/pull/450)
+
 ## 3.4.3
 
 2022-01-31
@@ -563,4 +597,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/3.4.3...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/3.6.0...HEAD

data/README.md

@@ -74,6 +74,8 @@ For system wide usage, install licensed to a location on `$PATH`, e.g. `/usr/loc
 
 ## Usage
 
+See [getting started](./docs/getting_started.md) for guidance using Licensed as part of your developer workflow.
+
 ### Available commands
 
 See the [commands documentation](./docs/commands) for documentation on available commands, or run `licensed -h` to see all of the current available commands.
@@ -86,18 +88,6 @@ A configuration file is required for most commands.  See the [configuration file
 
 Licensed can enumerate dependency for many languages, package managers, and frameworks.  See the [sources documentation](./docs/sources) for the list of currently available sources.  Sources can be explicitly enabled and disabled as a [configuration option](./docs/configuration/dependency_source_enumerators.md.md).
 
-### Automation
-
-#### Bundler
-
-The [bundler-licensed plugin](https://github.com/sergey-alekseev/bundler-licensed) runs `licensed cache` automatically when using `bundler`.  See the linked repo for usage and details.
-
-#### GitHub Actions
-
-The [licensed-ci](https://github.com/marketplace/actions/licensed-ci) GitHub Action runs `licensed` as part of an opinionated CI workflow and can be configured to run on any GitHub Action event.  See the linked actions for usage and details.
-
-The [setup-licensed](https://github.com/marketplace/actions/setup-github-licensed) GitHub Action installs `licensed` to the workflow environment.  See the linked actions for usage and details.
-
 ## Development
 
 To get started after checking out the repo, run

data/docs/configuration/customizing_licensee.md

@@ -0,0 +1,13 @@
+# Customize Licensee's behavior
+
+Licensed uses [Licensee](https://github.com/licensee/licensee) to detect and evaluate OSS licenses for project dependencies found during source enumeration.  Licensed can optionally [customize Licensee's behavior](https://github.com/licensee/licensee/blob/jonabc-patch-1/docs/customizing.md#customizing-licensees-behavior) based on options set in the configuration file.
+
+**NOTE** Matching licenses based on package manager metadata and README references is always enabled and cannot currently be configured.
+
+```yml
+licensee:
+  # the confidence threshold is an integer between 1 and 100. the value represents
+  # the minimum percentage confidence that Licensee must have to report a matched license
+  # https://github.com/licensee/licensee/blob/master/docs/customizing.md#adjusting-the-confidence-threshold
+  confidence_threshold: 90 # default value: 98
+```

data/docs/configuration.md

@@ -4,7 +4,7 @@ A configuration file specifies the details of enumerating and operating on licen
 
 Configuration can be specified in either YML or JSON formats, with examples given in YML.  The example
 below describes common configuration values and their purposes.  See [configuration options documentation](./configuration)
-for in depth information.  
+for in depth information.
 
 Additionally, some dependency sources have their own specific configuration options.  See the [source documentation](./sources) for details.
 

data/docs/getting_started.md

@@ -0,0 +1,40 @@
+# Getting Started
+
+Licensed's core workflow is a multi-step process:
+
+1. `licensed cache` ([docs](./commands/cache.md)) is run manually and/or in an automated workflow
+   - Creates or updates files in a git repo containing metadata including licenses and other legal text for each dependency used by a project
+1. `licensed status` ([docs](./commands/status.md)) is run manually and/or in an automated workflow
+   - Validate that every detected dependency has a metadata file written in the repository, and that each dependency's stored metadata passes a number of compliance checks
+1. Any detected errors/warnings are manually [resolved](./commands/status.md#status-errors-and-resolutions)
+1. Repeat the above steps until all dependencies have metadata files stored in the repository and `licensed status` is not reporting any errors.
+
+## Caching dependency metadata
+
+Caching depedency metadata into the repository brings the metadata contents closer to the dependencies where they are used, making status validation faster and possible in offline scenarios.  Keeping metadata alongside your code in git gives teams an easily auditable trail for dependency updates over time, and ties into common review practices to ensure that changes aren't quietly ignored.
+
+Caching metadata should be done whenever project code changes, to ensure that metadata files are in sync with the current state of the project code.
+
+## Checking dependency metadata status
+
+Dependency metadata checks verify that every dependency
+
+- has a metadata file available, or has been explicitly ignored by the project owners or OSS experts
+- is using an approved OSS license, or has been reviewed and signed off by an OSS expert
+- is up to date with the current state of a project
+
+Checking dependencies for compliance violations should be performed whenever code changes in a repository.  Moving compliance checks inline in the development workflow reduces friction later, and can even prevent costly situations later if a non-compliant dependency would need to be removed from a project.
+
+## Automated workflows
+
+Integrating github/licensed into your workflow can be tedious, and luckily there are a few automated tools available to make usage easier.
+
+### Bundler
+
+The [bundler-licensed plugin](https://github.com/sergey-alekseev/bundler-licensed) runs `licensed cache` automatically when using `bundler`.  See the linked repo for usage and details.
+
+### GitHub Actions
+
+The [licensed-ci](https://github.com/marketplace/actions/licensed-ci) GitHub Action runs `licensed` as part of an opinionated CI workflow and can be configured to run on any GitHub Action event to automatically update the cached metadata files and check their status.  See the linked action for usage and details.
+
+The [setup-licensed](https://github.com/marketplace/actions/setup-github-licensed) GitHub Action installs `licensed` to the workflow environment.  See the linked actions for usage and details.

data/docs/sources/composer.md

@@ -12,3 +12,12 @@ The default composer application file location is `<repository root>/composer.ph
 composer:
   application_path: "/path/to/composer"
 ```
+
+### Dev dependencies
+
+By default licensed ignores all dev dependencies. To consider dev dependencies as well, use the `composer.include_dev` configuration setting.
+
+```yml
+composer:
+  include_dev: true
+```

data/docs/sources/npm.md

@@ -2,11 +2,21 @@
 
 The npm source will detect dependencies `package.json` is found at an apps `source_path`.  It uses `npm list` to enumerate dependencies and metadata.
 
-### Including development dependencies
+## Including development dependencies
 
-By default, the npm source will exclude all non-development dependencies.  To include development or test dependencies, set `production_only: false` in the licensed configuration.
+By default, the npm source will exclude all development dependencies. To include development or test dependencies, set `production_only: false` in the licensed configuration.
 
 ```yml
 npm:
   production_only: false
 ```
+
+## Using licensed with npm workspaces
+
+Licensed requires npm version 8.5.0 or later to enumerate dependencies inside of npm workspaces.  For the best results, treat each workspace directory as a separate app `source_path`:
+
+```yml
+apps:
+  - source_path: path/to/workspace/a
+  - source_path: path/to/workspace/b
+```

data/docs/sources/pip.md

@@ -2,11 +2,6 @@
 
 The pip source uses `pip` CLI commands to enumerate dependencies and properties. It is expected that `pip` is available in the `virtual_env_dir` specific directory before running `licensed`.
 
-Your repository root should also contain a `requirements.txt` file which contains all the packages and dependences that are needed. You can generate one with `pip` using the command:
-```
-pip freeze > requirements.txt
-```
-
 A `virtualenv` directory is required before running `licensed`. You can setup a `virtualenv` by running the command:
 ```
 virtualenv <your_venv_dir>
@@ -20,5 +15,5 @@ You have to add this setting to your licensed configuration file.
 An example usage of this might look like:
 ```yaml
 python:
-    virtual_env_dir:"/path/to/your/venv_dir"
+    virtual_env_dir: "/path/to/your/venv_dir"
 ```

data/lib/licensed/commands/cache.rb

@@ -29,6 +29,18 @@ module Licensed
         files.clear
       end
 
+      # Run the command for an application configurations.
+      # Applies a licensee configuration for the duration of the operation.
+      #
+      # report - A Licensed::Report object for this command
+      #
+      # Returns whether the command succeeded
+      def run_app(app, report)
+        with_licensee_configuration(app, report) do
+          super
+        end
+      end
+
       # Run the command for all enumerated dependencies found in a dependency source,
       # recording results in a report.
       # Enumerating dependencies in the source is skipped if a :sources option
@@ -70,15 +82,8 @@ module Licensed
         report["filename"] = filename.to_s
         report["version"] = dependency.version
 
-        if options[:force] || save_dependency_record?(dependency, cached_record)
-          if dependency.record.matches?(cached_record)
-            # use the cached license value if the license text wasn't updated
-            dependency.record["license"] = cached_record["license"]
-          elsif cached_record && app.reviewed?(dependency.record)
-            # if the license text changed and the dependency is set as reviewed
-            # force a re-review of the dependency
-            dependency.record["review_changed_license"] = true
-          end
+        if save_dependency_record?(dependency, cached_record)
+          update_dependency_from_cached_record(app, dependency, cached_record)
 
           dependency.record.save(filename)
           report["cached"] = true
@@ -107,6 +112,7 @@ module Licensed
       # Returns true if dependency's record should be saved
       def save_dependency_record?(dependency, cached_record)
         return true if cached_record.nil?
+        return true if options[:force]
 
         cached_version = cached_record["version"]
         return true if cached_version.nil? || cached_version.empty?
@@ -114,6 +120,23 @@ module Licensed
         false
       end
 
+      # Update dependency metadata from the cached record, to support:
+      # 1. continuity between cache runs to cut down on churn
+      # 2. notifying users when changed content needs to be reviewed
+      def update_dependency_from_cached_record(app, dependency, cached_record)
+        return if cached_record.nil?
+        return if options[:force]
+
+        if dependency.record.matches?(cached_record)
+          # use the cached license value if the license text wasn't updated
+          dependency.record["license"] = cached_record["license"]
+        elsif app.reviewed?(dependency.record)
+          # if the license text changed and the dependency is set as reviewed
+          # force a re-review of the dependency
+          dependency.record["review_changed_license"] = true
+        end
+      end
+
       # Clean up cached files that dont match current dependencies
       #
       # Returns nothing
@@ -136,6 +159,22 @@ module Licensed
       def files
         @files ||= Set.new
       end
+
+      # Configure licensee for the duration of a yielded operation
+      def with_licensee_configuration(app, report)
+        licensee_configuration = app["licensee"]
+        return yield unless licensee_configuration
+
+        report["licensee"] = licensee_configuration
+
+        if new_threshold = licensee_configuration["confidence_threshold"]
+          old_threshold, Licensee.confidence_threshold = Licensee.confidence_threshold, new_threshold
+        end
+
+        yield
+      ensure
+        Licensee.confidence_threshold = old_threshold if old_threshold
+      end
     end
   end
 end

data/lib/licensed/sources/composer.rb

@@ -28,7 +28,10 @@ module Licensed
       end
 
       def packages
-        JSON.parse(File.read(composer_lock))["packages"]
+        packages = JSON.parse(File.read(composer_lock))
+        return packages["packages"] unless include_dev?
+
+        packages["packages"] + packages["packages-dev"]
       end
 
       # Returns the output from running `php composer.phar` to get package metadata
@@ -56,6 +59,11 @@ module Licensed
       def composer_lock
         config.pwd.join("composer.lock")
       end
+
+      # Returns whether to include dev packages based on the licensed configuration settings
+      def include_dev?
+        config.dig("composer", "include_dev") == true
+      end
     end
   end
 end

data/lib/licensed/sources/npm.rb

@@ -24,11 +24,13 @@ module Licensed
       end
 
       def enabled?
-        Licensed::Shell.tool_available?("npm") && File.exist?(config.pwd.join("package.json"))
+        Licensed::Shell.tool_available?("npm") && File.exist?(package_json_path)
       end
 
       def enumerate_dependencies
         packages.map do |name, package|
+          next if package["name"] == project_name
+
           errors = package["problems"] unless package["path"]
           Dependency.new(
             name: name,
@@ -147,12 +149,38 @@ module Licensed
       end
 
       def peer_dependency(parent, name)
-        parent&.dig("peerDependencies", name)
+        return unless parent.is_a?(Hash)
+
+        peerDependencies = parent["peerDependencies"]
+        # "peerDependencies" could be set to the string "[Circular]"
+        return unless peerDependencies.is_a?(Hash)
+
+        peerDependencies[name]
       end
 
       def extract_version(parent, name)
         parent&.dig("_dependencies", name) || peer_dependency(parent, name)
       end
+
+      # Returns the current projects name
+      def project_name
+        return unless package_json
+        package_json["name"]
+      end
+
+      ## Returns the parse package.json for the current project
+      def package_json
+        return unless File.exist?(package_json_path)
+
+        @package_json ||= JSON.parse(File.read(package_json_path))
+      rescue JSON::ParserError => e
+        message = "Licensed was unable to parse package.json. JSON Error: #{e.message}"
+        raise Licensed::Sources::Source::Error, message
+      end
+
+      def package_json_path
+        @package_json_path ||= File.join(config.pwd, "package.json")
+      end
     end
   end
 end

data/lib/licensed/sources/pip.rb

@@ -7,17 +7,14 @@ require "parallel"
 module Licensed
   module Sources
     class Pip < Source
-      VERSION_OPERATORS = %w(< > <= >= == !=).freeze
-      PACKAGE_REGEX = /^([\w\.-]+)(#{VERSION_OPERATORS.join("|")})?/
+      PACKAGE_INFO_SEPARATOR = "\n---\n"
 
       def enabled?
-        return unless virtual_env_pip && Licensed::Shell.tool_available?(virtual_env_pip)
-        File.exist?(config.pwd.join("requirements.txt"))
+        virtual_env_pip && Licensed::Shell.tool_available?(virtual_env_pip)
       end
 
       def enumerate_dependencies
-        Parallel.map(packages_from_requirements_txt, in_threads: Parallel.processor_count) do |package_name|
-          package = package_info(package_name)
+        packages.map do |package|
           location = File.join(package["Location"], package["Name"].gsub("-", "_") +  "-" + package["Version"] + ".dist-info")
           Dependency.new(
             name: package["Name"],
@@ -34,25 +31,45 @@ module Licensed
 
       private
 
-      def packages_from_requirements_txt
-        File.read(config.pwd.join("requirements.txt"))
-            .lines
-            .reject { |line| line.include?("://") }
-            .map { |line| line.strip.match(PACKAGE_REGEX) { |match| match.captures.first } }
-            .compact
+      # Returns parsed information for all packages used by the project,
+      # using `pip list` to determine what packages are used and `pip show`
+      # to gather package information
+      def packages
+        all_packages = pip_show_command(package_names)
+        all_packages.split(PACKAGE_INFO_SEPARATOR).reduce([]) do |accum, val|
+          accum << parse_package_info(val)
+        end
+      end
+
+      # Returns the names of all of the packages used by the current project,
+      # as returned from `pip list`
+      def package_names
+        @package_names ||= begin
+          JSON.parse(pip_list_command).map { |package| package["name"] }
+        rescue JSON::ParserError => e
+          message = "Licensed was unable to parse the output from 'npm list'. JSON Error: #{e.message}"
+          raise Licensed::Sources::Source::Error, message
+        end
       end
 
-      def package_info(package_name)
-        p_info = pip_command(package_name).lines
-        p_info.each_with_object(Hash.new(0)) { |pkg, a|
+      # Returns a hash filled with package info parsed from the email-header formatted output
+      # returned by `pip show`
+      def parse_package_info(package_info)
+        package_info.lines.each_with_object(Hash.new(0)) { |pkg, a|
           k, v = pkg.split(":", 2)
           next if k.nil? || k.empty?
           a[k.strip] = v&.strip
         }
       end
 
-      def pip_command(*args)
-        Licensed::Shell.execute(virtual_env_pip, "--disable-pip-version-check", "show", *args)
+      # Returns the output from `pip list --format=json`
+      def pip_list_command
+        Licensed::Shell.execute(virtual_env_pip, "--disable-pip-version-check", "list", "--format=json")
+      end
+
+      # Returns the output from `pip show <package> <package> ...`
+      def pip_show_command(packages)
+        Licensed::Shell.execute(virtual_env_pip, "--disable-pip-version-check", "show", *packages)
       end
 
       def virtual_env_pip
@@ -63,7 +80,10 @@ module Licensed
       def virtual_env_dir
         return @virtual_env_dir if defined?(@virtual_env_dir)
         @virtual_env_dir = begin
-          venv_dir = config.dig("python", "virtual_env_dir")
+          python_config = config["python"]
+          return unless python_config.is_a?(Hash)
+
+          venv_dir = python_config["virtual_env_dir"]
           File.expand_path(venv_dir, config.root) if venv_dir
         end
       end

data/lib/licensed/version.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 module Licensed
-  VERSION = "3.4.3".freeze
+  VERSION = "3.6.0".freeze
 
   def self.previous_major_versions
     major_version = Gem::Version.new(Licensed::VERSION).segments.first

data/licensed.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
 
   spec.required_ruby_version = ">= 2.3.0"
 
-  spec.add_dependency "licensee", ">= 9.14.0", "< 10.0.0"
+  spec.add_dependency "licensee", ">= 9.15.2", "< 10.0.0"
   spec.add_dependency "thor", ">= 0.19"
   spec.add_dependency "pathname-common_prefix", "~> 0.0.1"
   spec.add_dependency "tomlrb", ">= 1.2", "< 3.0"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: licensed
 version: !ruby/object:Gem::Version
-  version: 3.4.3
+  version: 3.6.0
 platform: ruby
 authors:
 - GitHub
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-01-31 00:00:00.000000000 Z
+date: 2022-03-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: licensee
@@ -16,7 +16,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 9.14.0
+        version: 9.15.2
     - - "<"
       - !ruby/object:Gem::Version
         version: 10.0.0
@@ -26,7 +26,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 9.14.0
+        version: 9.15.2
     - - "<"
       - !ruby/object:Gem::Version
         version: 10.0.0
@@ -261,10 +261,12 @@ files:
 - docs/configuration/application_source.md
 - docs/configuration/configuration_root.md
 - docs/configuration/configuring_multiple_apps.md
+- docs/configuration/customizing_licensee.md
 - docs/configuration/dependency_source_enumerators.md
 - docs/configuration/ignoring_dependencies.md
 - docs/configuration/metadata_cache.md
 - docs/configuration/reviewing_dependencies.md
+- docs/getting_started.md
 - docs/migrations/v2.md
 - docs/migrations/v3.md
 - docs/packaging.md