licensed 3.5.0 → 3.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d2fafd3b11ba6f63760979021e1628aa1d4ae0e0cd5b0b413b06c6163a1b64fd
-  data.tar.gz: d5fe530e59e0091b44f0ae08403f72707f6dc4db391a6d7e39e4b620c01b1da9
+  metadata.gz: 03de72b78b61eb285dc7f28303375a782ac1061997cd8a1dc7e664874d017ab6
+  data.tar.gz: af3660d69f8a235b7b40b08ede58f0bc8250ead97aa3a19c51cebfb88922c5e9
 SHA512:
-  metadata.gz: a2b691823b7cbc692fb155bc672772fea156932a3d7d38111634927a0babf427343f992eded7d219041575e64e698a9821fdf1e9e12390f8e29a2c95a42e340a
-  data.tar.gz: 61872b213231ead8c7ce5d209af8332964e8562e8809dbbf7c3e5fd95b84cfffcc7b9c4391f0e81d6e094325293bbe61520458d4cc1233072f4b0c41b04cd165
+  metadata.gz: 2e8125e6de6e0892a1afca36dd6cf5ae6091d101b7bbc58a3fd3086c1ac58838323f4979d859ea2de7e3ecc32f4af222f3eab433d6cccd95b73687b1f2626eb3
+  data.tar.gz: 4c18508e167faeb8d81dfb1548e9f5ed71fbc38c18b0a610dbc230911f7940062deadc4bed7a153e39ff8d94642bcab82b61a9ad7948860f508a84dc0b2f6175

data/CHANGELOG.md

@@ -6,6 +6,36 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 3.7.1
+
+### Fixed
+
+- Dependencies' legal notice file matching has been made more strict to reduce false positives on code files containing the word `legal` (https://github.com/github/licensed/pull/510)
+
+## 3.7.0
+
+### Changed
+
+- Pip and pipenv sources will find dependency licenses under `dist-info/license_files` when available (https://github.com/github/licensed/pull/504)
+
+## 3.6.0
+
+2022-03-17
+
+### Added
+
+- Composer dev dependencies can optionally be included in enumerated PHP dependencies (:tada: @digilist https://github.com/github/licensed/pull/486)
+- Getting started usage documentation (https://github.com/github/licensed/pull/483)
+- Initial support for NPM workspaces (https://github.com/github/licensed/pull/485)
+
+### Changed
+
+- Transitive dependencies are now enumerated by the `pip` source (https://github.com/github/licensed/pull/480)
+
+### Fixed
+
+- `licensed cache --force` will now correctly overwrite existing license classifications (https://github.com/github/licensed/pull/473)
+
 ## 3.5.0
 
 2022-02-24
@@ -579,4 +609,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/3.4.4...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/3.7.1...HEAD

data/README.md

@@ -74,6 +74,8 @@ For system wide usage, install licensed to a location on `$PATH`, e.g. `/usr/loc
 
 ## Usage
 
+See [getting started](./docs/getting_started.md) for guidance using Licensed as part of your developer workflow.
+
 ### Available commands
 
 See the [commands documentation](./docs/commands) for documentation on available commands, or run `licensed -h` to see all of the current available commands.
@@ -86,18 +88,6 @@ A configuration file is required for most commands.  See the [configuration file
 
 Licensed can enumerate dependency for many languages, package managers, and frameworks.  See the [sources documentation](./docs/sources) for the list of currently available sources.  Sources can be explicitly enabled and disabled as a [configuration option](./docs/configuration/dependency_source_enumerators.md.md).
 
-### Automation
-
-#### Bundler
-
-The [bundler-licensed plugin](https://github.com/sergey-alekseev/bundler-licensed) runs `licensed cache` automatically when using `bundler`.  See the linked repo for usage and details.
-
-#### GitHub Actions
-
-The [licensed-ci](https://github.com/marketplace/actions/licensed-ci) GitHub Action runs `licensed` as part of an opinionated CI workflow and can be configured to run on any GitHub Action event.  See the linked actions for usage and details.
-
-The [setup-licensed](https://github.com/marketplace/actions/setup-github-licensed) GitHub Action installs `licensed` to the workflow environment.  See the linked actions for usage and details.
-
 ## Development
 
 To get started after checking out the repo, run

data/docs/configuration/customizing_licensee.md

@@ -8,6 +8,6 @@ Licensed uses [Licensee](https://github.com/licensee/licensee) to detect and eva
 licensee:
   # the confidence threshold is an integer between 1 and 100. the value represents
   # the minimum percentage confidence that Licensee must have to report a matched license
-  # https://github.com/licensee/licensee/blob/jonabc-patch-1/docs/customizing.md#adjusting-the-confidence-threshold
+  # https://github.com/licensee/licensee/blob/master/docs/customizing.md#adjusting-the-confidence-threshold
   confidence_threshold: 90 # default value: 98
 ```

data/docs/getting_started.md

@@ -0,0 +1,40 @@
+# Getting Started
+
+Licensed's core workflow is a multi-step process:
+
+1. `licensed cache` ([docs](./commands/cache.md)) is run manually and/or in an automated workflow
+   - Creates or updates files in a git repo containing metadata including licenses and other legal text for each dependency used by a project
+1. `licensed status` ([docs](./commands/status.md)) is run manually and/or in an automated workflow
+   - Validate that every detected dependency has a metadata file written in the repository, and that each dependency's stored metadata passes a number of compliance checks
+1. Any detected errors/warnings are manually [resolved](./commands/status.md#status-errors-and-resolutions)
+1. Repeat the above steps until all dependencies have metadata files stored in the repository and `licensed status` is not reporting any errors.
+
+## Caching dependency metadata
+
+Caching depedency metadata into the repository brings the metadata contents closer to the dependencies where they are used, making status validation faster and possible in offline scenarios.  Keeping metadata alongside your code in git gives teams an easily auditable trail for dependency updates over time, and ties into common review practices to ensure that changes aren't quietly ignored.
+
+Caching metadata should be done whenever project code changes, to ensure that metadata files are in sync with the current state of the project code.
+
+## Checking dependency metadata status
+
+Dependency metadata checks verify that every dependency
+
+- has a metadata file available, or has been explicitly ignored by the project owners or OSS experts
+- is using an approved OSS license, or has been reviewed and signed off by an OSS expert
+- is up to date with the current state of a project
+
+Checking dependencies for compliance violations should be performed whenever code changes in a repository.  Moving compliance checks inline in the development workflow reduces friction later, and can even prevent costly situations later if a non-compliant dependency would need to be removed from a project.
+
+## Automated workflows
+
+Integrating github/licensed into your workflow can be tedious, and luckily there are a few automated tools available to make usage easier.
+
+### Bundler
+
+The [bundler-licensed plugin](https://github.com/sergey-alekseev/bundler-licensed) runs `licensed cache` automatically when using `bundler`.  See the linked repo for usage and details.
+
+### GitHub Actions
+
+The [licensed-ci](https://github.com/marketplace/actions/licensed-ci) GitHub Action runs `licensed` as part of an opinionated CI workflow and can be configured to run on any GitHub Action event to automatically update the cached metadata files and check their status.  See the linked action for usage and details.
+
+The [setup-licensed](https://github.com/marketplace/actions/setup-github-licensed) GitHub Action installs `licensed` to the workflow environment.  See the linked actions for usage and details.

data/docs/sources/composer.md

@@ -12,3 +12,12 @@ The default composer application file location is `<repository root>/composer.ph
 composer:
   application_path: "/path/to/composer"
 ```
+
+### Dev dependencies
+
+By default licensed ignores all dev dependencies. To consider dev dependencies as well, use the `composer.include_dev` configuration setting.
+
+```yml
+composer:
+  include_dev: true
+```

data/docs/sources/npm.md

@@ -2,7 +2,7 @@
 
 The npm source will detect dependencies `package.json` is found at an apps `source_path`.  It uses `npm list` to enumerate dependencies and metadata.
 
-### Including development dependencies
+## Including development dependencies
 
 By default, the npm source will exclude all development dependencies. To include development or test dependencies, set `production_only: false` in the licensed configuration.
 
@@ -10,3 +10,13 @@ By default, the npm source will exclude all development dependencies. To include
 npm:
   production_only: false
 ```
+
+## Using licensed with npm workspaces
+
+Licensed requires npm version 8.5.0 or later to enumerate dependencies inside of npm workspaces.  For the best results, treat each workspace directory as a separate app `source_path`:
+
+```yml
+apps:
+  - source_path: path/to/workspace/a
+  - source_path: path/to/workspace/b
+```

data/docs/sources/pip.md

@@ -2,11 +2,6 @@
 
 The pip source uses `pip` CLI commands to enumerate dependencies and properties. It is expected that `pip` is available in the `virtual_env_dir` specific directory before running `licensed`.
 
-Your repository root should also contain a `requirements.txt` file which contains all the packages and dependences that are needed. You can generate one with `pip` using the command:
-```
-pip freeze > requirements.txt
-```
-
 A `virtualenv` directory is required before running `licensed`. You can setup a `virtualenv` by running the command:
 ```
 virtualenv <your_venv_dir>

data/lib/licensed/commands/cache.rb

@@ -82,15 +82,8 @@ module Licensed
         report["filename"] = filename.to_s
         report["version"] = dependency.version
 
-        if options[:force] || save_dependency_record?(dependency, cached_record)
-          if dependency.record.matches?(cached_record)
-            # use the cached license value if the license text wasn't updated
-            dependency.record["license"] = cached_record["license"]
-          elsif cached_record && app.reviewed?(dependency.record)
-            # if the license text changed and the dependency is set as reviewed
-            # force a re-review of the dependency
-            dependency.record["review_changed_license"] = true
-          end
+        if save_dependency_record?(dependency, cached_record)
+          update_dependency_from_cached_record(app, dependency, cached_record)
 
           dependency.record.save(filename)
           report["cached"] = true
@@ -119,6 +112,7 @@ module Licensed
       # Returns true if dependency's record should be saved
       def save_dependency_record?(dependency, cached_record)
         return true if cached_record.nil?
+        return true if options[:force]
 
         cached_version = cached_record["version"]
         return true if cached_version.nil? || cached_version.empty?
@@ -126,6 +120,23 @@ module Licensed
         false
       end
 
+      # Update dependency metadata from the cached record, to support:
+      # 1. continuity between cache runs to cut down on churn
+      # 2. notifying users when changed content needs to be reviewed
+      def update_dependency_from_cached_record(app, dependency, cached_record)
+        return if cached_record.nil?
+        return if options[:force]
+
+        if dependency.record.matches?(cached_record)
+          # use the cached license value if the license text wasn't updated
+          dependency.record["license"] = cached_record["license"]
+        elsif app.reviewed?(dependency.record)
+          # if the license text changed and the dependency is set as reviewed
+          # force a re-review of the dependency
+          dependency.record["review_changed_license"] = true
+        end
+      end
+
       # Clean up cached files that dont match current dependencies
       #
       # Returns nothing

data/lib/licensed/dependency.rb

@@ -3,7 +3,7 @@ require "licensee"
 
 module Licensed
   class Dependency < Licensee::Projects::FSProject
-    LEGAL_FILES_PATTERN = /(AUTHORS|NOTICE|LEGAL)(?:\..*)?\z/i
+    LEGAL_FILES_PATTERN = /#{File::SEPARATOR}(AUTHORS|NOTICE|LEGAL)(?:\..*)?\z/i
 
     attr_reader :name
     attr_reader :version

data/lib/licensed/sources/composer.rb

@@ -28,7 +28,10 @@ module Licensed
       end
 
       def packages
-        JSON.parse(File.read(composer_lock))["packages"]
+        packages = JSON.parse(File.read(composer_lock))
+        return packages["packages"] unless include_dev?
+
+        packages["packages"] + packages["packages-dev"]
       end
 
       # Returns the output from running `php composer.phar` to get package metadata
@@ -56,6 +59,11 @@ module Licensed
       def composer_lock
         config.pwd.join("composer.lock")
       end
+
+      # Returns whether to include dev packages based on the licensed configuration settings
+      def include_dev?
+        config.dig("composer", "include_dev") == true
+      end
     end
   end
 end

data/lib/licensed/sources/npm.rb

@@ -24,11 +24,13 @@ module Licensed
       end
 
       def enabled?
-        Licensed::Shell.tool_available?("npm") && File.exist?(config.pwd.join("package.json"))
+        Licensed::Shell.tool_available?("npm") && File.exist?(package_json_path)
       end
 
       def enumerate_dependencies
         packages.map do |name, package|
+          next if package["name"] == project_name
+
           errors = package["problems"] unless package["path"]
           Dependency.new(
             name: name,
@@ -159,6 +161,26 @@ module Licensed
       def extract_version(parent, name)
         parent&.dig("_dependencies", name) || peer_dependency(parent, name)
       end
+
+      # Returns the current projects name
+      def project_name
+        return unless package_json
+        package_json["name"]
+      end
+
+      ## Returns the parse package.json for the current project
+      def package_json
+        return unless File.exist?(package_json_path)
+
+        @package_json ||= JSON.parse(File.read(package_json_path))
+      rescue JSON::ParserError => e
+        message = "Licensed was unable to parse package.json. JSON Error: #{e.message}"
+        raise Licensed::Sources::Source::Error, message
+      end
+
+      def package_json_path
+        @package_json_path ||= File.join(config.pwd, "package.json")
+      end
     end
   end
 end

data/lib/licensed/sources/pip.rb

@@ -7,24 +7,20 @@ require "parallel"
 module Licensed
   module Sources
     class Pip < Source
-      VERSION_OPERATORS = %w(< > <= >= == !=).freeze
-      PACKAGE_REGEX = /^([\w\.-]+)(#{VERSION_OPERATORS.join("|")})?/
+      PACKAGE_INFO_SEPARATOR = "\n---\n"
 
       def enabled?
-        return unless virtual_env_pip && Licensed::Shell.tool_available?(virtual_env_pip)
-        File.exist?(config.pwd.join("requirements.txt"))
+        !pip_command.empty? && Licensed::Shell.tool_available?(pip_command.join(""))
       end
 
       def enumerate_dependencies
-        Parallel.map(packages_from_requirements_txt, in_threads: Parallel.processor_count) do |package_name|
-          package = package_info(package_name)
-          location = File.join(package["Location"], package["Name"].gsub("-", "_") +  "-" + package["Version"] + ".dist-info")
+        packages.map do |package|
           Dependency.new(
             name: package["Name"],
             version: package["Version"],
-            path: location,
+            path: package_license_location(package),
             metadata: {
-              "type"        => Pip.type,
+              "type"        => self.class.type,
               "summary"     => package["Summary"],
               "homepage"    => package["Home-page"]
             }
@@ -32,32 +28,63 @@ module Licensed
         end
       end
 
+      protected
+
+      # Returns the command to run pip
+      def pip_command
+        return [] unless virtual_env_dir
+        Array(File.join(virtual_env_dir, "bin", "pip"))
+      end
+
       private
 
-      def packages_from_requirements_txt
-        File.read(config.pwd.join("requirements.txt"))
-            .lines
-            .reject { |line| line.include?("://") }
-            .map { |line| line.strip.match(PACKAGE_REGEX) { |match| match.captures.first } }
-            .compact
+      # Returns the location of license files in the package, checking for the inclusion of a new `license_files`
+      # folder per https://peps.python.org/pep-0639/
+      def package_license_location(package)
+        dist_info = File.join(package["Location"], package["Name"].gsub("-", "_") +  "-" + package["Version"] + ".dist-info")
+        license_files = File.join(dist_info, "license_files")
+        return File.exist?(license_files) ? license_files : dist_info
+      end
+
+      # Returns parsed information for all packages used by the project,
+      # using `pip list` to determine what packages are used and `pip show`
+      # to gather package information
+      def packages
+        all_packages = pip_show_command(package_names)
+        all_packages.split(PACKAGE_INFO_SEPARATOR).reduce([]) do |accum, val|
+          accum << parse_package_info(val)
+        end
+      end
+
+      # Returns the names of all of the packages used by the current project,
+      # as returned from `pip list`
+      def package_names
+        @package_names ||= begin
+          JSON.parse(pip_list_command).map { |package| package["name"] }
+        rescue JSON::ParserError => e
+          message = "Licensed was unable to parse the output from 'npm list'. JSON Error: #{e.message}"
+          raise Licensed::Sources::Source::Error, message
+        end
       end
 
-      def package_info(package_name)
-        p_info = pip_command(package_name).lines
-        p_info.each_with_object(Hash.new(0)) { |pkg, a|
+      # Returns a hash filled with package info parsed from the email-header formatted output
+      # returned by `pip show`
+      def parse_package_info(package_info)
+        package_info.lines.each_with_object(Hash.new(0)) { |pkg, a|
           k, v = pkg.split(":", 2)
           next if k.nil? || k.empty?
           a[k.strip] = v&.strip
         }
       end
 
-      def pip_command(*args)
-        Licensed::Shell.execute(virtual_env_pip, "--disable-pip-version-check", "show", *args)
+      # Returns the output from `pip list --format=json`
+      def pip_list_command
+        Licensed::Shell.execute(*pip_command, "--disable-pip-version-check", "list", "--format=json")
       end
 
-      def virtual_env_pip
-        return unless virtual_env_dir
-        File.join(virtual_env_dir, "bin", "pip")
+      # Returns the output from `pip show <package> <package> ...`
+      def pip_show_command(packages)
+        Licensed::Shell.execute(*pip_command, "--disable-pip-version-check", "show", *packages)
       end
 
       def virtual_env_dir

data/lib/licensed/sources/pipenv.rb

@@ -4,44 +4,16 @@ require "parallel"
 
 module Licensed
   module Sources
-    class Pipenv < Source
+    class Pipenv < Pip
       def enabled?
         Licensed::Shell.tool_available?("pipenv") && File.exist?(config.pwd.join("Pipfile.lock"))
       end
 
-      def enumerate_dependencies
-        Parallel.map(pakages_from_pipfile_lock, in_threads: Parallel.processor_count) do |package_name|
-          package = package_info(package_name)
-          location = File.join(package["Location"], package["Name"].gsub("-", "_") +  "-" + package["Version"] + ".dist-info")
-          Dependency.new(
-            name: package["Name"],
-            version: package["Version"],
-            path: location,
-            metadata: {
-              "type"        => Pipenv.type,
-              "summary"     => package["Summary"],
-              "homepage"    => package["Home-page"]
-            }
-          )
-        end
-      end
-
-      private
-
-      def pakages_from_pipfile_lock
-        Licensed::Shell.execute("pipenv", "run", "pip", "list")
-            .lines
-            .drop(2)  # Header
-            .map { |line| line.strip.split.first.strip }
-      end
+      protected
 
-      def package_info(package_name)
-        p_info = Licensed::Shell.execute("pipenv", "run", "pip", "--disable-pip-version-check", "show", package_name).lines
-        p_info.each_with_object(Hash.new(0)) { |pkg, a|
-          k, v = pkg.split(":", 2)
-          next if k.nil? || k.empty?
-          a[k.strip] = v&.strip
-        }
+      # Returns the command to run pip
+      def pip_command
+        %w(pipenv run pip)
       end
     end
   end

data/lib/licensed/sources/source.rb

@@ -14,8 +14,13 @@ module Licensed
       class << self
         attr_reader :sources
         def inherited(klass)
-          # add child source classes are defined,
-          # add them to the known sources list
+          # register the inherited class as a source on the Licensed::Sources::Source class
+          Licensed::Sources::Source.register_source(klass)
+        end
+
+        def register_source(klass)
+          # add the source class to the known sources list
+          return unless klass < Licensed::Sources::Source
           (@sources ||= []) << klass
         end
 

data/lib/licensed/version.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 module Licensed
-  VERSION = "3.5.0".freeze
+  VERSION = "3.7.1".freeze
 
   def self.previous_major_versions
     major_version = Gem::Version.new(Licensed::VERSION).segments.first

data/licensed.gemspec

@@ -35,7 +35,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "rake", ">= 12.3.3"
   spec.add_development_dependency "minitest", "~> 5.8"
   spec.add_development_dependency "mocha", "~> 1.0"
-  spec.add_development_dependency "rubocop", "~> 0.49", "< 1.20"
+  spec.add_development_dependency "rubocop", "~> 1.26", "< 1.27"
   spec.add_development_dependency "rubocop-github", "~> 0.6"
-  spec.add_development_dependency "byebug", "~> 11.0.1"
+  spec.add_development_dependency "byebug", "~> 11.1.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: licensed
 version: !ruby/object:Gem::Version
-  version: 3.5.0
+  version: 3.7.1
 platform: ruby
 authors:
 - GitHub
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2022-02-24 00:00:00.000000000 Z
+date: 2022-05-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: licensee
@@ -188,20 +188,20 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.49'
+        version: '1.26'
     - - "<"
       - !ruby/object:Gem::Version
-        version: '1.20'
+        version: '1.27'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.49'
+        version: '1.26'
     - - "<"
       - !ruby/object:Gem::Version
-        version: '1.20'
+        version: '1.27'
 - !ruby/object:Gem::Dependency
   name: rubocop-github
   requirement: !ruby/object:Gem::Requirement
@@ -222,14 +222,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 11.0.1
+        version: 11.1.3
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 11.0.1
+        version: 11.1.3
 description: Licensed automates extracting and validating the licenses of dependencies.
 email:
 - opensource+licensed@github.com
@@ -266,6 +266,7 @@ files:
 - docs/configuration/ignoring_dependencies.md
 - docs/configuration/metadata_cache.md
 - docs/configuration/reviewing_dependencies.md
+- docs/getting_started.md
 - docs/migrations/v2.md
 - docs/migrations/v3.md
 - docs/packaging.md