licensed 3.1.0 → 3.2.3

data/.ruby-version

@@ -1 +1 @@
-2.4.0
+2.7.4

data/CHANGELOG.md

@@ -6,6 +6,56 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## 3.2.3
+
+2021-09-14
+
+### Fixed
+
+- Bundler source will no longer infinitely recurse when enumerating specifications (https://github.com/github/licensed/pull/402)
+- Using the `--sources` command line option will no longer delete skipped sources' cached files (https://github.com/github/licensed/pull/401)
+
+## 3.2.2
+
+2021-09-09
+
+### Fixed
+
+- Bundler source works properly again when used outside of `bundle exec` (https://github.com/github/licensed/pull/397)
+
+## 3.2.1
+
+2021-09-06
+
+### Changed
+
+- Updated multiple dependency versions (:tada: @mmorel-35 https://github.com/github/licensed/pull/385, https://github.com/github/licensed/pull/389)
+- Go homepage links use pkg.go.dev instead of godoc.org (:tada: @mmorel-35 https://github.com/github/licensed/commit/73cfbbe954a3e8c8cbaf8b68253053b157e01b79)
+- Local development ruby version changed to 2.7.4 (https://github.com/github/licensed/pull/393)
+
+### Fixed
+
+- Bundler source correctly finds platform specific dependencies (https://github.com/github/licensed/pull/392)
+
+## 3.2.0
+
+2021-08-19
+
+### Added
+
+- Application names can be dynamically generated based on the path to the application source  (https://github.com/github/licensed/pull/375)
+
+### Changed
+
+- Updated command documentation (https://github.com/github/licensed/pull/378, https://github.com/github/licensed/pull/380/files)
+- Updated configuration documentation (https://github.com/github/licensed/pull/375)
+- Cache and status commands give additional diagnostic output when using JSON and YAML formatters (https://github.com/github/licensed/pull/378)
+- Status command will give users a link to documentation when compliance checks fail (https://github.com/github/licensed/pull/381)
+
+### Fixed
+
+- The bundler source correctly checks that the path bundler specifies a gem is loaded from is a file (https://github.com/github/licensed/pull/379)
+
 ## 3.1.0
 
 2021-06-16
@@ -447,4 +497,4 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 Initial release :tada:
 
-[Unreleased]: https://github.com/github/licensed/compare/3.1.0...HEAD
+[Unreleased]: https://github.com/github/licensed/compare/3.2.3...HEAD

data/README.md

@@ -37,13 +37,13 @@ See the [v2 migration documentation](./docs/migrations/v2.md) for more info on m
 
 Licensed uses the `libgit2` bindings for Ruby provided by `rugged`. `rugged` requires `cmake` and `pkg-config` which you may need to install before you can install Licensed.
 
-   >  Ubuntu
-
-    sudo apt-get install cmake pkg-config
-
-   >  OS X
+```bash
+# Ubuntu
+sudo apt-get install cmake pkg-config
 
-    brew install cmake pkg-config
+# macOS
+brew install cmake pkg-config
+```
 
 ### With a Gemfile
 
@@ -56,7 +56,7 @@ gem 'licensed', :group => 'development'
 And then execute:
 
 ```bash
-$ bundle
+$> bundle
 ```
 
 ### As an executable
@@ -64,24 +64,27 @@ $ bundle
 Download a package from GitHub and extract the executable.  Executable packages are available for each release starting with version 1.2.0.
 
 ```bash
-$ curl -sSL https://github.com/github/licensed/releases/download/<version>/licensed-<version>-<os>-x64.tar.gz > licensed.tar.gz
-$ tar -xzf licensed.tar.gz
-$ rm -f licensed.tar.gz
-$ ./licensed list
+$> curl -sSL https://github.com/github/licensed/releases/download/<version>/licensed-<version>-<os>-x64.tar.gz > licensed.tar.gz
+$> tar -xzf licensed.tar.gz
+$> rm -f licensed.tar.gz
+$> ./licensed list
 ```
 
 For system wide usage, install licensed to a location on `$PATH`, e.g. `/usr/local/bin`.
 
 ## Usage
 
-- `licensed list`: Output enumerated dependencies only.
-- `licensed cache`: Cache licenses and metadata.
-- `licensed status`: Check status of dependencies' cached licenses.
-- `licensed notices`: Write a `NOTICE` file for each application configuration.
-- `licensed version`: Show current installed version of Licensed. Aliases: `-v|--version`
-- `licensed env`: Output environment information from the licensed configuration.
+### Available commands
+
+See the [commands documentation](./docs/commands) for documentation on available commands, or run `licensed -h` to see all of the current available commands.
 
-See the [commands documentation](./docs/commands.md) for additional documentation, or run `licensed -h` to see all of the current available commands.
+### Configuration options
+
+A configuration file is required for most commands.  See the [configuration file documentation](./docs/configuration.md) for more details on the configuration format and available configuration options.
+
+### Available dependency sources
+
+Licensed can enumerate dependency for many languages, package managers, and frameworks.  See the [sources documentation](./docs/sources) for the list of currently available sources.  Sources can be explicitly enabled and disabled as a [configuration option](./docs/configuration/dependency_source_enumerators.md.md).
 
 ### Automation
 
@@ -95,80 +98,22 @@ The [licensed-ci](https://github.com/marketplace/actions/licensed-ci) GitHub Act
 
 The [setup-licensed](https://github.com/marketplace/actions/setup-github-licensed) GitHub Action installs `licensed` to the workflow environment.  See the linked actions for usage and details.
 
-### Configuration
-
-All commands, except `version`, accept a `-c|--config` option to specify a path to a configuration file or directory.
-
-If a directory is specified, `licensed` will look in that directory for a file named (in order of preference):
-1. `.licensed.yml`
-2. `.licensed.yaml`
-3. `.licensed.json`
-
-If the option is not specified, the value will be set to the current directory.
-
-See the [configuration file documentation](./docs/configuration.md) for more details on the configuration format.
-
-### Sources
-
-Dependencies will be automatically detected for all of the following sources by default.
-1. [Bower](./docs/sources/bower.md)
-1. [Bundler](./docs/sources/bundler.md)
-1. [Cabal](./docs/sources/cabal.md)
-1. [Composer](./docs/sources/composer.md)
-1. [Git Submodules (git_submodule)](./docs/sources/git_submodule.md)
-1. [Go](./docs/sources/go.md)
-1. [Go Dep (dep)](./docs/sources/dep.md)
-1. [Gradle](./docs/sources/gradle.md)
-1. [Manifest lists (manifests)](./docs/sources/manifests.md)
-1. [Mix](./docs/sources/mix.md)
-1. [npm](./docs/sources/npm.md)
-1. [NuGet](./docs/sources/nuget.md)
-1. [Pip](./docs/sources/pip.md)
-1. [Pipenv](./docs/sources/pipenv.md)
-1. [Swift](./docs/sources/swift.md)
-1. [Yarn](./docs/sources/yarn.md)
-
-You can disable any of them in the configuration file:
-
-```yml
-sources:
-  bundler: false
-  npm: false
-  bower: false
-  cabal: false
-```
-
 ## Development
 
 To get started after checking out the repo, run
+
 1. `script/bootstrap` to install dependencies
 2. `script/setup` to setup test fixtures.
   - `script/setup -f` will force a clean test fixture environment
-3. `script/cibuild` to run the tests.
+3. `script/cibuild` to run the tests
 
 You can also run `script/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-#### Adding sources
-
-When adding new dependency sources, ensure that `script/bootstrap` scripting and tests are only run if the required tooling is available on the development machine.
-
-* See `script/bootstrap` for examples of gating scripting based on whether tooling executables are found.
-* Use `Licensed::Shell.tool_available?` when writing test files to gate running a test suite when tooling executables aren't available.
-```ruby
-if Licensed::Shell.tool_available?('bundle')
-  describe Licensed::Source::Bundler do
-    ...
-  end
-end
-```
-
-See the [documentation on adding new sources](./docs/adding_a_new_source.md) for more information.
-
-#### Adding Commands
+### Adding a new source
 
-See the [documentation on commands](./docs/commands.md) for information about adding a new CLI command.
+See the [documentation on adding new sources](./docs/adding_a_new_source.md) for detailed information on what's required to add a new dependency source enumerator.
 
 ## Contributing
 

data/docker/Dockerfile.build-linux

@@ -1,4 +1,4 @@
-FROM ruby:2.4-slim-stretch
+FROM ruby:2.6.8-slim-stretch
 
 RUN apt-get update \
     && apt-get install -y --no-install-recommends cmake make gcc pkg-config squashfs-tools git curl bison rsync \

data/docs/adding_a_new_source.md

@@ -4,13 +4,15 @@
 
 Dependency enumerators inherit and override the [`Licensed::Sources::Source`](../lib/licensed/sources/source.rb) class.
 
-#### Required method overrides
+### Required method overrides
+
 1. `Licensed::Sources::Source#enabled?`
    - Returns whether dependencies can be enumerated in the current environment.
 2. `Licensed::Sources::Source#enumerate_dependencies`
    - Returns an enumeration of `Licensed::Dependency` objects found which map to the dependencies of the current project.
 
-#### Optional method overrides
+### Optional method overrides
+
 1. `Licensed::Sources::Source.type`
    - Returns the name of the current dependency enumerator as it is found in a licensed configuration file.
 
@@ -22,12 +24,13 @@ whether `Licensed::Source::Sources#enumerate_dependencies` should be called on t
 Determining whether dependencies should be enumerated depends on whether all the tools or files needed to find dependencies are present.
 For example, to enumerate `npm` dependencies the `npm` CLI tool must be found with `Licensed::Shell.tool_available?` and a `package.json` file needs to exist in the licensed app's configured [`source_path`](./configuration.md#configuration-paths).
 
-#### Gating functionality when required tools are not available.
+### Gating functionality when required tools are not available.
 
 When adding new dependency sources, ensure that `script/bootstrap` scripting and tests are only run if the required tooling is available on the development machine.
 
-* See `script/bootstrap` for examples of gating scripting based on whether tooling executables are found.
-* Use `Licensed::Shell.tool_available?` when writing test files to gate running a test suite when tooling executables aren't available.
+- See `script/bootstrap` for examples of gating scripting based on whether tooling executables are found.
+- Use `Licensed::Shell.tool_available?` when writing test files to gate running a test suite when tooling executables aren't available.
+
 ```ruby
 if Licensed::Shell.tool_available?('bundle')
   describe Licensed::Source::Bundler do
@@ -47,11 +50,11 @@ Relying on external tools always has a risk that the tool could change.  It's ge
 or other implementation details as these could change over time.  CLI tools that provides the necessary information are generally preferred
 as they will more likely have requirements for backwards compatibility.
 
-#### Creating dependency objects
+### Creating dependency objects
 
 Creating a new `Licensed::Dependency` object requires name, version, and path arguments.  Dependency objects optionally accept a path to use as search root when finding licenses along with any other metadata that is useful to identify the dependency.
 
-##### `Licensed::Dependency` arguments
+#### `Licensed::Dependency` arguments
 
 1. name (required)
    - The name of the dependency. Together with the version, this should uniquely identify the dependency.
@@ -71,7 +74,7 @@ Creating a new `Licensed::Dependency` object requires name, version, and path ar
 6. errors (optional)
   - Any errors found when loading dependency information.
 
-##### Creating specialized Dependency objects
+#### Creating specialized Dependency objects
 
 `Licensed::Dependency` objects inherit from `Licensee::Projects::FsProject` and can override or extend the default `Licensee` behavior to find files for a dependency.
 

data/docs/commands/README.md

@@ -0,0 +1,59 @@
+# Commands
+
+Run `licensed -h` to see help content for running licensed commands.
+
+- [cache](cache.md)
+- [env](env.md)
+- [list](list.md)
+- [migrate](migrate.md)
+- [notices](notices.md)
+- [status](status.md)
+- [version](verison.md)
+
+Most commands accept a `-c`/`--config` option to specify a path to a configuration file or directory. If a directory is specified, `licensed` will look in that directory for a file named (in order of preference):
+
+1. `.licensed.yml`
+2. `.licensed.yaml`
+3. `.licensed.json`
+
+If the option is not specified, the value will be set to the current directory.
+
+## Adding a new command
+
+### Implement new `Command` class
+
+Licensed commands inherit and override the [`Licensed::Sources::Command`](../lib/licensed/commands/command.rb) class.
+
+### Required method overrides
+
+1. `Licensed::Commands::Command#evaluate_dependency`
+   - Runs a command execution on an application dependency.
+
+The `evaluate_dependency` method should contain the specific command logic.  This method has access to the application configuration, dependency source enumerator and dependency currently being evaluated as well as a reporting hash to contain information about the command execution.
+
+### Optional method overrides
+
+The following methods break apart the different levels of command execution.  Each method wraps lower levels of command execution in a corresponding reporter method.
+
+1. `Licensed::Commands::Command#run`
+   - Runs `run_app` for each application configuration found.  Wraps the execution of all applications in `Reporter#report_run`.
+2. `Licensed::Commands::Command#run_app`
+   - Runs `run_source` for each dependency source enumerator enabled for the application configuration.  Wraps the execution of all sources in `Reporter#report_app`.
+3. `Licensed::Commands::Command#run_source`
+   - Runs `run_dependency` for each dependency found in the source.  Wraps the execution of all dependencies in `Reporter#report_source`.
+4. `Licensed::Commands::Command#run_dependency`
+   - Runs `evaluate_dependency` for the dependency.  Wraps the execution of all dependencies in `Reporter#report_dependency`.
+
+As an example, `Licensed::Commands::Command#run_app` calls `Reporter#report_app` to wrap every call to `Licensed::Commands::Command#run_source`.
+
+### Specifying additional report data
+
+The `run` methods can be overridden and pass a block to `super` to provide additional reporting data or functionality.
+
+```ruby
+def run_app(app)
+  super do |report|
+    report["my_app_data"] = true
+  end
+end
+```

data/docs/commands/cache.md

@@ -0,0 +1,35 @@
+# `licensed cache`
+
+The cache command finds all dependencies and ensures that each dependency has an up-to-date cached record.
+
+Dependency records will be saved if:
+
+1. The `force` option is set
+2. No cached record is found
+3. The cached record's version is different than the current dependency's version
+   - If the cached record's license text contents matches the current dependency's license text then the `license` metadata from the cached record is retained for the new saved record.
+
+After the cache command is run, any cached records that don't match up to a current application dependency will be deleted.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--force`: if set, forces all dependency metadata files to be recached
+   - default value: not set
+
+## Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- cached: true when the dependency's cached metadata file was updated, false otherwise
+- version: the version of the enumerated dependency
+- license: the dependency's SPDX license identifier
+- filename: the full path on disk to the dependency's cached metadata file, if available
+- warnings: any warning messages encountered while enumerating and caching dependency metadata, if available

data/docs/commands/env.md

@@ -0,0 +1,10 @@
+# `licensed env`
+
+Prints the runtime environment used by licensed after loading a configuration file.  This can be different from the configuration file inputs, for example all paths will be given as absolute file paths and glob paths may be expanded.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--format`/`-f`: the output format
+   - default value: `yaml`

data/docs/commands/list.md

@@ -0,0 +1,23 @@
+# `licensed list`
+
+The list command finds and prints the dependencies for all sources in all configured applications.  No additional actions are taken on dependencies.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--licenses`/`-l`: if set, includes each dependency's detected license in the output
+   - default value: not set
+
+### Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- version: the version of the enumerated dependency
+- license: (optional) the dependency's SPDX license identifier

data/docs/commands/migrate.md

@@ -0,0 +1,10 @@
+# `licensed migrate`
+
+Migrates the licensed configuration and cached metadata files from a previous version to the most recent version.  This is not required for all major version updates.  See [migrations documentation](../migrations) for details on the migrations needed for each major version.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--from`/`-f`: the licensed version to migrate from
+   - required

data/docs/commands/notices.md

@@ -0,0 +1,12 @@
+# `licensed notices`
+
+Outputs license and notice text for all dependencies in each app into a `NOTICE` file in the app's `cache_path`.  If an app uses a shared cache path, the file name will contain the app name as well, e.g. `NOTICE.my_app`.
+
+`NOTICE` file contents are retrieved from cached records, with the assumption that cached records have already been reviewed in a compliance workflow.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources

data/docs/commands/status.md

@@ -0,0 +1,74 @@
+# `licensed status`
+
+The status command finds all dependencies and checks whether each dependency has a valid cached record.
+
+A dependency will fail the status checks if:
+
+1. No cached record is found
+2. The cached record's version is different than the current dependency's version
+3. The cached record's `licenses` data is empty
+4. The cached record's `license` metadata doesn't match an `allowed` license from the dependency's application configuration.
+   - If `license: other` is specified and all of the `licenses` entries match an `allowed` license a failure will not be logged
+5. The cached record is flagged for re-review.
+   - This occurs when the record's license text has changed since the record was reviewed.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--force`: if set, forces all dependency metadata files to be recached
+   - default value: not set
+
+## Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- allowed: true if the dependency has passed all checks, false otherwise
+- version: the version of the enumerated dependency
+- license: the dependency's SPDX license identifier
+- filename: the full path on disk to the dependency's cached metadata file, if available
+- errors: any error messages from failed status checks, if available
+
+## Status errors and resolutions
+
+### cached dependency record not found
+
+**Cause:** A dependency was found while running `licensed status` that does not have a corresponding cached metadata file
+**Resolution:** Run `licensed cache` to update the metadata cache and create the missing metadata file
+
+### cached dependency record out of date
+
+**Cause:** A dependency was found while running `licensed status` with a different version than is contained in the dependency's cached metadata file
+**Resolution:** Run `licensed cache` to update the out-of-date metadata files
+
+### missing license text
+
+**Cause:** A license determination was made, e.g. from package metadata, but no license text was found.
+**Resolution:** Manually verify whether the dependency includes a file containing license text.  If the dependency code that was downloaded locally does not contain the license text, please check the dependency source at the version listed in the dependency's cached metadata file to see if there is license text that can be used.
+
+If the dependency does not include license text but does specify that it uses a specific license, please copy the standard license text from a [well known source](https://opensource.org/licenses).
+
+### license text has changed and needs re-review. if the new text is ok, remove the `review_changed_license` flag from the cached record
+
+**Cause:** A dependency that is set as [reviewed] in the licensed configuration file has substantially changed and should be re-reviewed.
+**Resolution:** Review the changes to the license text and classification, along with other metadata contained in the cached file for the dependency.  If the dependency is still allowable for use in your project, remove the `review_changed_license` key from the cached record file.
+
+### license needs review
+
+**Cause:** A dependency is using a license that is not in the configured [allowed list of licenses][allowed], and the dependency has not been marked [ignored] or [reviewed].
+**Resolution:** Review the dependency's usage and specified license with someone familiar with OSS licensing and compliance rules to determine whether the dependency is allowable.  Some common resolutions:
+
+1. The dependency's specified license text differed enough from the standard license text that it was not recognized and classified as `other`.  If, with human review, the license text is recognizable then update the `license: other` value in the cached metadata file to the correct license.
+   - An updated classification will persist through version upgrades until the detected license contents have changed.  The determination is made by [licensee/licensee](https://github.com/licensee/licensee), the library which this tool uses to detect and classify license contents.
+1. The dependency might need to be marked as [ignored] or [reviewed] if either of those scenarios are applicable.
+1. If the used license should be allowable without review (if your entity has a legal team, they may want to review this assessment), ensure the license SPDX is set as [allowed] in the licensed configuration file.
+
+[allowed]: ../configuration/allowed_licenses.md
+[ignored]: ../configuration/ignoring_dependencies.md
+[reviewed]: ../configuration/reviewing_dependencies.md