licensed 2.15.2 → 3.2.0

data/docs/commands/README.md

@@ -0,0 +1,59 @@
+# Commands
+
+Run `licensed -h` to see help content for running licensed commands.
+
+- [cache](cache.md)
+- [env](env.md)
+- [list](list.md)
+- [migrate](migrate.md)
+- [notices](notices.md)
+- [status](status.md)
+- [version](verison.md)
+
+Most commands accept a `-c`/`--config` option to specify a path to a configuration file or directory. If a directory is specified, `licensed` will look in that directory for a file named (in order of preference):
+
+1. `.licensed.yml`
+2. `.licensed.yaml`
+3. `.licensed.json`
+
+If the option is not specified, the value will be set to the current directory.
+
+## Adding a new command
+
+### Implement new `Command` class
+
+Licensed commands inherit and override the [`Licensed::Sources::Command`](../lib/licensed/commands/command.rb) class.
+
+### Required method overrides
+
+1. `Licensed::Commands::Command#evaluate_dependency`
+   - Runs a command execution on an application dependency.
+
+The `evaluate_dependency` method should contain the specific command logic.  This method has access to the application configuration, dependency source enumerator and dependency currently being evaluated as well as a reporting hash to contain information about the command execution.
+
+### Optional method overrides
+
+The following methods break apart the different levels of command execution.  Each method wraps lower levels of command execution in a corresponding reporter method.
+
+1. `Licensed::Commands::Command#run`
+   - Runs `run_app` for each application configuration found.  Wraps the execution of all applications in `Reporter#report_run`.
+2. `Licensed::Commands::Command#run_app`
+   - Runs `run_source` for each dependency source enumerator enabled for the application configuration.  Wraps the execution of all sources in `Reporter#report_app`.
+3. `Licensed::Commands::Command#run_source`
+   - Runs `run_dependency` for each dependency found in the source.  Wraps the execution of all dependencies in `Reporter#report_source`.
+4. `Licensed::Commands::Command#run_dependency`
+   - Runs `evaluate_dependency` for the dependency.  Wraps the execution of all dependencies in `Reporter#report_dependency`.
+
+As an example, `Licensed::Commands::Command#run_app` calls `Reporter#report_app` to wrap every call to `Licensed::Commands::Command#run_source`.
+
+### Specifying additional report data
+
+The `run` methods can be overridden and pass a block to `super` to provide additional reporting data or functionality.
+
+```ruby
+def run_app(app)
+  super do |report|
+    report["my_app_data"] = true
+  end
+end
+```

data/docs/commands/cache.md

@@ -0,0 +1,35 @@
+# `licensed cache`
+
+The cache command finds all dependencies and ensures that each dependency has an up-to-date cached record.
+
+Dependency records will be saved if:
+
+1. The `force` option is set
+2. No cached record is found
+3. The cached record's version is different than the current dependency's version
+   - If the cached record's license text contents matches the current dependency's license text then the `license` metadata from the cached record is retained for the new saved record.
+
+After the cache command is run, any cached records that don't match up to a current application dependency will be deleted.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--force`: if set, forces all dependency metadata files to be recached
+   - default value: not set
+
+## Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- cached: true when the dependency's cached metadata file was updated, false otherwise
+- version: the version of the enumerated dependency
+- license: the dependency's SPDX license identifier
+- filename: the full path on disk to the dependency's cached metadata file, if available
+- warnings: any warning messages encountered while enumerating and caching dependency metadata, if available

data/docs/commands/env.md

@@ -0,0 +1,10 @@
+# `licensed env`
+
+Prints the runtime environment used by licensed after loading a configuration file.  This can be different from the configuration file inputs, for example all paths will be given as absolute file paths and glob paths may be expanded.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--format`/`-f`: the output format
+   - default value: `yaml`

data/docs/commands/list.md

@@ -0,0 +1,23 @@
+# `licensed list`
+
+The list command finds and prints the dependencies for all sources in all configured applications.  No additional actions are taken on dependencies.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--licenses`/`-l`: if set, includes each dependency's detected license in the output
+   - default value: not set
+
+### Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- version: the version of the enumerated dependency
+- license: (optional) the dependency's SPDX license identifier

data/docs/commands/migrate.md

@@ -0,0 +1,10 @@
+# `licensed migrate`
+
+Migrates the licensed configuration and cached metadata files from a previous version to the most recent version.  This is not required for all major version updates.  See [migrations documentation](../migrations) for details on the migrations needed for each major version.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--from`/`-f`: the licensed version to migrate from
+   - required

data/docs/commands/notices.md

@@ -0,0 +1,12 @@
+# `licensed notices`
+
+Outputs license and notice text for all dependencies in each app into a `NOTICE` file in the app's `cache_path`.  If an app uses a shared cache path, the file name will contain the app name as well, e.g. `NOTICE.my_app`.
+
+`NOTICE` file contents are retrieved from cached records, with the assumption that cached records have already been reviewed in a compliance workflow.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources

data/docs/commands/status.md

@@ -0,0 +1,73 @@
+# `licensed status`
+
+The status command finds all dependencies and checks whether each dependency has a valid cached record.
+
+A dependency will fail the status checks if:
+
+1. No cached record is found
+2. The cached record's version is different than the current dependency's version
+3. The cached record's `licenses` data is empty
+4. The cached record's `license` metadata doesn't match an `allowed` license from the dependency's application configuration.
+   - If `license: other` is specified and all of the `licenses` entries match an `allowed` license a failure will not be logged
+5. The cached record is flagged for re-review.
+   - This occurs when the record's license text has changed since the record was reviewed.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--force`: if set, forces all dependency metadata files to be recached
+   - default value: not set
+
+## Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- allowed: true if the dependency has passed all checks, false otherwise
+- version: the version of the enumerated dependency
+- license: the dependency's SPDX license identifier
+- filename: the full path on disk to the dependency's cached metadata file, if available
+- errors: any error messages from failed status checks, if available
+
+## Status errors and resolutions
+
+### cached dependency record not found
+
+**Cause:** A dependency was found while running `licensed status` that does not have a corresponding cached metadata file
+**Resolution:** Run `licensed cache` to update the metadata cache and create the missing metadata file
+
+### cached dependency record out of date
+
+**Cause:** A dependency was found while running `licensed status` with a different version than is contained in the dependency's cached metadata file
+**Resolution:** Run `licensed cache` to update the out-of-date metadata files
+
+### missing license text
+
+**Cause:** A license determination was made, e.g. from package metadata, but no license text was found.
+**Resolution:** Manually verify whether the dependency includes a file containing license text.  If the dependency code that was downloaded locally does not contain the license text, please check the dependency source at the version listed in the dependency's cached metadata file to see if there is license text that can be used.
+
+If the dependency does not include license text but does specify that it uses a specific license, please copy the standard license text from a [well known source](https://opensource.org/licenses).
+
+### license text has changed and needs re-review. if the new text is ok, remove the `review_changed_license` flag from the cached record
+
+**Cause:** A dependency that is set as [reviewed] in the licensed configuration file has substantially changed and should be re-reviewed.
+**Resolution:** Review the changes to the license text and classification, along with other metadata contained in the cached file for the dependency.  If the dependency is still allowable for use in your project, remove the `review_changed_license` key from the cached record file.
+
+### license needs review
+
+**Cause:** A dependency is using a license that is not in the configured [allowed list of licenses][allowed], and the dependency has not been marked [ignored] or [reviewed].
+**Resolution:** Review the dependency's usage and specified license with someone familiar with OSS licensing and compliance rules to determine whether the dependency is allowable.  Some common resolutions:
+
+1. The dependency's specified license text differed enough from the standard license text that it was not recognized and classified as `other`.  If, with human review, the license text is recognizable then update the `license: other` value in the cached metadata file to the correct license.
+1. The dependency might need to be marked as [ignored] or [reviewed] if either of those scenarios are applicable.
+1. If the used license should be allowable without review (if your entity has a legal team, they may want to review this assessment), ensure the license SPDX is set as [allowed] in the licensed configuration file.
+
+[allowed]: ../configuration/allowed_licenses.md
+[ignored]: ../configuration/ignoring_dependencies.md
+[reviewed]: ../configuration/reviewing_dependencies.md

data/docs/commands/version.md

@@ -0,0 +1,3 @@
+# `licensed version`
+
+Displays the current licensed version.  Can also be run with `licensed -v` and `licensed --version`.

data/docs/configuration.md

@@ -2,75 +2,12 @@
 
 A configuration file specifies the details of enumerating and operating on license metadata for apps.
 
-Configuration can be specified in either YML or JSON formats.  Examples below are given in YML.
+Configuration can be specified in either YML or JSON formats, with examples given in YML.  The example
+below describes common configuration values and their purposes.  See [configuration options documentation](./configuration)
+for in depth information.  
 
-## Configuration Paths
-
-`licensed` requires a path to enumerate dependencies at (`source_path`) and a path to store cached metadata (`cache_path`).
-
-To determine these paths across multiple environments where absolute paths will differ, a known root path is needed to evaluate relative paths against.
-In using a root, relative source and cache paths can be specified in the configuration file.
-
-When using a configuration file, the root property can be set as either a path that can be expanded from the configuration file directory using `File.expand_path`, or the value `true` to use the configuration file directory as the root.
-
-When creating a `Licensed::Dependency` manually with a `root` property, the property must be an absolute path - no path expansion will occur.
-
-If a root path is not specified, it will default to using the following, in order of precedence
-1. the root of the local git repository, if run inside a git repository
-2. the current directory
-
-### Source path glob patterns
-
-The `source_path` property can use a glob path to share configuration properties across multiple application entrypoints.
-
-For example, there is a common pattern in Go projects to include multiple executable entrypoints under folders in `cmd`.  Using a glob pattern allows users to avoid manually configuring and maintaining multiple licensed application `source_path`s.  Using a glob pattern will also ensure that any new entrypoints matching the pattern are automatically picked up by licensed commands as they are added.
-
-```yml
-sources:
-  go: true
-
-# treat all directories under `cmd` as separate apps
-source_path: cmd/*
-```
-
-Glob patterns are syntactic sugar for, and provide the same functionality as, manually specifying multiple `source_path` values. See the instructions on [specifying multiple apps](./#specifying-multiple-apps) below for additional considerations when using multiple apps.
-
-## Restricting sources
-
-The `sources` configuration property specifies which sources `licensed` will use to enumerate dependencies.
-By default, `licensed` will generally try to enumerate dependencies from all sources.  As a result,
-the configuration property should be used to explicitly disable sources rather than to enable a particular source.
-
-Be aware that this configuration is separate from an individual sources `#enabled?` method, which determines
-whether the source is valid for the current project.  Even if a source is enabled in the configuration
-it may still determine that it can't enumerate dependencies for a project.
+Additionally, some dependency sources have their own specific configuration options.  See the [source documentation](./sources) for details.
 
-```yml
-sources:
-  bower: true
-  bundler: false
-```
-
-`licensed` determines which sources will try to enumerate dependencies based on the following rules:
-1. If no sources are configured, all sources are enabled
-2. If no sources are set to true, any unconfigured sources are enabled
-```yml
-sources:
-  bower: false
-  # all other sources are enabled by default since there are no sources set to true
-```
-3. If any sources are set to true, any unconfigured sources are disabled
-```yml
-sources:
-  bower: true
-  # all other sources are disabled by default because a source was set to true
-```
-
-## Applications
-
-What is an "app"?  In the context of `licensed`, an app is a combination of a source path and a cache path.
-
-Configuration can be set up for single or multiple applications in the same repo.  There are a number of settings available for each app:
 ```yml
 # If not set, defaults to the directory name of `source_path`
 name: 'My application'
@@ -129,100 +66,11 @@ reviewed:
   bower:
   - classlist # public domain
   - octicons
-```
-
-### Specifying a single app
-To specify a single app, either include a single app with `source_path` in the `apps` configuration, or remove the `apps` setting entirely.
-
-If the configuration does not contain an `apps` value, the root configuration will be used as an app definition.  In this scenario, the `source_path` is not a required value and will default to the directory that `licensed` was executed from.
-
-If the configuration contains an `apps` value with a single app configuration, `source_path` must be specified.  Additionally, the applications inherited `cache_path` value will contain the application name.  See [Inherited cache_path values](#inherited_cache_path_values)
-
-### Specifying multiple apps
-The configuration file can specify multiple source paths to enumerate metadata, each with their own configuration.
-
-Nearly all configuration settings can be inherited from root configuration to app configuration.  Only `source_path` is required to define an app.
 
-Here are some examples:
-
-#### Inheriting configuration
-```yml
-sources:
-  go: true
-  bundler: false
-
-ignored:
-  bundler:
-    - some-internal-gem
-
-reviewed:
-  bundler:
-    - bcrypt-ruby
-
-cache_path: 'path/to/cache'
-apps:
-  - source_path: 'path/to/app1'
-  - source_path: 'path/to/app2'
-    sources:
-      bundler: true
-      go: false
-```
-
-In this example, two apps have been declared.  The first app, with `source_path` `path/to/app1`, inherits all configuration settings from the root configuration.  The second app, with `source_path` `path/to/app2`, overrides the `sources` configuration and inherits all other settings.
-
-#### Default app names
-An app will not inherit a name set from the root configuration.  If not provided, the `name` value will default to the directory name from `source_path`.
-```yml
+# A single configuration file can be used to enumerate dependencies for multiple
+# projects.  Each configuration is referred to as an "application" and must include
+# a source path, at a minimum
 apps:
-  - source_path: 'path/to/app1'
-  - source_path: 'path/to/app2'
+  - source_path: path/to/application1
+  - source_path: path/to/application2
 ```
-
-In this example, the apps have names of `app1` and `app2`, respectively.
-
-#### Inherited cache_path values
-When an app inherits a `cache_path` from the root configuration, it will automatically append it's name to the end of the path to separate it's metadata from other apps.  To force multiple apps to use the same path to cached metadata, explicitly set the `cache_path` value for each app.
-```yml
-cache_path: 'path/to/cache'
-apps:
-  - source_path: 'path/to/app1'
-    name: 'app1'
-  - source_path: 'path/to/app2'
-    name: 'app2'
-  - source_path: 'path/to/app3'
-    name: 'app3'
-    cache_path: 'path/to/app3/cache'
-```
-
-In this example `app1` and `app2` have `cache_path` values of `path/to/cache/app1` and `path/to/cache/app2`, respectively.  `app3` has an explicit path set to `path/to/app3/cache`
-
-```yml
-apps:
-  - source_path: 'path/to/app1'
-```
-
-In this example, the root configuration will contain a default cache path of `.licenses`.  `app1` will inherit this value and append it's name, resulting in a cache path of `.licenses/app1`.
-
-### Sharing caches between apps
-
-Dependency caches can be shared between apps by setting the same cache path on each app.
-
-```yaml
-apps:
-  - source_path: "path/to/app1"
-    cache_path: ".licenses/apps"
-  - source_path: "path/to/app2"
-    cache_path: ".licenses/apps"
-```
-
-When using a source path with a glob pattern, the apps created from the glob pattern can share a dependency by setting an explicit cache path and setting `shared_cache` to true.
-
-```yaml
-source_path: "path/to/apps/*"
-cache_path: ".licenses/apps"
-shared_cache: true
-```
-
-## Source specific configuration
-
-See the [source documentation](./sources) for details on any source specific configuration.

data/docs/configuration/README.md

@@ -0,0 +1,11 @@
+# Configuration options
+
+1. [Application source path](./application_source.md)
+1. [Dependency metadata cache](./metadata_cache.md)
+1. [Configuring multiple applications / monorepo support](./configuring_multiple_apps.md)
+1. [Configuration root](./configuration_root.md)
+1. [Application name](./application_name.md)
+1. [Dependency source enumerators](./dependency_source_enumerators.md)
+1. [Allowed licenses](./allowed_licenses.md)
+1. [Ignoring dependencies](./ignoring_dependencies.md)
+1. [Reviewing dependencies](./reviewing_dependencies.md)

data/docs/configuration/allowed_licenses.md

@@ -0,0 +1,17 @@
+# Allowed licenses
+
+**Key**: allowed
+**Default Value**: none
+
+The list of allowed licenses is used with the [status command](../commands/status.md) to detail which licenses are allowable for use in the current project and do not need further review.  If a dependency uses a license that is not included in the allowed list, and the dependency is not on the ignored or reviewed dependency lists, it will be flagged and the status command will fail.
+
+This configuration value accepts an array of lower-cased [open source license SPDX identifiers](https://spdx.org/licenses/).
+
+```yml
+# accepts lowercase SPDX license identifiers
+allowed:
+  - mit
+  - bsd-2-clause
+  - bsd-3-clause
+  - isc
+```