licensed 3.1.0 → 3.2.0

data/docs/commands/env.md

@@ -0,0 +1,10 @@
+# `licensed env`
+
+Prints the runtime environment used by licensed after loading a configuration file.  This can be different from the configuration file inputs, for example all paths will be given as absolute file paths and glob paths may be expanded.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--format`/`-f`: the output format
+   - default value: `yaml`

data/docs/commands/list.md

@@ -0,0 +1,23 @@
+# `licensed list`
+
+The list command finds and prints the dependencies for all sources in all configured applications.  No additional actions are taken on dependencies.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--licenses`/`-l`: if set, includes each dependency's detected license in the output
+   - default value: not set
+
+### Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- version: the version of the enumerated dependency
+- license: (optional) the dependency's SPDX license identifier

data/docs/commands/migrate.md

@@ -0,0 +1,10 @@
+# `licensed migrate`
+
+Migrates the licensed configuration and cached metadata files from a previous version to the most recent version.  This is not required for all major version updates.  See [migrations documentation](../migrations) for details on the migrations needed for each major version.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--from`/`-f`: the licensed version to migrate from
+   - required

data/docs/commands/notices.md

@@ -0,0 +1,12 @@
+# `licensed notices`
+
+Outputs license and notice text for all dependencies in each app into a `NOTICE` file in the app's `cache_path`.  If an app uses a shared cache path, the file name will contain the app name as well, e.g. `NOTICE.my_app`.
+
+`NOTICE` file contents are retrieved from cached records, with the assumption that cached records have already been reviewed in a compliance workflow.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources

data/docs/commands/status.md

@@ -0,0 +1,73 @@
+# `licensed status`
+
+The status command finds all dependencies and checks whether each dependency has a valid cached record.
+
+A dependency will fail the status checks if:
+
+1. No cached record is found
+2. The cached record's version is different than the current dependency's version
+3. The cached record's `licenses` data is empty
+4. The cached record's `license` metadata doesn't match an `allowed` license from the dependency's application configuration.
+   - If `license: other` is specified and all of the `licenses` entries match an `allowed` license a failure will not be logged
+5. The cached record is flagged for re-review.
+   - This occurs when the record's license text has changed since the record was reviewed.
+
+## Options
+
+- `--config`/`-c`: the path to the licensed configuration file
+   - default value: `./.licensed.yml`
+- `--sources`/`-s`: runtime filter on which dependency sources are run.  Sources must also be enabled in the licensed configuration file.
+   - default value: not set, all configured sources
+- `--format`/`-f`: the output format
+   - default value: `yaml`
+- `--force`: if set, forces all dependency metadata files to be recached
+   - default value: not set
+
+## Reported Data
+
+The following data is reported for each dependency when the YAML or JSON report formats are used
+
+- name: the licensed recognized name for the dependency including the app and source name
+   - e.g. the full name for the `thor` bundler dependency used by this tool is `licensed.bundler.thor`
+- allowed: true if the dependency has passed all checks, false otherwise
+- version: the version of the enumerated dependency
+- license: the dependency's SPDX license identifier
+- filename: the full path on disk to the dependency's cached metadata file, if available
+- errors: any error messages from failed status checks, if available
+
+## Status errors and resolutions
+
+### cached dependency record not found
+
+**Cause:** A dependency was found while running `licensed status` that does not have a corresponding cached metadata file
+**Resolution:** Run `licensed cache` to update the metadata cache and create the missing metadata file
+
+### cached dependency record out of date
+
+**Cause:** A dependency was found while running `licensed status` with a different version than is contained in the dependency's cached metadata file
+**Resolution:** Run `licensed cache` to update the out-of-date metadata files
+
+### missing license text
+
+**Cause:** A license determination was made, e.g. from package metadata, but no license text was found.
+**Resolution:** Manually verify whether the dependency includes a file containing license text.  If the dependency code that was downloaded locally does not contain the license text, please check the dependency source at the version listed in the dependency's cached metadata file to see if there is license text that can be used.
+
+If the dependency does not include license text but does specify that it uses a specific license, please copy the standard license text from a [well known source](https://opensource.org/licenses).
+
+### license text has changed and needs re-review. if the new text is ok, remove the `review_changed_license` flag from the cached record
+
+**Cause:** A dependency that is set as [reviewed] in the licensed configuration file has substantially changed and should be re-reviewed.
+**Resolution:** Review the changes to the license text and classification, along with other metadata contained in the cached file for the dependency.  If the dependency is still allowable for use in your project, remove the `review_changed_license` key from the cached record file.
+
+### license needs review
+
+**Cause:** A dependency is using a license that is not in the configured [allowed list of licenses][allowed], and the dependency has not been marked [ignored] or [reviewed].
+**Resolution:** Review the dependency's usage and specified license with someone familiar with OSS licensing and compliance rules to determine whether the dependency is allowable.  Some common resolutions:
+
+1. The dependency's specified license text differed enough from the standard license text that it was not recognized and classified as `other`.  If, with human review, the license text is recognizable then update the `license: other` value in the cached metadata file to the correct license.
+1. The dependency might need to be marked as [ignored] or [reviewed] if either of those scenarios are applicable.
+1. If the used license should be allowable without review (if your entity has a legal team, they may want to review this assessment), ensure the license SPDX is set as [allowed] in the licensed configuration file.
+
+[allowed]: ../configuration/allowed_licenses.md
+[ignored]: ../configuration/ignoring_dependencies.md
+[reviewed]: ../configuration/reviewing_dependencies.md

data/docs/commands/version.md

@@ -0,0 +1,3 @@
+# `licensed version`
+
+Displays the current licensed version.  Can also be run with `licensed -v` and `licensed --version`.

data/docs/configuration.md

@@ -2,87 +2,12 @@
 
 A configuration file specifies the details of enumerating and operating on license metadata for apps.
 
-Configuration can be specified in either YML or JSON formats.  Examples below are given in YML.
+Configuration can be specified in either YML or JSON formats, with examples given in YML.  The example
+below describes common configuration values and their purposes.  See [configuration options documentation](./configuration)
+for in depth information.  
 
-## Configuration Paths
+Additionally, some dependency sources have their own specific configuration options.  See the [source documentation](./sources) for details.
 
-`licensed` requires a path to enumerate dependencies at (`source_path`) and a path to store cached metadata (`cache_path`).
-
-To determine these paths across multiple environments where absolute paths will differ, a known root path is needed to evaluate relative paths against.
-In using a root, relative source and cache paths can be specified in the configuration file.
-
-When using a configuration file, the root property can be set as either a path that can be expanded from the configuration file directory using `File.expand_path`, or the value `true` to use the configuration file directory as the root.
-
-When creating a `Licensed::Dependency` manually with a `root` property, the property must be an absolute path - no path expansion will occur.
-
-If a root path is not specified, it will default to using the following, in order of precedence
-1. the root of the local git repository, if run inside a git repository
-2. the current directory
-
-### Source paths
-
-A source path is the directory in which licensed should run to enumerate dependencies.  This is often dependent on the project type, for example the bundler source should be run from the directory containing a `Gemfile` or `gems.rb` while the go source should be run from the directory containing an entrypoint function.
-
-#### Using glob patterns
-
-The `source_path` property can use one or more glob patterns to share configuration properties across multiple application entrypoints.
-
-For example, there is a common pattern in Go projects to include multiple executable entrypoints under folders in `cmd`.  Using a glob pattern allows users to avoid manually configuring and maintaining multiple licensed application `source_path`s.  Using a glob pattern will also ensure that any new entrypoints matching the pattern are automatically picked up by licensed commands as they are added.
-
-```yml
-sources:
-  go: true
-
-# treat all directories under `cmd` as separate apps
-source_path: cmd/*
-```
-
-In order to better filter the results from glob patterns, the `source_path` property also accepts an array of inclusion and exclusion glob patterns similar to gitignore files.  Inclusion patterns will add matching directory paths to resulting set of source paths, while exclusion patterns will remove matching directory paths.
-
-```yml
-source_path:
-  - "projects/*" # include by default all directories under "projects"
-  - "!projects/*Test" # exclude all projects ending in "Test"
-```
-
-Glob patterns are syntactic sugar for, and provide the same functionality as, manually specifying multiple `source_path` values. See the instructions on [specifying multiple apps](./#specifying-multiple-apps) below for additional considerations when using multiple apps.
-
-## Restricting sources
-
-The `sources` configuration property specifies which sources `licensed` will use to enumerate dependencies.
-By default, `licensed` will generally try to enumerate dependencies from all sources.  As a result,
-the configuration property should be used to explicitly disable sources rather than to enable a particular source.
-
-Be aware that this configuration is separate from an individual sources `#enabled?` method, which determines
-whether the source is valid for the current project.  Even if a source is enabled in the configuration
-it may still determine that it can't enumerate dependencies for a project.
-
-```yml
-sources:
-  bower: true
-  bundler: false
-```
-
-`licensed` determines which sources will try to enumerate dependencies based on the following rules:
-1. If no sources are configured, all sources are enabled
-2. If no sources are set to true, any unconfigured sources are enabled
-```yml
-sources:
-  bower: false
-  # all other sources are enabled by default since there are no sources set to true
-```
-3. If any sources are set to true, any unconfigured sources are disabled
-```yml
-sources:
-  bower: true
-  # all other sources are disabled by default because a source was set to true
-```
-
-## Applications
-
-What is an "app"?  In the context of `licensed`, an app is a combination of a source path and a cache path.
-
-Configuration can be set up for single or multiple applications in the same repo.  There are a number of settings available for each app:
 ```yml
 # If not set, defaults to the directory name of `source_path`
 name: 'My application'
@@ -141,100 +66,11 @@ reviewed:
   bower:
   - classlist # public domain
   - octicons
-```
-
-### Specifying a single app
-To specify a single app, either include a single app with `source_path` in the `apps` configuration, or remove the `apps` setting entirely.
-
-If the configuration does not contain an `apps` value, the root configuration will be used as an app definition.  In this scenario, the `source_path` is not a required value and will default to the directory that `licensed` was executed from.
 
-If the configuration contains an `apps` value with a single app configuration, `source_path` must be specified.  Additionally, the applications inherited `cache_path` value will contain the application name.  See [Inherited cache_path values](#inherited_cache_path_values)
-
-### Specifying multiple apps
-The configuration file can specify multiple source paths to enumerate metadata, each with their own configuration.
-
-Nearly all configuration settings can be inherited from root configuration to app configuration.  Only `source_path` is required to define an app.
-
-Here are some examples:
-
-#### Inheriting configuration
-```yml
-sources:
-  go: true
-  bundler: false
-
-ignored:
-  bundler:
-    - some-internal-gem
-
-reviewed:
-  bundler:
-    - bcrypt-ruby
-
-cache_path: 'path/to/cache'
-apps:
-  - source_path: 'path/to/app1'
-  - source_path: 'path/to/app2'
-    sources:
-      bundler: true
-      go: false
-```
-
-In this example, two apps have been declared.  The first app, with `source_path` `path/to/app1`, inherits all configuration settings from the root configuration.  The second app, with `source_path` `path/to/app2`, overrides the `sources` configuration and inherits all other settings.
-
-#### Default app names
-An app will not inherit a name set from the root configuration.  If not provided, the `name` value will default to the directory name from `source_path`.
-```yml
+# A single configuration file can be used to enumerate dependencies for multiple
+# projects.  Each configuration is referred to as an "application" and must include
+# a source path, at a minimum
 apps:
-  - source_path: 'path/to/app1'
-  - source_path: 'path/to/app2'
+  - source_path: path/to/application1
+  - source_path: path/to/application2
 ```
-
-In this example, the apps have names of `app1` and `app2`, respectively.
-
-#### Inherited cache_path values
-When an app inherits a `cache_path` from the root configuration, it will automatically append it's name to the end of the path to separate it's metadata from other apps.  To force multiple apps to use the same path to cached metadata, explicitly set the `cache_path` value for each app.
-```yml
-cache_path: 'path/to/cache'
-apps:
-  - source_path: 'path/to/app1'
-    name: 'app1'
-  - source_path: 'path/to/app2'
-    name: 'app2'
-  - source_path: 'path/to/app3'
-    name: 'app3'
-    cache_path: 'path/to/app3/cache'
-```
-
-In this example `app1` and `app2` have `cache_path` values of `path/to/cache/app1` and `path/to/cache/app2`, respectively.  `app3` has an explicit path set to `path/to/app3/cache`
-
-```yml
-apps:
-  - source_path: 'path/to/app1'
-```
-
-In this example, the root configuration will contain a default cache path of `.licenses`.  `app1` will inherit this value and append it's name, resulting in a cache path of `.licenses/app1`.
-
-### Sharing caches between apps
-
-Dependency caches can be shared between apps by setting the same cache path on each app.
-
-```yaml
-apps:
-  - source_path: "path/to/app1"
-    cache_path: ".licenses/apps"
-  - source_path: "path/to/app2"
-    cache_path: ".licenses/apps"
-```
-
-When using a source path with a glob pattern, the apps created from the glob pattern can share a dependency by setting an explicit cache path and setting `shared_cache` to true.
-
-```yaml
-source_path: "path/to/apps/*"
-cache_path: ".licenses/apps"
-shared_cache: true
-```
-
-## Source specific configuration
-
-See the [source documentation](./sources) for details on any source specific configuration.

data/docs/configuration/README.md

@@ -0,0 +1,11 @@
+# Configuration options
+
+1. [Application source path](./application_source.md)
+1. [Dependency metadata cache](./metadata_cache.md)
+1. [Configuring multiple applications / monorepo support](./configuring_multiple_apps.md)
+1. [Configuration root](./configuration_root.md)
+1. [Application name](./application_name.md)
+1. [Dependency source enumerators](./dependency_source_enumerators.md)
+1. [Allowed licenses](./allowed_licenses.md)
+1. [Ignoring dependencies](./ignoring_dependencies.md)
+1. [Reviewing dependencies](./reviewing_dependencies.md)

data/docs/configuration/allowed_licenses.md

@@ -0,0 +1,17 @@
+# Allowed licenses
+
+**Key**: allowed
+**Default Value**: none
+
+The list of allowed licenses is used with the [status command](../commands/status.md) to detail which licenses are allowable for use in the current project and do not need further review.  If a dependency uses a license that is not included in the allowed list, and the dependency is not on the ignored or reviewed dependency lists, it will be flagged and the status command will fail.
+
+This configuration value accepts an array of lower-cased [open source license SPDX identifiers](https://spdx.org/licenses/).
+
+```yml
+# accepts lowercase SPDX license identifiers
+allowed:
+  - mit
+  - bsd-2-clause
+  - bsd-3-clause
+  - isc
+```

data/docs/configuration/application_name.md

@@ -0,0 +1,63 @@
+# Application name
+
+**Key**: name  
+**Default value**: The directory name of the application's source path
+
+The name of the application is primarily used for organizational and display purposes.  Application names are not included with
+dependency metadata information included in cached files.
+
+```yml
+source_path: path/to/application1
+name: application1
+```
+
+## Dynamically generated application names
+
+### Source path directory name
+
+When not specified, an application's name will be the directory name from the application's source path.
+
+```yml
+# if not explicitly set the name will be inferred from the source path
+source_path: path/to/application1
+# name: application1
+
+# or, use the `directory_name` name generator to explicitly set this behavior
+source_path: path/to/application1
+name:
+  generator: directory_name
+# name: application1
+```
+
+### Relative path from configuration root
+
+Application names can be created from the path from the configuration root to the application source path.
+This can be useful when specifying multiple applications using a glob pattern in a directory structure where directory names
+are not unique.
+
+As an example, given the following directory structure and configuration YML, the resulting application names
+would be `linux.installer`, `windows.installer` and `mac.installer`.  The optional arguments `separator` and `depth` are used
+to better control the resulting application name.
+
+```text
+path
+|_to
+  |_linux
+    |_installer
+  |_windows
+    |_installer
+  |_mac
+    |_installer
+```
+
+```yml
+source_path: '**/installer'
+name:
+  generator: relative_path
+  # separator controls what character separates path parts in the application name
+  # default: "-"
+  separator: '.'
+  # depth controls how many of the path parts are used in the application name
+  # default: 0 / all path parts
+  depth: 2
+```

data/docs/configuration/application_source.md

@@ -0,0 +1,64 @@
+# Application source path
+
+**Key**: source_path  
+**Default value**:
+
+- if the `apps` key is not present, then the current working directory where `licensed` was executed
+- if the `apps` key is present, then `nil`
+
+The source path is the directory in which licensed should run to enumerate dependencies.  This is often dependent
+on the project type, for example the bundler source should be run from the directory containing a `Gemfile` or `gems.rb`
+while the go source should be run from the directory containing an entrypoint function.
+
+The source path is required to run `licensed`.  A default value is available only when the configuration file specifies a single application.
+When multiple applications are configured, each application must specify a source path.
+
+Paths can be given as absolute or relative paths, and can use special path identifiers. If a relative path is given, it will be based on the application's root path.
+
+```yml
+# when apps is not set, a source path does not need to be specified.  it will default to the users current directory
+sources:
+  bundler: true
+
+# ------
+# or a path can be given as either an absolute or relative path
+sources:
+  bundler: true
+source_path: path/to/application1
+
+# ------
+# when apps is set, each application must specify a source_path
+sources:
+  bundler: true
+apps:
+  - source_path: relative/path/to/application1
+  - source_path: /absolute/path/to/application2
+  - source_path: ~/path/from/home/to/application3
+```
+
+## Expanding source paths with glob patterns
+
+The `source_path` property can use one or more glob patterns to share configuration properties across multiple application entrypoints.
+
+For example, there is a common pattern in Go projects to include multiple executable entrypoints under folders in `cmd`.  Using a glob pattern allows users to avoid manually configuring and maintaining multiple licensed application `source_path`s.  Using a glob pattern will also ensure that any new entrypoints matching the pattern are automatically picked up by licensed commands as they are added.
+
+```yml
+sources:
+  go: true
+
+# treat all directories under `cmd` as separate apps
+source_path: cmd/*
+```
+
+In order to better filter the results from glob patterns, the `source_path` property also accepts an array of
+inclusion and exclusion glob patterns similar to gitignore files.  Inclusion patterns will add matching directory
+paths to resulting set of source paths, while exclusion patterns will remove matching directory paths.
+
+```yml
+source_path:
+  - "projects/*" # include by default all directories under "projects"
+  - "!projects/*Test" # exclude all projects ending in "Test"
+```
+
+Glob patterns are syntactic sugar for, and provide the same functionality as, manually specifying multiple `source_path` values.
+See the instructions on [specifying multiple apps](../configuration.md#specifying-multiple-apps) below for additional considerations when using multiple apps.