licensed 0.11.1 → 1.0.0

data/docs/sources/bower.md

@@ -0,0 +1,5 @@
+# Bower
+
+The bower source will detect dependencies when the source is enabled and either `.bowerrc` or `bower.json` files are found at an apps `source_path`.
+
+It enumerates dependencies and metadata from parsing `.bower.json` files for bower components.

data/docs/sources/bundler.md

@@ -0,0 +1,7 @@
+# Bundler (rubygem)
+
+The bundler source will detect dependencies when the source is enabled, `Gemfile` and `Gemfile.lock` files are found at an apps `source_path`.  The source uses the `Bundler` API to enumerate dependencies from `Gemfile` and `Gemfile.lock`.
+
+The bundler source will exclude gems in the `:development` and `:test` groups.  Be aware that if you have a local
+bundler configuration (e.g. `.bundle`), that configuration will be respected as well.  For example, if you have a local
+configuration set for `without: [':server']`, the bundler source will exclude all gems in the `:server` group.

data/docs/sources/cabal.md

@@ -0,0 +1,39 @@
+# Cabal
+
+The cabal source uses the `ghc-pkg` command to enumerate dependencies and provide metadata.  It is un-opinionated on GHC packagedb locations and requires some configuration to ensure that all packages are properly found.
+
+The rubygem source will detect dependencies when the source is enabled and a `.cabal` file is found at an apps `source_path`.
+
+### Specifying GHC packagedb locations through environment
+You can configure the `cabal` source to use specific packagedb locations by setting the `GHC_PACKAGE_PATH` environment variable before running `licensed`.
+
+For example, the following is a useful configuration for use with `cabal new-build`.
+```bash
+ghc_version="$(ghc --numeric-version)"
+CABAL_STORE_PACKAGE_DB="$(cd ~/.cabal/store/ghc-$ghc_version/package.db && pwd)"
+LOCAL_PACKAGE_DB="$ROOT/dist-newstyle/packagedb/ghc-$ghc_version"
+GLOBAL_PACKAGE_DB="$(ghc-pkg list --global | head -n 1)"
+export GHC_PACKAGE_PATH="$LOCAL_PACKAGE_DB:$CABAL_STORE_PACKAGE_DB:$GLOBAL_PACKAGE_DB:$GHC_PACKAGE_PATH"
+
+bundle exec licensed <args>
+```
+
+### Specifying GHC packagedb locations through configuration
+Alternatively, the `cabal` source can use packagedb locations set in the app configuration.  The following is an example configuration identical to the above environment configuration.
+
+```yml
+cabal:
+  ghc_package_db:
+    - ~/.cabal/store/ghc-<ghc_version>/package.db
+    - dist-newstyle/packagedb/ghc-<ghc_version>
+    - global
+```
+
+Ordering is preserved when evaluating the configuration values.  Paths are expanded from the root of the `git` repository and used as values for CLI `--package-db="<path>"` arguments.  Additionally, in all specified paths, `<ghc_version>` will be replaced with the result of `ghc --numeric-version`.
+
+The `global` and `user` keywords are supported and will expand to the `--global` and `--user` CLI arguments, respectively.
+
+Like most other settings, the `cabal` configuration can be specified for the root configuration, or per-app.  If specified at root, it will be inherited by all apps unless explicitly overridden.
+
+### Stack sandboxes
+The current recommended way to use `licensed` with stack is to run the `licensed` command with `stack exec`: `stack exec -- bundle exec licensed <args>`.  This will allow stack to control the GHC packagedb locations.

data/docs/sources/go.md

@@ -0,0 +1,12 @@
+# Go
+
+The go source uses `go` CLI commands to enumerate dependencies and properties.  It is expected that `go` projects have been built, and that `GO_PATH` and `GO_ROOT` are properly set before running `licensed`.
+
+Source paths for go projects should point to a location that contains an entrypoint to the executable or library.
+
+An example usage might see a configuration like:
+```YAML
+source_path: go/path/src/github.com/BurntSushi/toml/cmd/tomlv
+```
+
+Note that this configuration points directly to the tomlv command source, which contains `func main`.

data/docs/sources/manifests.md

@@ -0,0 +1,26 @@
+# Manifests
+
+The manifest source can be used when no package managers are available.
+
+Manifest files are used to match source files with their corresponding packages to find package dependencies.  Manifest file paths can be specified in the app configuration with the following setting:
+```yml
+manifest:
+  path: 'path/to/manifest.json'
+```
+
+If a manifest path is not specified for an app, the file will be looked for at the apps `<cache_path>/manifest.json`.
+
+The manifest can be a JSON or YAML file with a single root object and properties mapping file paths to package names.
+```JSON
+{
+  "file1": "package1",
+  "path/to/file2": "package1",
+  "other/file3": "package2"
+}
+```
+
+File paths are relative to the git repository root.  Package names will be used for the metadata file names at `path/to/cache/manifest/<package>.txt`
+
+If multiple source files map to a single package and they share a common path under the git repository root, that directory will be used to find license information, if available.
+
+It is the responsibility of the repository owner to maintain the manifest file.

data/docs/sources/npm.md

@@ -0,0 +1,3 @@
+# NPM
+
+The npm source will detect dependencies when the source is enabled and `package.json` is found at an apps `source_path`.  It uses `npm list` to enumerate dependencies and metadata.

data/docs/sources/stack.md

@@ -0,0 +1,3 @@
+# HaskellStack
+
+It is not recommended to use this source.  Please see [cabal documentation](./cabal.md) for using the cabal source with stack.

data/exe/licensed

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require "licensed/cli"
 Licensed::CLI.start

data/lib/licensed.rb

@@ -1,16 +1,18 @@
+# frozen_string_literal: true
 require "licensed/version"
+require "licensed/shell"
 require "licensed/configuration"
 require "licensed/license"
 require "licensed/dependency"
+require "licensed/git"
 require "licensed/source/bundler"
 require "licensed/source/bower"
 require "licensed/source/manifest"
 require "licensed/source/npm"
-require "licensed/source/stack"
 require "licensed/source/go"
 require "licensed/source/cabal"
 require "licensed/command/cache"
-require "licensed/command/verify"
+require "licensed/command/status"
 require "licensed/command/list"
 require "licensed/ui/shell"
 require "octokit"
@@ -25,18 +27,20 @@ module Licensed
   GITHUB_URL = %r{\Ahttps://github.com/([a-z0-9]+(-[a-z0-9]+)*/(\w|\.|\-)+)}
   LICENSE_CONTENT_TYPE = "application/vnd.github.drax-preview+json"
 
+  # Load license content from a GitHub url.  Returns nil if the url does not point
+  # to a GitHub repository, or if the license content is not found
   def self.from_github(url)
     return unless use_github && match = GITHUB_URL.match(url)
 
     license_url = Octokit::Repository.path(match[1]) + "/license"
-    response = octokit.get license_url, :accept => LICENSE_CONTENT_TYPE
-    content = Base64.decode64(response["content"]).force_encoding('UTF-8')
+    response = octokit.get license_url, accept: LICENSE_CONTENT_TYPE
+    content = Base64.decode64(response["content"]).force_encoding("UTF-8")
     Licensee::ProjectFiles::LicenseFile.new(content, response["name"])
   rescue Octokit::NotFound
     nil
   end
 
   def self.octokit
-    @octokit ||=  Octokit::Client.new(:access_token => ENV["GITHUB_TOKEN"])
+    @octokit ||=  Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
   end
 end

data/lib/licensed/cli.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require "licensed"
 require "thor"
 
@@ -5,35 +6,42 @@ module Licensed
   class CLI < Thor
 
     desc "cache", "Cache the licenses of dependencies"
-    method_option :force, :type => :boolean,
-      :desc => "Overwrite licenses even if version has not changed."
-    method_option :offline, :type => :boolean,
-      :desc => "Do not make network calls."
-    method_option "license-dir", :type => :string,
-      :desc => "Path to license storage directory (defaults to vendor/licenses)"
+    method_option :force, type: :boolean,
+      desc: "Overwrite licenses even if version has not changed."
+    method_option :offline, type: :boolean,
+      desc: "Do not make network calls."
+    method_option :config, aliases: "-c", type: :string,
+      desc: "Path to licensed configuration file"
     def cache
       Licensed.use_github = false if options[:offline]
       run Licensed::Command::Cache.new(config), force: options[:force]
     end
 
-    desc "verify", "Verify licenses of dependencies"
-    method_option "license-dir", :type => :string,
-      :desc => "Path to license storage directory (defaults to vendor/licenses)"
-    def verify
-      run Licensed::Command::Verify.new(config)
+    desc "status", "Check status of dependencies' cached licenses"
+    method_option :config, aliases: "-c", type: :string,
+      desc: "Path to licensed configuration file"
+    def status
+      run Licensed::Command::Status.new(config)
     end
 
     desc "list", "List dependencies"
-    method_option "license-dir", :type => :string,
-      :desc => "Path to license storage directory (defaults to vendor/licenses)"
+    method_option :config, aliases: "-c", type: :string,
+      desc: "Path to licensed configuration file"
     def list
       run Licensed::Command::List.new(config)
     end
 
     private
 
+    # Returns a configuration object for the CLI command
     def config
-      @config ||= Configuration.new(options)
+      @config ||= Configuration.load_from(config_path)
+    end
+
+    # Returns a config path from the CLI if set.
+    # Defaults to the current directory if CLI option is not set
+    def config_path
+      options["config"] || Dir.pwd
     end
 
     def run(command, *args)

data/lib/licensed/command/cache.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 module Licensed
   module Command
     class Cache
@@ -8,49 +9,65 @@ module Licensed
       end
 
       def run(force: false)
-        @config.sources.each do |source|
-          @config.ui.info "Caching licenses for #{source.type} dependencies:"
+        summary = @config.apps.flat_map do |app|
+          app_name = app["name"]
+          @config.ui.info "Caching licenes for #{app_name}:"
 
-          dependencies(source).each do |dependency|
-            filename = @config.path.join("#{source.type}/#{dependency["name"]}.txt")
+          # load the app environment
+          Dir.chdir app.source_path do
 
-            if File.exist?(filename)
-              license = Licensed::License.read(filename)
+            # map each available app source to it's dependencies
+            app.sources.map do |source|
+              @config.ui.info "  #{source.type} dependencies:"
 
-              # Version did not change, no need to re-cache
-              if !force && dependency["version"] == license["version"]
-                @config.ui.info "  Using #{dependency["name"]} (#{dependency["version"]})"
-                next
-              end
-            end
+              names = []
+              cache_path = app.cache_path.join(source.type)
 
-            @config.ui.info "  Caching #{dependency["name"]} (#{dependency["version"]})"
+              # exclude ignored dependencies
+              dependencies = source.dependencies.select { |d| !app.ignored?(d) }
 
-            dependency.detect_license!
-            dependency.save(filename)
-          end
+              # ensure each dependency is cached
+              dependencies.each do |dependency|
+                name = dependency["name"]
+                version = dependency["version"]
+
+                names << name
+                filename = cache_path.join("#{name}.txt")
 
-          # Clean up dependencies that have been removed
-          names = dependencies(source).map { |d| d["name"] }
+                if File.exist?(filename)
+                  license = Licensed::License.read(filename)
 
-          license_source_path = @config.path.join(source.type)
-          Dir.glob(license_source_path.join("**/*.txt")).each do |file|
-            file_path = Pathname.new(file)
-            relative_path = file_path.relative_path_from(license_source_path).to_s
-            FileUtils.rm(file) unless names.include?(relative_path.chomp(".txt"))
+                  # Version did not change, no need to re-cache
+                  if !force && version == license["version"]
+                    @config.ui.info "    Using #{name} (#{version})"
+                    next
+                  end
+                end
+
+                @config.ui.info "    Caching #{name} (#{version})"
+
+                dependency.detect_license!
+                dependency.save(filename)
+              end
+
+              # Clean up cached files that dont match current dependencies
+              Dir.glob(cache_path.join("**/*.txt")).each do |file|
+                file_path = Pathname.new(file)
+                relative_path = file_path.relative_path_from(cache_path).to_s
+                FileUtils.rm(file) unless names.include?(relative_path.chomp(".txt"))
+              end
+
+              "* #{app_name} #{source.type} dependencies: #{dependencies.size}"
+            end
           end
         end
 
         @config.ui.confirm "License caching complete!"
-        @config.sources.each do |source|
-          @config.ui.confirm "* #{source.type} dependencies: #{dependencies(source).size}"
+        summary.each do |message|
+          @config.ui.confirm message
         end
       end
 
-      def dependencies(source)
-        source.dependencies.select { |d| !@config.ignored?(d) }
-      end
-
       def success?
         true
       end

data/lib/licensed/command/list.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 module Licensed
   module Command
     class List
@@ -8,21 +9,28 @@ module Licensed
       end
 
       def run
-        @config.sources.each do |source|
-          @config.ui.info "Displaying licenses for #{source.type} dependencies:"
+        @config.apps.each do |app|
+          @config.ui.info "Displaying dependencies for #{app['name']}"
+          Dir.chdir app.source_path do
+            app.sources.each do |source|
+              @config.ui.info "  #{source.type} dependencies:"
 
-          dependencies(source).each do |dependency|
-            @config.ui.info "  Found #{dependency['name']} (#{dependency['version']})"
-          end
+              source_dependencies = dependencies(app, source)
+              source_dependencies.each do |dependency|
+                @config.ui.info "    Found #{dependency['name']} (#{dependency['version']})"
+              end
 
-          @config.ui.confirm "* #{source.type} dependencies: #{dependencies(source).size}"
+              @config.ui.confirm "  * #{source.type} dependencies: #{source_dependencies.size}"
+            end
+          end
         end
       end
 
-      def dependencies(source)
+      # Returns an apps non-ignored dependencies, sorted by name
+      def dependencies(app, source)
         source.dependencies
-              .select { |d| !@config.ignored?(d) }
-              .sort_by { |d| d['name'] }
+              .select { |d| !app.ignored?(d) }
+              .sort_by { |d| d["name"] }
       end
 
       def success?

data/lib/licensed/command/status.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+require "yaml"
+
+module Licensed
+  module Command
+    class Status
+      attr_reader :config
+
+      def initialize(config)
+        @config = config
+      end
+
+      def allowed_or_reviewed?(app, dependency)
+        app.allowed?(dependency) || app.reviewed?(dependency)
+      end
+
+      def app_dependencies(app)
+        app.sources.flat_map(&:dependencies).select { |d| !app.ignored?(d) }
+      end
+
+      def run
+        @results = @config.apps.flat_map do |app|
+          Dir.chdir app.source_path do
+            dependencies = app_dependencies(app)
+            @config.ui.info "Checking licenses for #{app['name']}: #{dependencies.size} dependencies"
+
+            results = dependencies.map do |dependency|
+              filename = app.cache_path.join(dependency["type"], "#{dependency["name"]}.txt")
+
+              warnings = []
+
+              # verify cached license data for dependency
+              if File.exist?(filename)
+                license = License.read(filename)
+
+                if license["version"] != dependency["version"]
+                  warnings << "cached license data out of date"
+                end
+                warnings << "missing license text" if license.text.strip.empty?
+                unless allowed_or_reviewed?(app, license)
+                  warnings << "license needs reviewed: #{license["license"]}."
+                end
+              else
+                warnings << "cached license data missing"
+              end
+
+              if warnings.size > 0
+                @config.ui.error("F", false)
+                [filename, warnings]
+              else
+                @config.ui.confirm(".", false)
+                nil
+              end
+            end.compact
+
+            unless results.empty?
+              @config.ui.warn "\n\nWarnings:"
+
+              results.each do |filename, warnings|
+                @config.ui.info "\n#{filename}:"
+                warnings.each do |warning|
+                  @config.ui.error "  - #{warning}"
+                end
+              end
+            end
+
+            puts "\n#{dependencies.size} dependencies checked, #{results.size} warnings found."
+            results
+          end
+        end
+      end
+
+      def success?
+        @results.empty?
+      end
+    end
+  end
+end

data/lib/licensed/configuration.rb

@@ -1,38 +1,50 @@
+# frozen_string_literal: true
 require "pathname"
 
 module Licensed
-  class Configuration < Hash
-    attr_accessor :ui
-
-    def initialize(options = {})
+  class AppConfiguration < Hash
+    DEFAULT_CACHE_PATH = ".licenses".freeze
+    DEFAULT_CONFIG_FILES = [
+      ".licensed.yml".freeze,
+      ".licensed.yaml".freeze,
+      ".licensed.json".freeze
+    ].freeze
+
+    def initialize(options = {}, inherited_options = {})
       super()
-      self.path = options["license-dir"] if options["license-dir"]
-      update config_path.exist? ? YAML.load_file(config_path) : {}
+
+      # update order:
+      # 1. anything inherited from root config
+      # 2. app defaults
+      # 3. explicitly configured app settings
+      update(inherited_options)
+      update(defaults_for(options, inherited_options))
+      update(options)
 
       self["sources"] ||= {}
       self["reviewed"] ||= {}
       self["ignored"] ||= {}
-      self["whitelist"] ||= []
+      self["allowed"] ||= []
 
-      @ui = Licensed::UI::Shell.new
+      verify_arg "source_path"
+      verify_arg "cache_path"
     end
 
-    def path
-      @path ||= Pathname.new("vendor/licenses")
+    # Returns the path to the app cache directory as a Pathname
+    def cache_path
+      Licensed::Git.repository_root.join(self["cache_path"])
     end
 
-    def path=(value)
-      @path = Pathname.new(value)
-    end
-
-    def config_path
-      path.join("config.yml")
+    # Returns the path to the app source directory as a Pathname
+    def source_path
+      Licensed::Git.repository_root.join(self["source_path"])
     end
 
     def pwd
-      Pathname.new(Dir.pwd)
+      Pathname.pwd
     end
 
+    # Returns an array of enabled app sources
     def sources
       @sources ||= [
         Source::Bundler.new(self),
@@ -40,16 +52,16 @@ module Licensed
         Source::Cabal.new(self),
         Source::Go.new(self),
         Source::Manifest.new(self),
-        Source::NPM.new(self),
-        Source::Stack.new(self)
+        Source::NPM.new(self)
       ].select(&:enabled?)
     end
 
+    # Returns whether a source type is enabled
     def enabled?(source_type)
       self["sources"].fetch(source_type, true)
     end
 
-    # Is the given dependency approved?
+    # Is the given dependency reviewed?
     def reviewed?(dependency)
       Array(self["reviewed"][dependency["type"]]).include?(dependency["name"])
     end
@@ -59,21 +71,111 @@ module Licensed
       Array(self["ignored"][dependency["type"]]).include?(dependency["name"])
     end
 
-    # Is the license of the dependency whitelisted?
-    def whitelisted?(dependency)
-      Array(self["whitelist"]).include?(dependency["license"])
+    # Is the license of the dependency allowed?
+    def allowed?(dependency)
+      Array(self["allowed"]).include?(dependency["license"])
     end
 
+    # Ignore a dependency
     def ignore(dependency)
       (self["ignored"][dependency["type"]] ||= []) << dependency["name"]
     end
 
+    # Set a dependency as reviewed
     def review(dependency)
       (self["reviewed"][dependency["type"]] ||= []) << dependency["name"]
     end
 
-    def whitelist(license)
-      self["whitelist"] << license
+    # Set a license as explicitly allowed
+    def allow(license)
+      self["allowed"] << license
+    end
+
+    def defaults_for(options, inherited_options)
+      name = options["name"] || File.basename(options["source_path"])
+      cache_path = inherited_options["cache_path"] || DEFAULT_CACHE_PATH
+      {
+        "name" => name,
+        "cache_path" => File.join(cache_path, name)
+      }
+    end
+
+    def verify_arg(property)
+      return if self[property]
+      raise Licensed::Configuration::LoadError,
+        "App #{self["name"]} is missing required property #{property}"
+    end
+  end
+
+  class Configuration < AppConfiguration
+    class LoadError < StandardError; end
+
+    attr_accessor :ui
+
+    # Loads and returns a Licensed::Configuration object from the given path.
+    # The path can be relative or absolute, and can point at a file or directory.
+    # If the path given is a directory, the directory will be searched for a
+    # `config.yml` file.
+    def self.load_from(path)
+      config_path = Pathname.pwd.join(path)
+      config_path = find_config(config_path) if config_path.directory?
+      Configuration.new(parse_config(config_path))
+    end
+
+    def initialize(options = {})
+      @ui = Licensed::UI::Shell.new
+
+      apps = options.delete("apps") || []
+      super(default_options.merge(options))
+
+      self["apps"] = apps.map { |app| AppConfiguration.new(app, options) }
+    end
+
+    # Returns an array of the applications for this licensed configuration.
+    # If the configuration did not explicitly configure any applications,
+    # return self as an application configuration.
+    def apps
+      return [self] if self["apps"].empty?
+      self["apps"]
+    end
+
+    private
+
+    # Find a default configuration file in the given directory.
+    # File preference is given by the order of elements in DEFAULT_CONFIG_FILES
+    #
+    # Raises Licensed::Configuration::LoadError if a file isn't found
+    def self.find_config(directory)
+      config_file = DEFAULT_CONFIG_FILES.map { |file| directory.join(file) }
+                                        .find { |file| file.exist? }
+
+      config_file || raise(LoadError, "Licensed configuration not found in #{directory}")
+    end
+
+    # Parses the configuration given at `config_path` and returns the values
+    # as a Hash
+    #
+    # Raises Licensed::Configuration::LoadError if the file type isn't known
+    def self.parse_config(config_path)
+      return {} unless config_path.file?
+
+      extension = config_path.extname.downcase.delete "."
+      case extension
+      when "json"
+        JSON.parse(File.read(config_path))
+      when "yml", "yaml"
+        YAML.load_file(config_path)
+      else
+        raise LoadError, "Unknown file type #{extension} for #{config_path}"
+      end
+    end
+
+    def default_options
+      # manually set a cache path without additional name
+      {
+        "source_path" => Dir.pwd,
+        "cache_path" => DEFAULT_CACHE_PATH
+      }
     end
   end
 end