licensee 5.0.0 → 6.0.0b1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 177e5c94bb43c3603d0581bfaed3f873ab17c94e
-  data.tar.gz: a9218af2d2922cccbddf269df1bd35d710198c79
+  metadata.gz: a9a12c248b281de58f46188030a603b373bfb502
+  data.tar.gz: 384ffeaeef392f999d1b0c6f20efd6651abcf34d
 SHA512:
-  metadata.gz: 6a001f3ec27daba23c2c0a3d5902fd7f926861d3a5071aa4756591cb313884c77f6c022851f94fd440c1bd894ee45c3f6596b5835a062bf2926e1ff6abd99fd1
-  data.tar.gz: bab94b9c213be6caed4ab179cbe7b4b64bd928482a219ab98b63a24c0700a0190998e71dbfc747af597d9b2671863c0bdf4ab359903686be9200610ef7756323
+  metadata.gz: f90e04e86ea6a97828ac39c52ffeb3661082c5df4a4b1c3944a09555d1e39827c052870ea63541eef675639170ff18f3bc661a2b1e84fe0e8451a3fe299aa04c
+  data.tar.gz: 3c2a06a4e5f9c27fa26df6f8fbb0ff62ce730d0af1dcbaed1774c1b38990158a07813306892f47a9386a0661dae48be27a3bbd31d5c33b197f88cd73a8bcc23b

data/README.md

@@ -1,33 +1,22 @@
 # Licensee
-
-*A Ruby Gem to detect under what license a project is distributed.*
+_A Ruby Gem to detect under what license a project is distributed._
 
 [![Build Status](https://travis-ci.org/benbalter/licensee.svg?branch=master)](https://travis-ci.org/benbalter/licensee) [![Gem Version](https://badge.fury.io/rb/licensee.svg)](http://badge.fury.io/rb/licensee)
 
 ## The problem
-
-* You've got an open source project. How do you know what you can and can't do with the software?
-* You've got a bunch of open source projects, how do you know what their licenses are?
-* You've got a project with a license file, but which license is it? Has it been modified?
+- You've got an open source project. How do you know what you can and can't do with the software?
+- You've got a bunch of open source projects, how do you know what their licenses are?
+- You've got a project with a license file, but which license is it? Has it been modified?
 
 ## The solution
-
 Licensee automates the process of reading `LICENSE` files and compares their contents to known licenses using a several strategies (which we call "Matchers"). It attempts to determine a project's license in the following order:
+- If the license file has an explicit copyright notice, and nothing more (e.g., `Copyright (c) 2015 Ben Balter`), we'll assume the author intends to retain all rights, and thus the project isn't licensed.
+- If the license is an exact match to a known license. If we strip away whitespace and copyright notice, we might get lucky, and direct string comparison in Ruby is cheap.
+- If we still can't match the license, we use a fancy math thing called the [Sørensen–Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient), which is really good at calculating the similarity between two strings. By calculating the percent changed from the known license to the license file, you can tell, e.g., that a given license is 90% similar to the MIT license, that 10% likely representing the copyright line being properly adapted to the project.
 
-1. If the license file has an explicit copyright notice, and nothing more (e.g., `Copyright (c) 2015 Ben Balter`), we'll assume the author intends to retain all rights, and thus the project isn't licensed.
-
-2. If the license is an exact match to a known license. Licenses like GPL don't have a copyright notice that needs to be changed in the license itself, so if we strip away whitespace, we might get lucky, and direct string comparison in Ruby is cheap.
-
-3. If 90% of the lines match a known license. We use Git's internal change calculation method. To calculate diffs, Git hashes each line of both files, and compares the hashes to tell the percent changed. This method is fast, but is done on a line-by-line basis, so if the license is wrapped differently, or has extra words inserted, it's not going to match the license.
-
-4. If we still can't match the license, we use a fancy math thing called the [Levenshtein distance algorithm](https://en.wikipedia.org/wiki/Levenshtein_distance), which while very slow, is really good at calculating the similarity between two strings. By calculating the percent changed from the known license to the license file, you can tell, e.g., that a given license is 90% similar to the MIT license, that 10% likely representing the copyright line being properly adapted to the project.
-
-Licensee will even diff the distributed license with the original, so you can see exactly what, if anything's been changed.
-
-*Special thanks to [@vmg](https://github.com/vmg) for his Git prowess.*
+_Special thanks to [@vmg](https://github.com/vmg) for his Git and algorithmic prowess._
 
 ## Installation
-
 `gem install licensee` or add `gem 'licensee'` to your project's `Gemfile`.
 
 ## Usage
@@ -52,20 +41,7 @@ license.meta["permitted"]
 => ["commercial-use","modifications","distribution","sublicense","private-use"]
 ```
 
-## Diffing
-
-You can also generate a diff of the known license to the distributed license.
-
-```ruby
-puts Licensee.diff "/path/to/a/project"
--Copyright (c) [year] [fullname]
-+Copyright (c) 2014 Ben Balter
-```
-
-For a full list of diff options (HTML output, color output, etc.) see [Diffy](https://github.com/samg/diffy).
-
 ## Command line usage
-
 1. `cd` into a project directory
 2. execute the `licensee` command
 
@@ -78,54 +54,43 @@ Matcher: Licensee::GitMatcher
 ```
 
 ## What it looks at
-
-* `LICENSE`, `LICENSE.txt`, `COPYING`, etc. files in the root of the project, comparing the body to known licenses
-* Crowdsourced license content and metadata from [`choosealicense.com`](http://choosealicense.com)
+- `LICENSE`, `LICENSE.txt`, `COPYING`, etc. files in the root of the project, comparing the body to known licenses
+- Crowdsourced license content and metadata from [`choosealicense.com`](http://choosealicense.com)
 
 ## What it doesn't look at
-
-* Dependency licensing
-* References to licenses in `README`, `README.md`, etc.
-* Structured license data in package manager configuration files (like Gemfiles)
-* Every single possible license (just the most popular ones)
-* Compliance (e.g., whitelisting certain licenses)
+- Dependency licensing
+- References to licenses in `README`, `README.md`, etc.
+- Every single possible license (just the most popular ones)
+- Compliance (e.g., whitelisting certain licenses)
 
 If you're looking for dependency license checking and compliance, take a look at [LicenseFinder](https://github.com/pivotal/LicenseFinder).
 
 ## Huh? Why don't you look at X?
-
 Because reasons.
 
 ### Why not just look at the "license" field of [insert package manager here]?
-
 Because it's not legally binding. A license is a legal contract. You give up certain rights (e.g., the right to sue the author) in exchange for the right to use the software.
 
-Most popular licenses today *require* that the license itself be distributed along side the software. Simply putting the letters "MIT" or "GPL" in a configuration file doesn't really meet that requirement.
+Most popular licenses today _require_ that the license itself be distributed along side the software. Simply putting the letters "MIT" or "GPL" in a configuration file doesn't really meet that requirement.
 
 Not to mention, it doesn't tell you much about your rights as a user. Is it GPLv2? GPLv2 or later? Those files are designed to be read by computers (who can't enter into contracts), not humans (who can). It's great metadata, but that's about it.
 
 ### What about looking to see if the author said something in the readme?
-
 You could make an argument that, when linked or sufficiently identified, the terms of the license are incorporated by reference, or at least that the author's intent is there. There's a handful of reasons why this isn't ideal. For one, if you're using the MIT or BSD (ISC) license, along with a few others, there's templematic language, like the copyright notice, which would go unfilled.
 
 ### What about checking every single file for a copyright header?
-
 Because that's silly in the context of how software is developed today. You wouldn't put a copyright notice on each page of a book. Besides, it's a lot of work, as there's no standardized, cross-platform way to describe a project's license within a comment.
 
 Checking the actual text into version control is definitive, so that's what this project looks at.
 
 ## Bootstrapping a local development environment
-
 `script/bootstrap`
 
 ## Running tests
-
 `script/cibuild`
 
 ## Updating the licenses
-
 License data is pulled from `choosealicense.com`. To update the license data, simple run `script/vendor-licenses`.
 
 ## Roadmap
-
 See [proposed enhancements](https://github.com/benbalter/licensee/labels/enhancement).

data/bin/licensee

@@ -3,19 +3,18 @@ require_relative "../lib/licensee"
 
 path = ARGV[0] || Dir.pwd
 
-Licensee.package_manager_files = true
-project = Licensee::Project.new(path)
-license = project.matched_file
+project = Licensee::GitProject.new(path, detect_packages: true)
+file = project.matched_file
 
 if project.license_file
-  puts "License file: #{project.license_file.path}"
+  puts "License file: #{project.license_file.filename}"
   puts "Attribution: #{project.license_file.attribution}" if project.license_file.attribution
 end
 
-if license
-  puts "License: #{license.match ? license.match.meta['title'] : 'no license'}"
-  puts "Confidence: #{license.confidence}%"
-  puts "Method: #{license.matcher.class}"
+if file
+  puts "License: #{file.license ? file.license.meta['title'] : 'no license'}"
+  puts "Confidence: #{file.confidence}%"
+  puts "Method: #{file.matcher.class}"
 else
   puts "Unknown"
 end

data/lib/licensee.rb

@@ -1,25 +1,17 @@
-require 'uri'
-require 'yaml'
-require 'rugged'
-require 'levenshtein'
-
 require_relative "licensee/version"
 require_relative "licensee/content_helper"
 require_relative "licensee/license"
 require_relative "licensee/project"
 require_relative "licensee/project_file"
-require_relative "licensee/filesystem_repository"
-require_relative "licensee/matcher"
+
 require_relative "licensee/matchers/exact_matcher"
 require_relative "licensee/matchers/copyright_matcher"
-require_relative "licensee/matchers/git_matcher"
-require_relative "licensee/matchers/levenshtein_matcher"
+require_relative "licensee/matchers/dice_matcher"
 require_relative "licensee/matchers/package_matcher"
 require_relative "licensee/matchers/gemspec_matcher"
 require_relative "licensee/matchers/npm_bower_matcher"
 
 class Licensee
-
   # Over which percent is a match considered a match by default
   CONFIDENCE_THRESHOLD = 90
 
@@ -27,40 +19,24 @@ class Licensee
   DOMAIN = "http://choosealicense.com"
 
   class << self
-
-    attr_writer :confidence_threshold, :package_manager_files
+    attr_writer :confidence_threshold
 
     # Returns an array of Licensee::License instances
     def licenses(options={})
       Licensee::License.all(options)
     end
 
-    # Returns the license for a given git repo
+    # Returns the license for a given path
     def license(path)
-      Licensee::Project.new(path).license
-    end
-
-    # Array of matchers to use, in order of preference
-    # The order should be decending order of anticipated speed to match
-    def matchers
-      matchers = [
-        Licensee::CopyrightMatcher,
-        Licensee::ExactMatcher,
-        Licensee::GitMatcher,
-        Licensee::LevenshteinMatcher,
-        Licensee::GemspecMatcher,
-        Licensee::NpmBowerMatcher
-      ]
-      matchers.reject! { |m| m.package_manager? } unless package_manager_files?
-      matchers
+      begin
+        Licensee::GitProject.new(path).license
+      rescue Licensee::GitProject::InvalidRepository
+        Licensee::FSProject.new(path).license
+      end
     end
 
     def confidence_threshold
       @confidence_threshold ||= CONFIDENCE_THRESHOLD
     end
-
-    def package_manager_files?
-      @package_manager_files || false
-    end
   end
 end

data/lib/licensee/content_helper.rb

@@ -1,14 +1,13 @@
+require 'set'
+
 class Licensee
   module ContentHelper
-    def normalize_content(content)
+    def create_word_set(content)
       return unless content
-      content = content.downcase
-      content = content.gsub(/\A[[:space:]]+/, '')
-      content = content.gsub(/[[:space:]]+\z/, '')
-      content = content.gsub(/^#{CopyrightMatcher::REGEX}$/i, '')
-      content = content.gsub(/[[:space:]]+/, ' ')
-      content = content.gsub("\u0000", '') # Remove null byte which breaks Levenshtein
-      content.squeeze(' ').strip
+      content = content.dup
+      content.downcase!
+      content.gsub!(/^#{Matchers::Copyright::REGEX}$/i, '')
+      content.scan(/[\w']+/).to_set
     end
   end
 end

data/lib/licensee/license.rb

@@ -1,3 +1,6 @@
+require 'uri'
+require 'yaml'
+
 class Licensee
   class InvalidLicense < ArgumentError; end
   class License
@@ -39,10 +42,6 @@ class Licensee
 
     HIDDEN_LICENSES = %w[other no-license]
 
-    # Licenses that technically contain the license name or nickname
-    # But we are so short that GitMatcher may not catch if rewrapped
-    BODY_INCLUDES_WHITELIST = %w[mit]
-
     include Licensee::ContentHelper
 
     def initialize(key)
@@ -107,17 +106,8 @@ class Licensee
     alias_method :to_s, :body
     alias_method :text, :body
 
-    # License body with all whitespace replaced with a single space
-    def body_normalized
-      @body_normalized ||= normalize_content(body)
-    end
-
-    # Git-computed hash signature for the license file
-    def hashsig
-      @hashsig ||= Rugged::Blob::HashSignature.new(
-        body, Rugged::Blob::HashSignature::WHITESPACE_SMART) unless body.nil?
-    rescue Rugged::InvalidError
-      nil
+    def wordset
+      @wordset ||= create_word_set(body)
     end
 
     def inspect
@@ -132,20 +122,7 @@ class Licensee
       other != nil && key == other.key
     end
 
-    def body_includes_name?
-      return false if BODY_INCLUDES_WHITELIST.include?(key)
-      return @body_includes_name if defined? @body_includes_name
-      @body_includes_name = body_normalized.include?(name_without_version.downcase)
-    end
-
-    def body_includes_nickname?
-      return false if BODY_INCLUDES_WHITELIST.include?(key)
-      return @body_includes_nickname if defined? @body_includes_nickname
-      @body_includes_nickname = !!(nickname && body_normalized.include?(nickname.downcase))
-    end
-
     private
-
     def parts
       @parts ||= content.match(/\A(---\n.*\n---\n+)?(.*)/m).to_a if content
     end

data/lib/licensee/matchers/copyright_matcher.rb

@@ -1,24 +1,25 @@
 # encoding=utf-8
 class Licensee
-  class CopyrightMatcher < Matcher
+  module Matchers
+    class Copyright
+      REGEX = /\s*Copyright (©|\(c\)|\xC2\xA9)? ?(\d{4}|\[year\])(.*)?\s*/i
 
-    REGEX = /\s*Copyright (©|\(c\)|\xC2\xA9)? ?(\d{4}|\[year\])(.*)?\s*/i
+      def initialize(file)
+        @file = file
+      end
 
-    def match
-      # Note: must use content, and not content_normalized here
-      no_license if file.content.strip =~ /\A#{REGEX}\z/i
-    rescue
-      nil
-    end
-
-    def confidence
-      100
-    end
-
-    private
+      def match
+        # Note: must use content, and not content_normalized here
+        if @file.content.strip =~ /\A#{REGEX}\z/i
+          Licensee::License.find("no-license") 
+        end
+      rescue
+        nil
+      end
 
-    def no_license
-      @no_license ||= Licensee::License.find("no-license")
+      def confidence
+        100
+      end
     end
   end
 end

data/lib/licensee/matchers/dice_matcher.rb

@@ -0,0 +1,65 @@
+class Licensee
+  module Matchers
+    class Dice
+      def initialize(file)
+        @file = file
+      end
+
+      # Return the first potential license that is more similar
+      # than the confidence threshold
+      def match
+        return @match if defined? @match
+        matches = potential_licenses.map do |license|
+          if (sim = similarity(license)) >= Licensee.confidence_threshold
+            [license, sim]
+          end
+        end
+        matches.compact!
+        @match = if matches.empty?
+          nil
+        else
+          matches.max_by { |l, sim| sim }.first
+        end
+      end
+
+      # Sort all licenses, in decending order, by difference in
+      # length to the file
+      # Difference in lengths cannot exceed the file's length *
+      # the confidence threshold / 100
+      def potential_licenses
+        @potential_licenses ||= begin
+          licenses = Licensee.licenses(:hidden => true)
+          licenses = licenses.select do |license|
+            license.wordset && length_delta(license) <= max_delta
+          end
+          licenses.sort_by { |l| length_delta(l) }
+        end
+      end
+
+      # Calculate the difference between the file length and a given
+      # license's length
+      def length_delta(license)
+        (@file.wordset.size - license.wordset.size).abs
+      end
+
+      # Maximum possible difference between file length and license length
+      # for a license to be a potential license to be matched
+      def max_delta
+        @max_delta ||= (@file.wordset.size * (Licensee.confidence_threshold/100.0))
+      end
+
+      # Confidence that the matched license is a match
+      def confidence
+        @confidence ||= match ? similarity(match) : 0
+      end
+
+      private
+      # Calculate percent changed between file and potential license
+      def similarity(license)
+        overlap = (@file.wordset & license.wordset).size
+        total = @file.wordset.size + license.wordset.size
+        100.0 * (overlap * 2.0 / total)
+      end
+    end
+  end
+end

data/lib/licensee/matchers/exact_matcher.rb

@@ -1,11 +1,17 @@
 class Licensee
-  class ExactMatcher < Matcher
-    def match
-      Licensee.licenses(:hidden => true).find { |l| l.body_normalized == file.content_normalized }
-    end
+  module Matchers
+    class Exact
+      def initialize(file)
+        @file = file
+      end
+
+      def match
+        Licensee.licenses(:hidden => true).find { |l| l.wordset == @file.wordset }
+      end
 
-    def confidence
-      100
+      def confidence
+        100
+      end
     end
   end
 end