license_matcher 0.1.0.pre.alpha → 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (10) hide show
  1. checksums.yaml +4 -4
  2. data/Gemfile.lock +13 -1
  3. data/README.md +76 -5
  4. data/lib/license_matcher/native.bundle +0 -0
  5. data/lib/license_matcher/preprocess.rb +75 -0
  6. data/lib/license_matcher/rule_matcher.rb +601 -0
  7. data/lib/license_matcher/tf_ruby_matcher.rb +100 -0
  8. data/lib/license_matcher/url_matcher.rb +102 -0
  9. data/lib/license_matcher.rb +16 -2
  10. metadata +64 -4
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 3b4af3ec68010e357923af4f36968b826573648e
4
- data.tar.gz: c36ba2921d655acc1a073881bc5a8d901ae93d81
3
+ metadata.gz: 6b710a76802bc5254b2d0d9dd3c36376c613b7e5
4
+ data.tar.gz: fef8fe16e4028ed1c49710956dceb4ac13acee9b
5
5
  SHA512:
6
- metadata.gz: 5f0013742ef8d324c38a7fae607de3f7c7bff7bf71a5b0c9533eef7fe5736b57e1dc5fafe1ce5813b260a89e228f9c24bbb8d89920c5186bc601ccb9988580c3
7
- data.tar.gz: e325854f90a54dcebb6d9b5e6b4c5166399ccfa1815aa7e4e3f208141b082e09dadf498b081331ab823671fd255c65cdf25a941f6b942344feaf441b82fe0746
6
+ metadata.gz: ee54a1ae1b3258f9bc474a5a05c7221281be666976f6e1d547bb1673c4fa57507d0006a21b236d9ce02a55fa6e9e1ac9fe646a6c96d53c2d7255ab59d5e2c821
7
+ data.tar.gz: adb94097dea0e79e19ac4b2cf9285d42e190902f2d8f798e5eb7239d094e20ca12193efb7342ad9ee4065b2e5f2bc48ccf3cacc318419699d7a7f2b528fb6fab
data/Gemfile.lock CHANGED
@@ -1,8 +1,11 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- license_matcher (0.1.0)
4
+ license_matcher (0.1.0.pre.alpha)
5
5
  helix_runtime (~> 0.6.0)
6
+ narray (~> 0.6.1.2)
7
+ nokogiri (~> 1.8.0)
8
+ tf-idf-similarity (~> 0.1.6)
6
9
 
7
10
  GEM
8
11
  remote: https://rubygems.org/
@@ -13,6 +16,11 @@ GEM
13
16
  rake (>= 10.0)
14
17
  thor (~> 0.19.4)
15
18
  toml (~> 0.1.2)
19
+ mini_portile2 (2.2.0)
20
+ msgpack (1.1.0)
21
+ narray (0.6.1.2)
22
+ nokogiri (1.8.0)
23
+ mini_portile2 (~> 2.2.0)
16
24
  parslet (1.5.0)
17
25
  blankslate (~> 2.0)
18
26
  rake (10.5.0)
@@ -29,9 +37,12 @@ GEM
29
37
  diff-lcs (>= 1.2.0, < 2.0)
30
38
  rspec-support (~> 3.6.0)
31
39
  rspec-support (3.6.0)
40
+ tf-idf-similarity (0.1.6)
41
+ unicode_utils (~> 1.4)
32
42
  thor (0.19.4)
33
43
  toml (0.1.2)
34
44
  parslet (~> 1.5.0)
45
+ unicode_utils (1.4.0)
35
46
 
36
47
  PLATFORMS
37
48
  ruby
@@ -39,6 +50,7 @@ PLATFORMS
39
50
  DEPENDENCIES
40
51
  bundler (~> 1.15)
41
52
  license_matcher!
53
+ msgpack (~> 1.1.0)
42
54
  rake (~> 10.0)
43
55
  rspec (~> 3.4)
44
56
 
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # LicenseMatcher
2
2
 
3
- LicenseMatcher is a rubygem that can match a fulltext of Opensource License Text with SPDX id; So you dont have to guess is it **BSD** or **MIT** license, let the `LicenseMatcher` does the heavy lifting for you;
3
+ LicenseMatcher is a rubygem that matches a fulltext of Opensource License Text with the SPDX id; So you dont have to guess is it **BSD** or **MIT** license, let the `LicenseMatcher` does the heavy lifting for you;
4
4
 
5
5
 
6
6
  It uses [Fosslim](https://github.com/Fosslim/fosslim/) library underneath, which gives remarkable performance with lower memory cost than pure Ruby implementation;
@@ -34,16 +34,87 @@ run `bundle exec irb` on your commandline to fire up Ruby REPL;
34
34
  ```
35
35
  require 'license_matcher'
36
36
 
37
- # build index from SPDX data
38
- LicenseMatcher.build_index( "data/licenses", "data/index.msgpack")
37
+ # download pre-build index
38
+ curl -O https://github.com/Fosslim/license_matcher/blob/master/data/index.msgpack
39
+
40
+ # or build index from the SPDX data
41
+ LicenseMatcher::TFRustMatcher.build_index( "data/licenses", "data/index.msgpack")
39
42
 
40
43
  # match license text
41
44
  txt = File.read("fixtures/files/mit.txt");
42
- lm = LicenseMatcher.new("data/index.msgpack")
43
- lm.match(txt, 0.9)
45
+
46
+ lm = LicenseMatcher::TFRubyMatcher.new("data/index.msgpack")
47
+ lm.match_text(txt, 0.9)
48
+
49
+
50
+ ```
51
+
52
+
53
+ ## Matchers
54
+
55
+ It currently supports 4 different models:
56
+
57
+ * **UrlMatcher.match_url** - finds matching SPDX license by comparing URL with urls in the `licenses.json`
58
+
59
+ ```ruby
60
+ lm = LicenseMatcher::UrlMatcher.new
61
+ lm.match_url "https://opensource.org/licenses/AAL"
62
+
63
+ => "AAL"
64
+ ```
65
+
66
+ * **RuleMatcher.match_rule** - scans a text and returns the SPDX id, which rule matches longest substring in the license text
67
+
68
+ ```ruby
69
+ lm = LicenseMatcher::RuleMatcher.new
70
+ lm.match_rules "It is license under Apache 2.0 License."
71
+
72
+ => "Apache-2.0"
73
+ ```
74
+
75
+ * **TFRubyMatcher** - original Ruby implementation, uses TF/IDF and Cosine similarity;
44
76
 
45
77
  ```
78
+ lm = LicenseMatcher::TFRubyMatcher.new
46
79
 
80
+ txt = File.read "fixtures/files/mit.html"
81
+ clean_txt = lm.preprocess_html txt # NB! it may help to increase accuracy
82
+ lm.match_txt clean_txt
83
+ ```
84
+
85
+ * **TFRustMatcher** - uses simple Jaccard similarity;
86
+
87
+ ```
88
+ lm2 = LicenseMatcher::TFRustMatcher.new
89
+
90
+ txt = File.read "fixtures/files/mit.txt"
91
+ lm2.match_text txt
92
+ ```
93
+
94
+ ## Benchmarks
95
+
96
+ * initialization, Ruby version 1times, Rust version 1000x
97
+
98
+ ```
99
+ user system total real
100
+ TFRubyMatcher: 12.850000 0.180000 13.030000 ( 13.210955)
101
+ TFRustMatcher: 26.260000 9.400000 35.660000 ( 38.264632)
102
+ ```
103
+ * matching preprocessed short [MIT](https://raw.githubusercontent.com/Fosslim/license_matcher/master/data/spdx_licenses/plain/MIT) text 1000x times
104
+
105
+ ```
106
+ user system total real
107
+ TFRubyMatcher:102.410000 12.180000 114.590000 (116.308119)
108
+ TFRustMatcher: 7.170000 0.040000 7.210000 ( 7.266000)
109
+ ```
110
+
111
+ * matching preprocessed long [AGPL-3.0](https://raw.githubusercontent.com/Fosslim/license_matcher/master/data/spdx_licenses/plain/AGPL-3.0) text 1000x times
112
+
113
+ ```
114
+ user system total real
115
+ TFRubyMatcher:242.450000 21.960000 264.410000 (276.417704)
116
+ TFRustMatcher: 9.340000 0.070000 9.410000 ( 9.478597)
117
+ ```
47
118
 
48
119
  ## Development
49
120
 
data/lib/license_matcher/native.bundle CHANGED
Binary file
data/lib/license_matcher/preprocess.rb ADDED
@@ -0,0 +1,75 @@
1
+ require 'nokogiri'
2
+
3
+ module Preprocess
4
+ def preprocess_text(text)
5
+ text = safe_encode(text)
6
+
7
+ #remove markdown url tags
8
+ text = text.gsub(/\[.+?\]\(.+?\)/, ' ')
9
+
10
+ #remove spam words
11
+ text.gsub!(/\bTHE\b/i, '')
12
+
13
+ #remove some XML grabage
14
+ text = text.gsub(/\<\!\[CDATA.*?\]\]\>/, ' ').to_s
15
+ text = text.gsub(/\<\!--.+?--\>/, ' ').to_s
16
+ text = text.gsub(/<\!\[CDATA.+?\]>/, ' ').to_s
17
+
18
+ return text.to_s.strip.gsub(/\s+/, ' ')
19
+ end
20
+
21
+ def preprocess_html(html_text)
22
+ # if text is HTML doc, then
23
+ # extract text only from visible html tags
24
+ text = ""
25
+
26
+ html_doc = parse_html(html_text)
27
+ if html_doc
28
+ text = clean_html(html_doc)
29
+ else
30
+ p "match_html: failed to parse html document\n#{html_text}"
31
+ end
32
+
33
+ return text
34
+ end
35
+
36
+ def clean_html(html_doc)
37
+ body_text = ""
38
+ body_elements = html_doc.xpath(
39
+ '//p | //h1 | //h2 | //h3 | //h4 | //h5 | //h6 | //em | //strong | //b | //td | //pre
40
+ | //li[not(@id) and not(@class) and not(a)] | //section//section[@class="project-info"]
41
+ | //blockquote | //textarea'
42
+ ).to_a
43
+
44
+ #extract text from html tag and separate them by space
45
+ body_elements.each {|el| body_text += ' ' + el.text.to_s}
46
+
47
+ #REMOVE XML CDATA like opensource.org pages has
48
+ body_text = body_text.to_s.strip
49
+ body_text.gsub!(/\<\!\[CDATA.+?\]\]\>/i, ' ')
50
+
51
+ if body_text.empty?
52
+ p "match_html: document didnt pass noise filter, will use whole body content"
53
+ body_text = html_doc.xpath('//body').text.to_s.strip
54
+ end
55
+
56
+ return body_text
57
+ end
58
+
59
+ def parse_html(html_text)
60
+ begin
61
+ return Nokogiri.HTML(safe_encode(html_text))
62
+ rescue Exception => e
63
+ log.error "failed to parse html doc: \n #{html_text}"
64
+ return nil
65
+ end
66
+ end
67
+
68
+ def safe_encode(txt)
69
+ txt.to_s.encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
70
+ rescue
71
+ p "Failed to encode text:\n #{txt}i"
72
+ return ""
73
+ end
74
+
75
+ end
data/lib/license_matcher/rule_matcher.rb ADDED
@@ -0,0 +1,601 @@
1
+ require 'json'
2
+
3
+ module LicenseMatcher
4
+
5
+ class RuleMatcher
6
+ include Preprocess
7
+
8
+ attr_reader :licenses, :rules, :id_spdx_idx
9
+
10
+ DEFAULT_LICENSE_JSON = 'data/spdx_licenses/licenses.json'
11
+
12
+
13
+ def initialize(license_json_file = DEFAULT_LICENSE_JSON)
14
+
15
+ licenses_json_doc = read_json_file license_json_file
16
+ raise("Failed to read licenses.json") if licenses_json_doc.nil?
17
+
18
+ @rules = init_rules(licenses_json_doc)
19
+ @id_spdx_idx = init_id_idx(licenses_json_doc) #reverse index from downcased licenseID to case sensitive spdx id
20
+ true
21
+
22
+ end
23
+
24
+ def init_id_idx(licenses_json_doc)
25
+ idx = {}
26
+ licenses_json_doc.to_a.each do |spdx_item|
27
+ lic_id = spdx_item[:id].to_s.downcase
28
+ idx[lic_id] = spdx_item[:id]
29
+ end
30
+
31
+ idx
32
+ end
33
+
34
+ # finds matching regex rules in the text and sorts matches by length of match
35
+ # ps: not very efficient, but good enough to handle special cases;
36
+ # @args:
37
+ # text - string, a name of license,
38
+ # @returns:
39
+ # [[spdx_id, score, matching_rule, matching_length],...]
40
+ def match_rules(text, early_exit = false)
41
+ matches = []
42
+ text = preprocess_text(text)
43
+
44
+ #if text is already spdx_id, then shortcut matching
45
+ if @rules.has_key?(text.downcase)
46
+ return [[text.downcase, 1.0]]
47
+ end
48
+
49
+ text += ' ' # required to make wordborder matcher to work with 1word texts
50
+ @rules.each do |spdx_id, rules|
51
+ match_res = matches_any_rule?(rules, text)
52
+ unless match_res.nil?
53
+ matches << ([spdx_id, 1.0] + match_res)
54
+ break if early_exit == true
55
+ end
56
+ end
57
+
58
+ matches.sort do |a, b|
59
+ if (a.size == b.size and a.size == 4)
60
+ -1 * (a[3] <=> b[3])
61
+ else
62
+ 0
63
+ end
64
+ end
65
+ end
66
+
67
+ # if testable license text is in the ignore set return true
68
+ def ignore?(lic_text)
69
+ ignore_rules = get_ignore_rules
70
+ m = matches_any_rule?(ignore_rules, lic_text.to_s)
71
+ not m.nil?
72
+ end
73
+
74
+ def matches_any_rule?(rules, license_name)
75
+ res = nil
76
+ rules.each do |rule|
77
+ m = rule.match(license_name.to_s)
78
+ if m
79
+ res = [rule, m[0].size]
80
+ break
81
+ end
82
+ end
83
+
84
+ res
85
+ end
86
+
87
+
88
+ #-- helpers
89
+
90
+ def read_json_file(file_path)
91
+ JSON.parse(File.read(file_path), {symbolize_names: true})
92
+ rescue
93
+ log.info "Failed to read json file `#{file_path}`"
94
+ nil
95
+ end
96
+
97
+ # combines SPDX rules with custom handwritten rules
98
+ def init_rules(license_json_doc)
99
+ rules = {}
100
+ rules = build_rules_from_spdx_json(license_json_doc)
101
+
102
+ get_custom_rules.each do |spdx_id, custom_rules_array|
103
+ spdx_id = spdx_id.to_s.strip.downcase
104
+
105
+ if rules.has_key?(spdx_id)
106
+ rules[spdx_id].concat custom_rules_array
107
+ else
108
+ rules[spdx_id] = custom_rules_array
109
+ end
110
+ end
111
+
112
+ rules
113
+ end
114
+
115
+
116
+ # builds regex rules based on the LicenseJSON file
117
+ # rules are using urls, IDS, names and alternative names to build full string matching regexes
118
+ def build_rules_from_spdx_json(spdx_json)
119
+ spdx_rules = {}
120
+
121
+ sorted_spdx_json = spdx_json.sort_by {|x| x[:id]}
122
+ sorted_spdx_json.each do |spdx_item|
123
+ spdx_id = spdx_item[:id].to_s.downcase.strip
124
+ spdx_rules[spdx_id] = build_spdx_item_rules(spdx_item)
125
+ end
126
+
127
+ spdx_rules
128
+ end
129
+
130
+ def build_spdx_item_rules(spdx_item)
131
+ rules = []
132
+
133
+ #links are first, because it has highest confidence that they are talking about the license
134
+ spdx_item[:links].to_a.each do |link|
135
+ lic_url = link[:url].to_s.strip.gsub(/https?:\/\//i, '').gsub(/www\./, '').gsub(/\./, '\\.') #normalizes urls in the file
136
+
137
+ rules << Regexp.new("\\b[\\(]?https?:\/\/(www\.)?#{lic_url}[\/]?[\\)]?\\b".gsub(/\s+/, ''), Regexp::IGNORECASE)
138
+ end
139
+
140
+ #include also links to license texts
141
+ spdx_item[:text].to_a.each do |link|
142
+ lic_url = link[:url].to_s.strip.gsub(/https?:\/\//i, '').gsub(/www\./, '').gsub(/\./, '\\.') #normalizes urls in the file
143
+ rules << Regexp.new("\\b[\\(]?https?:\/\/(www\.)?#{lic_url}[\/]?[\\)]?\\b".gsub(/\s+/, ''), Regexp::IGNORECASE)
144
+ end
145
+
146
+
147
+ spdx_name = preprocess_text(spdx_item[:name])
148
+ spdx_name.gsub!(/\(.+?\)/, '') #remove SPDX ids in the license names
149
+ spdx_name.gsub!(/\./, '\\.') #mark version dots as not regex selector
150
+ spdx_name.gsub!(/[\*|\?|\+]/, '.') #replace regex selector with whatever mark ~> WTFPL name
151
+ spdx_name.gsub!(/\,/, '[\\,]?') #make comma optional
152
+ spdx_name.strip!
153
+ spdx_name.gsub!(/\s+/, '\\s\\+') #replace spaces with space selector
154
+
155
+ rules << Regexp.new("\\b#{spdx_name}\\b", Regexp::IGNORECASE)
156
+
157
+ #use spdx_id in full-text match if it's uniq and doest ambiquity like MIT, Fair, Glide
158
+ spdx_id = spdx_item[:id].to_s.strip.downcase
159
+ if spdx_id.match /[\d|-]|ware\z/
160
+ rules << Regexp.new("\\b[\\(]?#{spdx_id}[\\)]?\\b".gsub(/\s+/, '\\s').gsub(/\./, '\\.'), Regexp::IGNORECASE)
161
+ else
162
+ rules << Regexp.new("\\A[\\(]?#{spdx_id}[\\)]?\\b", Regexp::IGNORECASE)
163
+ end
164
+
165
+ spdx_item[:identifiers].to_a.each do |id|
166
+ rules << Regexp.new("\\b#{id[:identifier]}\\s".gsub(/\s+/, '\\s').gsub(/\./, '\\.'), Regexp::IGNORECASE)
167
+ end
168
+
169
+ spdx_item[:other_names].to_a.each do |alt|
170
+ rules << Regexp.new("\\b#{alt[:name]}\\b".gsub(/\s+/, '\\s'), Regexp::IGNORECASE)
171
+ end
172
+
173
+ rules
174
+ end
175
+
176
+
177
+ def get_ignore_rules
178
+ [
179
+ /\bProprietary\b/i, /\bOther\/Proprietary\b/i, /\ALICEN[C|S]E\.\w{2,8}\b/i,
180
+ /^LICEN[C|S]ING\.\w{2,8}\b/i, /^COPYING\.\w{2,8}/i,
181
+ /\ADFSG\s+APPROVED\b/i, /\ASee\slicense\sin\spackage\b/i,
182
+ /\ASee LICENSE\b/i,
183
+ /\AFree\s+for\s+non[-]?commercial\b/i, /\AFree\s+To\s+Use\b/i,
184
+ /\AFree\sFor\sHome\sUse\b/i, /\AFree\s+For\s+Educational\b/i,
185
+ /^Freely\s+Distributable\s*$/i, /^COPYRIGHT\s+\d{2,4}/i,
186
+ /^Copyright\s+\(c\)\s+\d{2,4}\b/i, /^COPYRIGHT\s*$/i, /^COPYRIGHT\.\w{2,8}\b/i,
187
+ /^\(c\)\s+\d{2,4}\d/,
188
+ /^LICENSE\s*$/i, /^FREE\s*$/i, /\ASee\sLicense\s*\b/i, /^TODO\s*$/i, /^FREEWARE\s*$/i,
189
+ /^All\srights\sreserved\s*$/i, /^COPYING\s*$/i, /^OTHER\s*$/i, /^NONE\s*$/i, /^DUAL\s*$/i,
190
+ /^KEEP\s+IT\s+REAL\s*\b/i, /\ABE\s+REAL\s*\z/i, /\APrivate\s*\z/i, /\ACommercial\s*\z/i,
191
+ /\ASee\s+LICENSE\s+file\b/i, /\ASee\sthe\sLICENSE\b/i, /\ALICEN[C|S]E\s*\z/i,
192
+ /^PUBLIC\s*$/i, /^see file LICENSE\s*$/i, /^__license__\s*$/i,
193
+ /\bLIEULA\b/i, /\AEULA\s*\z/i, /^qQuickLicen[c|s]e\b/i, /^For\sfun\b/i, /\AVarious\s*\z/i,
194
+ /^GNU\s*$/i, /^GNU[-|\s]?v3\s*$/i, /^OSI\s+Approved\s*$/i, /^OSI\s*$/i,
195
+ /\AOPEN\sSOURCE\sLICENSE\s?\z/i, /\AOPEN\s*\z/i, /\Aunknown\s*\z/i,
196
+ /^https?:\/\/github.com/i, /^https?:\/\/gitlab\.com/i
197
+ ]
198
+ end
199
+
200
+
201
+ def get_custom_rules
202
+ {
203
+ "AAL" => [/\bAAL\b/i, /\bAAL\s+License\b/i,
204
+ /\bAttribution\s+Assurance\s+License\b/i],
205
+ "AFL-1.1" => [/\bAFL[-|v]?1[^\.]\b/i, /\bafl[-|v]?1\.1\b/i],
206
+ "AFL-1.2" => [/\bAFL[-|v]?1\.2\b/i],
207
+ "AFL-2.0" => [/\bAFL[-|v]?2[^\.]\b/i, /\bAFL[-|v]?2\.0\b/i],
208
+ "AFL-2.1" => [/\bAFL[-|v]?2\.1\b/i],
209
+ "AFL-3.0" => [
210
+ /\bAFL[-|\s|\_]?v?3\.0\b/i, /\bAFL[-|\s|\_]?v?3/i,
211
+ /\AAcademic\s+Free\s+License\s*\z/i, /^AFL\s?\z/i,
212
+ /\bhttps?:\/\/opensource\.org\/licenses\/academic\.php\b/i,
213
+ /\AAcademic[-|\s]Free[-|\s]License[-]?\s*\z/i,
214
+ /\bAcademic.Free.License.\(AFL\)/i
215
+ ],
216
+ "AGPL-1.0" => [
217
+ /\bAGPL[-|v|_|\s]?1\.0\b/i,
218
+ /\bAGPL[-|v|_|\s]1\b/i , /\bAGPL[_|-|v]?2\b/i,
219
+ /\bAffero\s+General\s+Public\s+License\s+[v]?1\b/i,
220
+ /\bAGPL\s(?!(v|\d))/i #Matches only AGPL, but not AGPL v1, AGPL 1.0 etc
221
+ ],
222
+ "AGPL-3.0" => [
223
+ /\bAGPL[-|_|\s]?v?3\.0\b/i, /\bAGPL[-|\s|_]?v?3[\+]?\b/i,
224
+ /\bAPGLv?3[\+]?\b/i, #some packages has typos
225
+ /\bGNU\s+Affero\s+General\s+Public\s+License\s+[v]?3/i,
226
+ /\bAFFERO\sGNU\sPUBLIC\sLICENSE\sv3\b/i,
227
+ /\bGnu\sAffero\sPublic\sLicense\sv3+?\b/i,
228
+ /\bAffero\sGeneral\sPublic\sLicen[s|c]e[\,]?\sversion\s+3[\.0]?\b/i,
229
+ /\bAffero\sGeneral\sPublic\sLicense\sv?3\b/i,
230
+ /\bAGPL\sversion\s3[\.]?\b/i,
231
+ /\bGNU\sAGPL\sv?3[\.0]?\b/i,
232
+ /\bGNU\sAFFERO\sv?3\b/i,
233
+ /\bhttps?:\/\/gnu\.org\/licenses\/agpl\.html\b/i,
234
+ /\AAFFERO\sGENERAL\sPUBLIC\sLICENSE\s*\z/i,
235
+ /^AFFERO\s*\z/i
236
+ ],
237
+ "Aladdin" => [/\b[\(]?AFPL[\)]?\b/i, /\bAladdin\sFree\sPublic\sLicense\b/i],
238
+ "Amazon" => [/\bAmazon\sSoftware\sLicense\b/i],
239
+ "Apache-1.0" => [/\bAPACHE[-|_|\s]?v?1[^\.]/i, /\bAPACHE[-|\s]?v?1\.0\b/i],
240
+ "Apache-1.1" => [/\bAPACHE[-|_|\s]?v?1\.1\b/i],
241
+ "Apache-2.0" => [
242
+ /\bAPACHE\s+2\.0\b/i, /\bAPACHE[-|_|\s]?v?2\b/i,
243
+ /\bApache\sOpen\sSource\sLicense\s2\.0\b/i,
244
+ /\bAPACH[A|E]\s+Licen[c|s]e\s+[\(]?v?2\.0[\)]?\b/i,
245
+ /\bAPACHE\s+LICENSE\,?\s+VERSION\s+2\.0\b/i,
246
+ /\bApache\s+License\s+v?2\b/i,
247
+ /\bApache\s+Software\sLicense\b/i,
248
+ /\bApapche[-|\s|\_]?v?2\.0\b/i, /\bAL[-|\s|\_]2\.0\b/i,
249
+ /\bAPL\s+2\.0\b/i, /\bAPL[\.|-|v]?2\b/i, /\bASL\s+2\.0\b/i,
250
+ /\bASL[-|v|\s]?2\b/i, /\bALv2\b/i, /\bASF[-|\s]?2\.0\b/i,
251
+ /\AAPACHE\s*\z/i, /\AASL\s*\z/i, /\bASL\s+v?\.2\.0\b/i, /\AASF\s*\z/i,
252
+ /\AApache\s+license\s*\z/i,
253
+ ],
254
+ "APL-1.0" => [/\bapl[-|_|\s]?v?1\b/i, /\bAPL[-|_|\s]?v?1\.0\b/i, /^APL$/i],
255
+ "APSL-1.0" => [/\bAPSL[-|_|\s]?v?1\.0\b/i, /\bAPSL[-|_|\s]?v?1(?!\.)\b/i, /\AAPPLE\s+PUBLIC\s+SOURCE\s*\z/i],
256
+ "APSL-1.1" => [/\bAPSL[-|_|\s]?v?1\.1\b/i],
257
+ "APSL-1.2" => [/\bAPSL[-|_|\s]?v?1\.2\b/i],
258
+ "APSL-2.0" => [/\bAPSL[-|_|\s]?v?2\.0\b/i, /\bAPSL[-|_|\s]?v?2\b/i],
259
+
260
+ "Artistic-1.0-Perl" => [/\bArtistic[-|_|\s]?v?1\.0\-Perl\b/i, /\bPerlArtistic\b/i],
261
+ "Artistic-1.0" => [/\bartistic[-|_|\s]?v?1\.0(?!\-)\b/i, /\bartistic[-|_|\s]?v?1(?!\.)\b/i],
262
+ "Artistic-2.0" => [/\bARTISTIC[-|_|\s]?v?2\.0\b/i, /\bartistic[-|_|\s]?v?2\b/i,
263
+ /\bArtistic.2.0\b/i,
264
+ /\bARTISTIC\s+LICENSE\b/i, /\AARTISTIC\s*\z/i],
265
+ "Beerware" => [
266
+ /\bBEERWARE\b/i, /\bBEER\s+LICEN[c|s]E\b/i,
267
+ /\bBEER[-|\s]WARE\b/i, /^BEER\b/i,
268
+ /\bBuy\ssnare\sa\sbeer\b/i,
269
+ /\bFree\sas\sin\sbeer\b/i
270
+ ],
271
+ 'BitTorrent-1.1' => [/\bBitTorrent\sOpen\sSource\sLicense\b/i],
272
+ "0BSD" => [/\A0BSD\s*\z/i],
273
+ "BSD-2-CLAUSE" => [
274
+ /\bBSD[-|_|\s]?v?2\b/i, /^FREEBSD\b/i, /^OPENBSD\b/i,
275
+ /\bBSDLv2\b/i
276
+ ],
277
+ "BSD-3-CLAUSE" => [/\bBSD[-|_|\s]?v?3\b/i, /\bBSD[-|\s]3[-\s]CLAUSE\b/i,
278
+ /\bBDS[-|_|\s]3[-|\s]CLAUSE\b/i,
279
+ /\bthree-clause\sBSD\slicen[s|c]e\b/i,
280
+ /\ABDS\s*\z/i, /^various\/BSDish\s*$/],
281
+ "BSD-4-CLAUSE" => [
282
+ /\bBSD[-|_|\s]?v?4/i, /\ABSD\s*\z/i, /\ABSD\s+LI[s|c]EN[S|C]E\s*\z/i,
283
+ /\bBSD-4-CLAUSE\b/i,
284
+ /\bhttps?:\/\/en\.wikipedia\.org\/wiki\/BSD_licenses\b/i
285
+ ],
286
+ "BSL-1.0" => [
287
+ /\bBSL[-|_|\s]?v?1\.0\b/i, /\bbsl[-|_|\s]?v?1\b/i, /^BOOST\b/i,
288
+ /\bBOOST\s+SOFTWARE\s+LICENSE\b/i,
289
+ /\bBoost\sLicense\s1\.0\b/i
290
+ ],
291
+ "CC0-1.0" => [
292
+ /\bCC0[-|_|\s]?v?1\.0\b/i, /\bCC0[-|_|\s]?v?1\b/i,
293
+ /\bCC[-|\s]?[0|o]\b/i, /\bCreative\s+Commons\s+0\b/i,
294
+ /\bhttps?:\/\/creativecommons\.org\/publicdomain\/zero\/1\.0[\/]?\b/i,
295
+ /\bcc[-|\_]zero\b/i
296
+ ],
297
+ "CC-BY-1.0" => [/\bCC.BY.v?1\.0\b/i, /\bCC.BY.v?1\b/i, /^CC[-|_|\s]?BY$/i],
298
+ "CC-BY-2.0" => [/\bCC.BY.v?2\.0\b/i, /\bCC.BY.v?2(?!\.)\b/i],
299
+ "CC-BY-2.5" => [/\bCC.BY.v?2\.5\b/i],
300
+ "CC-BY-3.0" => [
301
+ /\bCC.BY.v?3\.0\b/i, /\b[\(]?CC.BY[\)]?.v?3\b/i,
302
+ /\bCreative\sCommons\sBY\s3\.0\b/i,
303
+ /\bhttps?:\/\/.+\/licenses\/by\/3\.0[\/]?/i
304
+ ],
305
+ "CC-BY-4.0" => [
306
+ /^CC[-|\s]?BY[-|\s]?v?4\.0$/i, /\bCC.BY.v?4\b/i, /\bCC.BY.4\.0\b/i,
307
+ /\bCREATIVE\s+COMMONS\s+ATTRIBUTION\s+[v]?4\.0\b/i,
308
+ /\ACREATIVE\s+COMMONS\s+ATTRIBUTION\s*\z\b/i
309
+ ],
310
+ "CC-BY-SA-1.0" => [/\bCC[-|\s]BY.SA.v?1\.0\b/i, /\bCC[-|\s]BY.SA.v?1\b/i],
311
+ "CC-BY-SA-2.0" => [/\bCC[-|\s]BY.SA.v?2\.0\b/i, /\bCC[-|\s]BY.SA.v?2(?!\.)\b/i],
312
+ "CC-BY-SA-2.5" => [/\bCC[-|\s]BY.SA.v?2\.5\b/i],
313
+ "CC-BY-SA-3.0" => [/\bCC[-|\s]BY.SA.v?3\.0\b/i, /\bCC[-|\s]BY.SA.v?3\b/i,
314
+ /\bCC3\.0[-|_|\s]BY.SA\b/i,
315
+ /\bhttps?:\/\/(www\.)?.+\/by.sa\/3\.0[\/]?/i],
316
+ "CC-BY-SA-4.0" => [
317
+ /CC[-|\s]BY.SA.v?4\.0$/i, /\bCC[-|\s]BY.SA.v?4\b/i,
318
+ /CCSA-4\.0/i
319
+ ],
320
+ "CC-BY-NC-1.0" => [/\bCC[-|\s]BY.NC[-|\s]?v?1\.0\b/i, /\bCC[-|\s]BY.NC[-|\s]?v?1\b/i],
321
+ "CC-BY-NC-2.0" => [/\bCC[-|\s]BY.NC[-|\s]?v?2\.0\b/i],
322
+ "CC-BY-NC-2.5" => [/\bCC[-|\s]BY.NC[-|\s]?v?2\.5\b/i],
323
+ "CC-BY-NC-3.0" => [/\bCC[-|\s]BY.NC[-|\s]?v?3\.0\b/i, /\bCC.BY.NC[-|\s]?v?3\b/i,
324
+ /\bCreative\s+Commons\s+Non[-]?Commercial[,]?\s+3\.0\b/i],
325
+ "CC-BY-NC-4.0" => [
326
+ /\bCC[-|\s]BY.NC[-|\s|_]?v?4\.0\b/i, /\bCC.BY.NC[-|\s|_]?v?4\b/i,
327
+ /\bhttps?:\/\/creativecommons\.org\/licenses\/by-nc\/3\.0[\/]?\b/i
328
+ ],
329
+ "CC-BY-NC-SA-1.0" => [ /\bCC[-|\s+]BY.NC.SA[-|\s+]v?1\.0\b/i,
330
+ /\bCC[-|\s+]BY.NC.SA[-|\s+]v?1\b/i
331
+ ],
332
+ "CC-BY-NC-SA-2.0" => [/\bCC[-|\s]?BY.NC.SA[-|\s]?v?2\.0\b/i],
333
+ "CC-BY-NC-SA-2.5" => [/\bCC[-|\s]?BY.NC.SA[-|\s]?v?2\.5\b/i],
334
+ "CC-BY-NC-SA-3.0" => [
335
+ /\bCC[-|\s]?BY.NC.SA[-|\s]?v?3\.0\b/i,
336
+ /\bCC[-|\s]?BY.NC.SA[-|\s]?v?3(?!\.)\b/i,
337
+ /\bBY[-|\s]NC[-|\s]SA\sv?3\.0\b/i,
338
+ /\bhttp:\/\/creativecommons.org\/licenses\/by-nc-sa\/3.0\/us[\/]?\b/i
339
+ ],
340
+ "CC-BY-NC-SA-4.0" => [/\bCC[-|\s]?BY.NC.SA[-|\s]?v?4\.0\b/i,
341
+ /\bCC[-|_|\s]BY.NC.SA[-|\s]?v?4(?!\.)\b/i,
342
+ /\bBY.NC.SA[-|\s|\_]v?4\.0\b/i],
343
+
344
+ "CC-BY-ND-1.0" => [/\bCC[-|\s]BY.ND[-|\s]?v?1\.0\b/i],
345
+ "CC-BY-ND-2.0" => [/\bCC[-|\s]BY.ND[-|\s]?v?2\.0\b/i],
346
+ "CC-BY-ND-2.5" => [/\bCC[-|\s]BY.ND[-|\s]?v?2\.5\b/i],
347
+ "CC-BY-ND-3.0" => [/\bCC[-|\s]BY.ND[-|\s]?v?3\.0\b/i],
348
+ "CC-BY-ND-4.0" => [
349
+ /\bCC[-|\s]BY.ND[-|\s]?v?4\.0\b/i,
350
+ /\bCC\sBY.NC.ND\s4\.0/i
351
+ ],
352
+
353
+ "CC-BY-NC-ND-3.0" => [/\bCC.BY.NC.ND.3\.0\b/i],
354
+ "CC-BY-NC-ND-4.0" => [/\bCC.BY.NC.ND.4\.0\b/i],
355
+ "CDDL-1.0" => [/\bCDDL[-|_|\s]?v?1\.0\b/i, /\bCDDL[-|_|\s]?v?1\b/i, /^CDDL$/i,
356
+ /\bCDDL\s+LICEN[C|S]E\b/i,
357
+ /\bCOMMON\sDEVELOPMENT\sAND\sDISTRIBUTION\sLICENSE\b/i
358
+ ],
359
+ "CECILL-B" => [/\bCECILL[-|_|\s]?B\b/i],
360
+ "CECILL-C" => [/\bCECILL[-|_|\s]?C\b/i],
361
+ "CECILL-1.0" => [
362
+ /\bCECILL[-|\s|_]?v?1\.0\b/i, /\bCECILL[-|\s|_]?v?1\b/i,
363
+ /\ACECILL\s?\z/i, /\bCECILL\s+v?1\.2\b/i,
364
+ /^http:\/\/www\.cecill\.info\/licences\/Licence_CeCILL-C_V1-en.html$/i,
365
+ /\bhttp:\/\/www\.cecill\.info\b/i
366
+ ],
367
+ "CECILL-2.1" => [
368
+ /\bCECILL[-|_|\s]?2\.1\b/i, /\bCECILL[\s|_|-]?v?2\b/i,
369
+ /\bCECILL\sVERSION\s2\.1\b/i
370
+ ],
371
+ "CPL-1.0" => [
372
+ /\bCPL[-|\s|_]?v?1\.0\b/i, /\bCPL[-|\s|_]?v?1\b/i,
373
+ /\bCommon\s+Public\s+License\b/i, /\ACPL\s*\z/i
374
+ ],
375
+ "CPAL-1.0" => [
376
+ /\bCommon\sPublic\sAttribution\sLicense\s1\.0\b/i,
377
+ /[\(]?\bCPAL\b[\)]?/i
378
+ ],
379
+ "CUSTOM" => [ /\bCUSTOM\s+LICENSE\b/i ],
380
+ "DBAD" => [
381
+ /\bDONT\sBE\sA\sDICK\b/i, /\ADBAD\s*\z/i,
382
+ /\bdbad[-|\s|\_]license\b/i, /\ADBAD-1\s*\z/i,
383
+ /\ADBAP\b/i,
384
+ /\bhttps?:\/\/www\.dbad-license\.org[\/]?\b/i
385
+ ],
386
+ "D-FSL-1.0" => [
387
+ /\bD-?FSL[-|_|\s]?v?1\.0\b/i, /\bD-?FSL[-|\s|_]?v?1\b/,
388
+ /\bGerman\sFREE\sSOFTWARE\b/i,
389
+ /\bDeutsche\sFreie\sSoftware\sLizenz\b/i
390
+ ],
391
+ "ECL-1.0" => [ /\bECL[-|\s|_]?v?1\.0\b/i, /\bECL[-|\s|_]?v?1\b/i ],
392
+ "ECL-2.0" => [
393
+ /\bECL[-|\s|_]?v?2\.0\b/i, /\bECL[-|\s|_]?v?2\b/i,
394
+ /\bEDUCATIONAL\s+COMMUNITY\s+LICENSE[,]?\sVERSION\s2\.0\b/i
395
+ ],
396
+ "EFL-1.0" => [/\bEFL[-|\s|_]?v?1\.0\b/i, /\bEFL[-|\s|_]?v?1\b/i ],
397
+ "EFL-2.0" => [
398
+ /\bEFL[-|\s|_]?v?2\.0\b/i, /\bEFL[-|\s|_]?v?2\b/i,
399
+ /\bEiffel\sForum\sLicense,?\sversion\s2/i,
400
+ /\bEiffel\sForum\sLicense\s2(?!\.)\b/i,
401
+ /\bEiffel\sForum\sLicense\b/i
402
+ ],
403
+ "EPL-1.0" => [
404
+ /\bEPL[-|\s|_]?v?1\.0\b/i, /\bEPL[-|\s|_]?v?1\b/i,
405
+ /\bECLIPSE\s+PUBLIC\s+LICENSE\s+[v]?1\.0\b/i,
406
+ /\bECLIPSE\s+PUBLIC\s+LICENSE\b/i,
407
+ /^ECLIPSE$/i, /\AEPL\s*\z/
408
+ ],
409
+ "ESA-1.0" => [
410
+ /\bESCL\s+[-|_]?\sType\s?1\b/,
411
+ /\bESA\sSOFTWARE\sCommunity\sLICENSE.+TYPE\s?1\b/i
412
+ ],
413
+ "EUPL-1.0" => [/\b[\(]?EUPL[-|\s]?v?1\.0[\)]?\b/i],
414
+ "EUPL-1.1" => [
415
+ /\b[\(]?EUPL[-|\s]?v?1\.1[\)]?\b/i,
416
+ /\bEUROPEAN\s+UNION\s+PUBLIC\s+LICENSE\s+1\.1\b/i,
417
+ /\bEuropean\sUnion\sPublic\sLicense\b/i,
418
+ /\bEUPL\s+V?\.?1\.1\b/i, /\AEUPL\s*\z/i
419
+ ],
420
+ "Fair" => [ /\bFAIR\s+LICENSE\b/i, /\AFair\s*\z/i],
421
+ "FreeType" => [ /\bFreeType\s+LICENSE\b/i],
422
+ "GFDL-1.0" => [
423
+ /\bGNU\sFree\sDocumentation\sLicense\b/i,
424
+ /\b[\(]?FDL[\)]?\b/
425
+ ],
426
+ "GPL-1.0" => [
427
+ /\bGPL[-|\s|_]?v?1\.0\b/i, /\bGPL[-|\s|_]?v?1\b/i,
428
+ /\bGNU\sPUBLIC\sLICEN[S|C]E\sv?1\b/i
429
+ ],
430
+ "GPL-2.0" => [
431
+ /\bGPL[-|\s|_]?v?2\.0/i, /\bGPL[-|\s|_]?v?2\b/i, /\bGPL\s+[v]?2\b/i,
432
+ /\bGNU\s+PUBLIC\s+LICENSE\s+v?2\.0\b/i,
433
+ /\bGNU\s+PUBLIC\s+License\sV?2\b/i,
434
+ /\bGNU\spublic\slicense\sversion\s2\b/i,
435
+ /\bGNU\sGeneral\sPublic\sLicense\sv?2\.0\b/i,
436
+ /\bGNU\sPublic\sLicense\s>=2\b/i,
437
+ /\bGNU\s+GPL\s+v2\b/i, /^GNUv?2\b/i, /^GLPv2\b/,
438
+ /\bWhatever\slicense\sPlone\sis\b/i
439
+ ],
440
+ "GPL-3.0" => [
441
+ /\bGNU\s+GENERAL\s+PUBLIC\s+License\s+[v]?3\b/i,
442
+ /\bGNU\s+General\s+Public\s+License[\,]?\sVersion\s3[\.0]?\b/i,
443
+ /\bGNU\sPublic\sLicense\sv?3\.0\b/i,
444
+ /\bGNU\s+PUBLIC\s+LICENSE\s+v?3\b/i,
445
+ /\bGnu\sPublic\sLicense\sversion\s3\b/i,
446
+ /\bGNU\sGeneral\sPublic\sLicense\sversion\s?3\b/i,
447
+ /\bGPL[-|\s|_]?v?3\.0\b/i, /\bGPL[-|\s|_]?v?[\.]?3\b/i, /\bGPL\s+3\b/i,
448
+ /\bGNU\s+PUBLIC\s+v3\+?\b/i,
449
+ /\bGNUGPL[-|\s|\_]?v?3\b/i, /\bGNU\s+PL\s+[v]?3\b/i,
450
+ /\bGLPv3\b/i, /\bGNU3\b/i, /GPvL3/i, /\bGNU\sGLP\sv?3\b/i,
451
+ /\AGNU\sGENERAL\sPUBLIC\sLICENSE\s*\z/i, /\A[\(]?GPL[\)]?\s*\z/i
452
+ ],
453
+
454
+ "IDPL-1.0" => [
455
+ /\bIDPL[-|\s|\_]?v?1\.0\b/,
456
+ /\bhttps?:\/\/www\.firebirdsql\.org\/index\.php\?op=doc\&id=idpl\b/i
457
+ ],
458
+ "IPL-1.0" => [/\bIBM\sOpen\sSource\sLicense\b/i, /\bIBM\sPublic\sLicen[s|c]e\b/i],
459
+ "ISC" => [/\bISC\s+LICENSE\b/i, /\b[\(]?ISCL[\)]?\b/i, /\bISC\b/i,
460
+ /\AICS\s*\z/i],
461
+ "JSON" => [/\bJSON\s+LICENSE\b/i],
462
+ "KINDLY" => [/\bKINDLY\s+License\b/i],
463
+ "LGPL-2.0" => [
464
+ /\bLGPL[-|\s|_]?v?2\.0\b/i, /\bLGPL[-|\s|_]?v?2(?!\.)\b/i,
465
+ /\bLesser\sGeneral\sPublic\sLicense\sv?2(?!\.)\b/i,
466
+ /\bLPGL[-|\s|\_]?v?2(?!\.)\b/i
467
+ ],
468
+ "LGPL-2.1" => [
469
+ /\bLGPL[-|\s|_]?v?2\.1\b/i,
470
+ /\bLesser\sGeneral\sPublic\sLicense\s+\(LGPL\)\s+Version\s+2\.1\b/i,
471
+ /\bLESSER\sGENERAL\sPUBLIC\sLICENSE[\,]?\sVersion\s2\.1[\,]?\b/i,
472
+ /\bLESSER\sGENERAL\sPUBLIC\sLICENSE[\,]?\sv?2\.1\b/i
473
+ ],
474
+ "LGPL-3.0" => [/\bLGPL[-|\s|_]?v?3\.0\b/i, /\bLGPL[-|\s|_]?v?3[\+]?\b/i,
475
+ /\bLGLP[\s|-|v]?3\.0\b/i, /^LPLv3\s*$/, /\bLPGL[-|\s|_]?v?3[\+]?\b/i,
476
+ /\bLESSER\s+GENERAL\s+PUBLIC\s+License\s+[v]?3\b/i,
477
+ /\bLesser\sGeneral\sPublic\sLicense\sv?\.?\s+3\.0\b/i,
478
+ /\bhttps?:\/\/www\.gnu\.org\/copyleft\/lesser.html\b/i,
479
+ /\bLESSER\sGENERAL\sPUBLIC\sLICENSE\sVersion\s3\b/i,
480
+ /\bLesser\sGeneral\sPublic\sLicense[\,]?\sversion\s3\.0\b/i,
481
+ /\bLESSER\sGENERAL\sPUBLIC\sLICENSE.+?version\s?3/i,
482
+ /\A[\(]?LGPL[\)]?\s*\z/i
483
+ ],
484
+ "MirOS" => [/\bMirOS\b/i],
485
+ "MIT" => [
486
+ /\bMIT\s+LICEN[S|C]E\b/i, /\AMITL?\s*\z/i, /\bEXPAT\b/i,
487
+ /\bMIT[-|\_]LICENSE\.\w{2,8}\b/i, /^MTI\b/i,
488
+ /\bMIT[-|\s|\_]?v?2\.0\b/i, /\AM\.I\.T[\.]?\s*\z/,
489
+ /\bMassachusetts-Institute-of-Technology-License/i
490
+ ],
491
+ "MITNFA" => [/\bMIT\s\+no\-false\-attribs\slicense\b/i],
492
+ "MPL-1.0" => [
493
+ /\bMPL[-|\s|\_]?v?1\.0\b/i, /\bMPL[-|\s|\_]?v?1(?!\.)\b/i,
494
+ /\bMozilla\sPublic\sLicense\sv?1\.0\b/i,
495
+ ],
496
+ "MPL-1.1" => [
497
+ /\bMozilla.Public.License\s+v?1\.1\b/i,
498
+ /\bMPL[-|\s|\_]?v?1\.1\b/i,
499
+ ],
500
+ "MPL-2.0" => [
501
+ /\bMPL[-|\s|\_]?v?2\.0\b/i, /\bMPL[-|\s|\_]?v?2\b/i,
502
+ /\bMOZILLA\s+PUBLIC\s+LICENSE\s+2\.0\b/i,
503
+ /\bMozilla\sPublic\sLicense[\,]?\s+v?[\.]?\s*2\.0\b/i,
504
+ /\bMOZILLA\s+PUBLIC\s+LICENSE[,]?\s+version\s+2\.0\b/i,
505
+ /\bMozilla\s+v?2\.0\b/i,
506
+ /\b[\(]?MPL\s+2\.0[\)]?\b/, /\bMPL\b/i,
507
+ /\AMozilla\sPublic\sLicense\s*\z/i
508
+ ],
509
+ "MS-PL" => [/\bMS-?PL\b/i],
510
+ "MS-RL" => [/\bMS-?RL\b/i, /\bMSR\-LA\b/i],
511
+ "ms_dotnet" => [/\bMICROSOFT\sSOFTWARE\sLICENSE\sTERMS\b/i],
512
+ "NASA-1.3" => [/\bNASA[-|\_|\s]?v?1\.3\b/i,
513
+ /\bNASA\sOpen\sSource\sAgreement\sversion\s1\.3\b/i],
514
+ "NCSA" => [/\bNCSA\s+License\b/i, /\bIllinois\/NCSA\sOpen\sSource\b/i, /\bNCSA\b/i ],
515
+ "NGPL" => [/\bNGPL\b/i],
516
+ "NOKIA" => [/\bNokia\sOpen\sSource\sLicense\b/i],
517
+ "NPL-1.1" => [/\bNetscape\sPublic\sLicense\b/i, /\b[(]?NPL[\)]?\b/i],
518
+
519
+ "NPOSL-3.0" => [/\bNPOSL[-|\s|\_]?v?3\.0\b/i, /\bNPOSL[-|\s|\_]?v?3\b/],
520
+ "OFL-1.0" => [/\bOFL[-|\s|\_]?v?1\.0\b/i, /\bOFL[-|\s|\_]?v?1(?!\.)\b/i,
521
+ /\bSIL\s+OFL\s+1\.0\b/i, /\ASIL\sOFL\s*\z/i ],
522
+ "OFL-1.1" => [
523
+ /\bOFL[-|\s|\_]?v?1\.1\b/i, /\bSIL\s+OFL\s+1\.1\b/i,
524
+ /\bSIL\sOpen\sFont\sLicense\b/i, /\bSIL\sOFL\s1\.1\b/i,
525
+ /\bOpen\sFont\sLicense\b/i ],
526
+
527
+ "OSL-1.0" => [/\bOSL[-|\s|\_]?v?1\.0\b/i, /\b\OSL[-|\s|\_]?v?1(?!\.)\b/i],
528
+ "OSL-2.0" => [/\bOSL[-|\s|\_]?v?2\.0\b/i, /\bOSL[-|\s|\_]?v?2(?!\.)\b/i],
529
+ "OSL-2.1" => [/\bOSL[-|\s|\_]?v?2\.1\b/i],
530
+ "OSL-3.0" => [
531
+ /\bOSL[-|\s|\_]?v?3\.0\b/i, /\bOSL[-|\s|\_]?v?3(?!\.)\b/i,
532
+ /\bOpen\sSoftware\sLicen[c|s]e\sv?3\.0\b/i,
533
+ /\bOSL[-|\s|\_]?v?\.?3\.[0|O]\b/i,
534
+ /\bOpen\sSoftware\sLicense\sversion\s3\.0\b/i,
535
+ /\AOSL\s*\z/i, /\bOpen-Software-License/i,
536
+ /\b[\(]?OSL[\)]?\s+v\s+3\.0\b/i
537
+ ],
538
+
539
+ "PHP-3.0" => [/^PHP\s?\z/i, /\bPHP\sLicense\s3\.0\b/i, /\APHP[-|\s]LICEN[S|C]E\s*\z/i],
540
+ "PHP-3.01" => [/\bPHP\sLicense\sversion\s3\.0\d\b/i],
541
+ "PIL" => [/\bStandard\sPIL\sLicense\b/i, /\APIL\s*\z/i],
542
+ "PostgreSQL" => [/\bPostgreSQL\b/i],
543
+ "Public Domain" => [/\bPublic\s+Domain\b/i],
544
+ "Python-2.0" => [
545
+ /\bPython[-|\s|\_]?v?2\.0\b/i, /\bPython[-|\s|\_]?v?2(?!\.)\b/i,
546
+ /\bPSF[-|\s|\_]?v?2\b/i, /\bPSFL\b/i, /\bPSF\b/i,
547
+ /\bPython\s+Software\s+Foundation\b/i,
548
+ /\APython\b/i, /\bPSL\b/i, /\bSAME\sAS\spython2\.3\b/i,
549
+ /\bhttps?:\/\/www\.opensource\.org\/licenses\/PythonSoftFoundation\.php\b/i,
550
+ /\bhttps?:\/\/opensource\.org\/licenses\/PythonSoftFoundation\.php\b/i
551
+ ],
552
+ "Repoze" => [/\bRepoze\sPublic\sLicense\b/i],
553
+ "RPL-1.1" => [/\bRPL[-|\s|_]?v?1\.1\b/i, /\bRPL[-|\s|_]?v?1(?!\.)\b/i],
554
+ "RPL-1.5" => [
555
+ /\bRPL[-|\s|_]?v?1\.5\b/i, /\ARPL\s*\z/i,
556
+ /\bhttps?:\/\/www\.opensource\.org\/licenses\/rpl\.php\b/i
557
+ ],
558
+ "Ruby" => [/\bRUBY\sLICEN[S|C]E\b/i, /\ARUBY\b/i, /\bRUBY\'s\b/i],
559
+ "QPL-1.0" => [/\bQPL[-|\s|_]?v?1\.0\b/i,
560
+ /\bQT\sPublic\sLicen[c|s]e\b/i,
561
+ /\bPyQ\sGeneral\sLicense\b/i],
562
+ "Sleepycat" => [/\bSleepyCat\b/i],
563
+ "SPL-1.0" => [
564
+ /\bSPL[-|\_|\s]?v?1\.0\b/i, /\bSun\sPublic\sLicense\b/i
565
+ ],
566
+ "W3C" => [/\bW3C\b/i],
567
+ "OpenSSL" => [/\bOPENSSL\b/i],
568
+ "Unicode-TOU" => [/\AUnicode-TOU[\s|\/|-]/i],
569
+ "UPL-1.0" => [/\bUniversal\sPermissive\sLicense\b/i],
570
+ "Unlicense" => [
571
+ /\bUNLI[C|S]EN[S|C]E\b/i, /\AUnlicen[s|c]ed\s*\z/i, /^go\sfor\sit\b/i,
572
+ /^Undecided\b/i,
573
+ /\bNO\s+LICEN[C|S]E\b/i, /\bNON[\s|-|\_]?LICENSE\b/i
574
+ ],
575
+ "Whiskeyware" => [/\bWH?ISKEY[-|\s|\_]?WARE\b/i],
576
+ "WTFPL" => [
577
+ /\bWTF[P|G]?L\b/i, /\bWTFPL[-|v]?2\b/i, /^WTF\b/i, /\AWTFP\s*\z/i,
578
+ /\bDo\s+whatever\s+you\s+want\b/i, /\bDWTFYW\b/i, /\AWTPFL\s*\z/i,
579
+ /\bDo\s+What\s+the\s+Fuck\s+You\s+Want\b/i, /\ADWTFYWT\s*\z/i,
580
+ /\ADo\sWHATEVER\b/i, /\ADWYW\b/i, /\bDWTFYWTP\b/i,
581
+ /\ADWHTFYWTPL\s*\z/i, /\AWhatever\s*\z/i,
582
+ /\bDO\s(THE\s)?FUCK\sWHAT\sYOU\sWANT\b/i
583
+ ],
584
+ "WXwindows" => [/\bwxWINDOWS\s+LIBRARY\sLICEN[C|S]E\b/i, /\AWXwindows\s*\z/i],
585
+ "X11" => [/\bX11\b/i],
586
+ "Zend-2.0" => [/\bZend\sFramework\b/i],
587
+ "ZPL-1.1" => [/\bZPL[-|\s|\_]?v?1\.1\b/i, /\bZPL[-|\s|\_]?v?1(?!\.)\b/i,
588
+ /\bZPL[-|\s|\_]?1\.0\b/i],
589
+ "ZPL-2.1" => [
590
+ /\bZPL[-|\s|\/|_|]?v?2\.1\b/i, /\bZPL[-|\s|_]?v?2(?!\.)\b/i,
591
+ /\bZPL\s+2\.\d\b/i, /\bZOPE\s+PUBLIC\s+LICENSE\b/i,
592
+ /\bZPL\s?$/i
593
+ ],
594
+ "zlib-acknowledgement" => [/\bZLIB[\/|-|\s]LIBPNG\b/i],
595
+ "ZLIB" => [/\bZLIB(?!\-|\/)\b/i]
596
+
597
+ }
598
+ end
599
+
600
+ end
601
+ end
data/lib/license_matcher/tf_ruby_matcher.rb ADDED
@@ -0,0 +1,100 @@
1
+ require 'narray'
2
+ require 'tf-idf-similarity'
3
+ require 'msgpack'
4
+
5
+ module LicenseMatcher
6
+
7
+ class TFRubyMatcher
8
+ include Preprocess
9
+
10
+ attr_reader :corpus, :model, :spdx_ids
11
+
12
+ DEFAULT_INDEX_PATH = 'data/index.msgpack'
13
+ DEFAULT_MIN_CONFIDENCE = 0.9
14
+ A_DOC_ROW = 3 # a array index to find the rows of indexed documents
15
+
16
+ def initialize(index_path = DEFAULT_INDEX_PATH)
17
+ spdx_ids, spdx_docs = read_corpus(index_path)
18
+
19
+ @spdx_ids = spdx_ids
20
+ @corpus = spdx_docs
21
+ @model = TfIdfSimilarity::BM25Model.new(@corpus, :library => :narray)
22
+
23
+ true
24
+ end
25
+
26
+ def match_text(text, min_confidence = DEFAULT_MIN_CONFIDENCE, is_processed_text = false)
27
+ return [] if text.to_s.empty?
28
+
29
+ text = preprocess_text(text) if is_processed_text == false
30
+ test_doc = TfIdfSimilarity::Document.new(text, {:id => "test"})
31
+
32
+ mat1 = @model.instance_variable_get(:@matrix)
33
+ mat2 = doc_tfidf_matrix(test_doc)
34
+
35
+ n_docs = @model.documents.size
36
+ dists = []
37
+ n_docs.times do |i|
38
+ dists << [i, cos_sim(mat1[i, true], mat2)]
39
+ end
40
+
41
+ doc_id, best_score = dists.sort {|a,b| b[1] <=> a[1]}.first
42
+ best_match = @model.documents[doc_id].id
43
+
44
+ if best_score.to_f > min_confidence
45
+ best_match
46
+ else
47
+ ""
48
+ end
49
+ end
50
+
51
+ def match_html(html_text, min_confidence = DEFAULT_MIN_CONFIDENCE)
52
+ match_text(preprocess_html(html_text), min_confidence)
53
+ end
54
+
55
+ #-- helpers
56
+ # Transforms document into TF-IDF matrix used for comparition
57
+ def doc_tfidf_matrix(doc)
58
+ arr = Array.new(@model.terms.size) do |i|
59
+ the_term = @model.terms[i]
60
+ if doc.term_count(the_term) > 0
61
+ #calc score only for words that exists in the test doc and the corpus of licenses
62
+ model.idf(the_term) * model.tf(doc, the_term)
63
+ else
64
+ 0.0
65
+ end
66
+ end
67
+
68
+ NArray[*arr]
69
+ end
70
+
71
+
72
+ # Calculates cosine similarity between 2 TF-IDF vector
73
+ def cos_sim(mat1, mat2)
74
+ length = (mat1 * mat2).sum
75
+ norm = Math::sqrt((mat1 ** 2).sum) * Math::sqrt((mat2 ** 2).sum)
76
+
77
+ ( norm > 0 ? length / norm : 0.0)
78
+ end
79
+
80
+ # Reads the content of licenses from the pre-built index
81
+ # NB! it is sensitive to the changes in the Fosslim/Index serialization
82
+ def read_corpus(index_path)
83
+ idx = MessagePack.unpack File.read index_path
84
+ spdx_ids = []
85
+ docs = []
86
+
87
+ idx[A_DOC_ROW].to_a.each do |doc_row|
88
+ _, spdx_id, content, _ = doc_row
89
+ txt = preprocess_text content
90
+ if txt
91
+ spdx_ids << spdx_id
92
+ docs << TfIdfSimilarity::Document.new(txt, :id => spdx_id)
93
+ end
94
+ end
95
+
96
+ [spdx_ids, docs]
97
+ end
98
+
99
+ end
100
+ end
data/lib/license_matcher/url_matcher.rb ADDED
@@ -0,0 +1,102 @@
1
+
2
+ module LicenseMatcher
3
+
4
+ class UrlMatcher
5
+ attr_reader :url_index
6
+
7
+ DEFAULT_LICENSE_JSON = 'data/spdx_licenses/licenses.json'
8
+
9
+ def initialize(license_json_file = DEFAULT_LICENSE_JSON)
10
+ licenses_json_doc = read_json_file license_json_file
11
+ raise("Failed to read licenses.json") if licenses_json_doc.nil?
12
+
13
+ @url_index = read_license_url_index(licenses_json_doc)
14
+ end
15
+
16
+ # Matches License.url with urls in Licenses.json and returns tuple [spdx_id, score]
17
+ def match_url(the_url)
18
+ the_url = the_url.to_s.strip
19
+ spdx_id = nil
20
+
21
+ case the_url
22
+ when 'http://jquery.org/license'
23
+ return ['mit', 1.0] #Jquery license page doesnt include any license text
24
+ when 'https://www.mozilla.org/en-US/MPL/'
25
+ return ['mpl-2.0', 1.0]
26
+ when 'http://fairlicense.org'
27
+ return ['fair', 1.0]
28
+ when 'http://www.aforgenet.com/framework/license.html'
29
+ return ['lgpl-3.0', 1.0]
30
+ when 'http://www.apache.org/licenses/'
31
+ return ['apache-2.0', 1.0]
32
+ when 'http://aws.amazon.com/apache2.0/'
33
+ return ['apache-2.0', 1.0]
34
+ when 'http://aws.amazon.com/asl/'
35
+ return ['amazon', 1.0]
36
+ when 'https://choosealicense.com/no-license/'
37
+ return ['no-license', 1.0]
38
+ when 'http://www.gzip.org/zlib/zlib_license.html'
39
+ return ['zlib', 1.0]
40
+ when 'http://zlib.net/zlib-license.html'
41
+ return ['zlib', 1.0]
42
+ when 'http://www.wtfpl.net/about/'
43
+ return ['wtfpl', 1.0]
44
+ end
45
+
46
+ #does url match with choosealicense.com
47
+ match = the_url.match(/\bhttps?:\/\/(www\.)?choosealicense\.com\/licenses\/([\S|^\/]+)[\/]?\b/i)
48
+ if match
49
+ return [match[2].to_s.downcase, 1.0]
50
+ end
51
+
52
+ match = the_url.match(/\bhttps?:\/\/(www\.)?creativecommons\.org\/licenses\/([\S|^\/]+)[\/]?\b/i)
53
+ if match
54
+ return ["cc-#{match[2].to_s.gsub(/\//, '-')}", 1.0]
55
+ end
56
+
57
+ #check through SPDX urls
58
+ @url_index.each do |lic_url, lic_id|
59
+ lic_url = lic_url.to_s.strip.gsub(/https?:\/\//i, '').gsub(/www\./, '') #normalizes urls in the file
60
+ matcher = Regexp.new("https?:\/\/(www\.)?#{lic_url}", Regexp::IGNORECASE)
61
+
62
+ if matcher.match(the_url)
63
+ spdx_id = lic_id.to_s.downcase
64
+ break
65
+ end
66
+ end
67
+
68
+ return [] if spdx_id.nil?
69
+
70
+ [spdx_id, 1.0]
71
+ end
72
+
73
+ # Reads license urls from the license.json and builds a map {url : spdx_id}
74
+ def read_license_url_index(spdx_licenses)
75
+ url_index = {}
76
+ spdx_licenses.each {|lic| url_index.merge! process_spdx_item(lic) }
77
+ url_index
78
+ end
79
+
80
+
81
+ def process_spdx_item(lic)
82
+ url_index = {}
83
+ lic_id = lic[:id].to_s.strip.downcase
84
+
85
+ return url_index if lic_id.empty?
86
+
87
+ lic[:links].to_a.each {|x| url_index[x[:url]] = lic_id }
88
+ lic[:text].to_a.each {|x| url_index[x[:url]] = lic_id }
89
+
90
+ url_index
91
+ end
92
+
93
+ def read_json_file(file_path)
94
+ JSON.parse(File.read(file_path), {symbolize_names: true})
95
+ rescue
96
+ log.info "Failed to read json file `#{file_path}`"
97
+ nil
98
+ end
99
+
100
+
101
+ end
102
+ end
data/lib/license_matcher.rb CHANGED
@@ -1,7 +1,21 @@
1
1
  require "helix_runtime"
2
2
 
3
3
  begin
4
- require "license_matcher/native"
4
+ require "license_matcher/native"
5
5
  rescue LoadError
6
- warn "Unable to load license_matcher/native. Please run `rake build`"
6
+ warn "Unable to load license_matcher/native. Please run `rake build`"
7
+ end
8
+
9
+ require 'license_matcher/preprocess'
10
+ require 'license_matcher/url_matcher'
11
+ require 'license_matcher/rule_matcher'
12
+ require 'license_matcher/tf_ruby_matcher'
13
+
14
+ module LicenseMatcher
15
+
16
+ # if class is missing from the module,
17
+ # then look from global ns
18
+ def self.const_missing(c)
19
+ Object.const_get(c)
20
+ end
7
21
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: license_matcher
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0.pre.alpha
4
+ version: 0.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Timo Sulg
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2017-09-13 00:00:00.000000000 Z
12
+ date: 2017-09-19 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: helix_runtime
@@ -25,6 +25,48 @@ dependencies:
25
25
  - - "~>"
26
26
  - !ruby/object:Gem::Version
27
27
  version: 0.6.0
28
+ - !ruby/object:Gem::Dependency
29
+ name: narray
30
+ requirement: !ruby/object:Gem::Requirement
31
+ requirements:
32
+ - - "~>"
33
+ - !ruby/object:Gem::Version
34
+ version: 0.6.1.2
35
+ type: :runtime
36
+ prerelease: false
37
+ version_requirements: !ruby/object:Gem::Requirement
38
+ requirements:
39
+ - - "~>"
40
+ - !ruby/object:Gem::Version
41
+ version: 0.6.1.2
42
+ - !ruby/object:Gem::Dependency
43
+ name: tf-idf-similarity
44
+ requirement: !ruby/object:Gem::Requirement
45
+ requirements:
46
+ - - "~>"
47
+ - !ruby/object:Gem::Version
48
+ version: 0.1.6
49
+ type: :runtime
50
+ prerelease: false
51
+ version_requirements: !ruby/object:Gem::Requirement
52
+ requirements:
53
+ - - "~>"
54
+ - !ruby/object:Gem::Version
55
+ version: 0.1.6
56
+ - !ruby/object:Gem::Dependency
57
+ name: nokogiri
58
+ requirement: !ruby/object:Gem::Requirement
59
+ requirements:
60
+ - - "~>"
61
+ - !ruby/object:Gem::Version
62
+ version: 1.8.0
63
+ type: :runtime
64
+ prerelease: false
65
+ version_requirements: !ruby/object:Gem::Requirement
66
+ requirements:
67
+ - - "~>"
68
+ - !ruby/object:Gem::Version
69
+ version: 1.8.0
28
70
  - !ruby/object:Gem::Dependency
29
71
  name: bundler
30
72
  requirement: !ruby/object:Gem::Requirement
@@ -67,6 +109,20 @@ dependencies:
67
109
  - - "~>"
68
110
  - !ruby/object:Gem::Version
69
111
  version: '3.4'
112
+ - !ruby/object:Gem::Dependency
113
+ name: msgpack
114
+ requirement: !ruby/object:Gem::Requirement
115
+ requirements:
116
+ - - "~>"
117
+ - !ruby/object:Gem::Version
118
+ version: 1.1.0
119
+ type: :development
120
+ prerelease: false
121
+ version_requirements: !ruby/object:Gem::Requirement
122
+ requirements:
123
+ - - "~>"
124
+ - !ruby/object:Gem::Version
125
+ version: 1.1.0
70
126
  description: "\n LicenseMatcher is rubygem, which uses Fosslim to match various
71
127
  OSS license\n with correct SPDX-id or EULA label.\n "
72
128
  email:
@@ -86,6 +142,10 @@ files:
86
142
  - Rakefile
87
143
  - lib/license_matcher.rb
88
144
  - lib/license_matcher/native.bundle
145
+ - lib/license_matcher/preprocess.rb
146
+ - lib/license_matcher/rule_matcher.rb
147
+ - lib/license_matcher/tf_ruby_matcher.rb
148
+ - lib/license_matcher/url_matcher.rb
89
149
  - lib/tasks/helix_runtime.rake
90
150
  homepage: https://www.github.com/fosslim
91
151
  licenses: []
@@ -101,9 +161,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
101
161
  version: '0'
102
162
  required_rubygems_version: !ruby/object:Gem::Requirement
103
163
  requirements:
104
- - - ">"
164
+ - - ">="
105
165
  - !ruby/object:Gem::Version
106
- version: 1.3.1
166
+ version: '0'
107
167
  requirements: []
108
168
  rubyforge_project:
109
169
  rubygems_version: 2.5.2