hiddengems 0.2.1 → 0.3.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +86 -30
- data/exe/hiddengems.rb +92 -15
- data/hiddengems.gemspec +3 -0
- data/lib/hiddengems/version.rb +1 -1
- metadata +44 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 61ca9fbd2ce6b45ccfafd3ee33feac087833e5f4ec40b8255134ce38523b2bee
|
|
4
|
+
data.tar.gz: 332a46385eb03d77ee543a87792d2a5674ed2e629bc8ebdf95c863461bb59c2f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 267c90327e59a508b1a1d8933bd569d6aa614a2f2d0702473ef9afcb239854d92ff9a80d48fec59a7de3d6faedb47a489592f94ed43fa3a7bab57084fe3f7035
|
|
7
|
+
data.tar.gz: 5c3221b36b09e2d364538db18e04381f8910b920a6b94eacfc76e3f42f29f299b2d47e6f73f4eec31658767bd38f7a10b18279842c6abcce968fbe166a138a3d
|
data/README.md
CHANGED
|
@@ -7,43 +7,99 @@ and sorts the search results by a number of ranking parameters:
|
|
|
7
7
|
- number of versions
|
|
8
8
|
- date of latest version
|
|
9
9
|
- average date of all released versions
|
|
10
|
+
- download linear regression trend slope
|
|
10
11
|
|
|
11
|
-
Using the above, it produces
|
|
12
|
-
works like this:
|
|
12
|
+
Using the above, it produces two tables:
|
|
13
13
|
|
|
14
|
-
The
|
|
15
|
-
The ranking sum that is loweest produces the highest "vitality" ranking.
|
|
14
|
+
The first is comprised of 5 lists of individual rankings sorted best -> worst for each metric.
|
|
16
15
|
|
|
17
|
-
The
|
|
16
|
+
The second table contains a list of gems and their descriptions sorted by the *sum* of all a given gems ranks from the first table. The lowest overall score is best. By default each individual ranking is given a weight of 1, but you can alter this with command line options (see below)
|
|
17
|
+
|
|
18
|
+
The rationale for this is that ranking based on download count alone (e.g. http://bestgems.org) doesn't always give you a good sense of where things stand for a given search term. Looking at download count alone, a gem that has been around for many years and has garnered many downloads but has stagnated will tend to retain its top ranking due to the fact that people will tend to download the top ranking gem first. In other words: it invites a positive feedback situation.
|
|
19
|
+
|
|
20
|
+
hiddengems.rb attempts to do a better job of ranking for a given search term. As it's grabbing all the requisite data at run time, it's not as fast as using an indexing site, but it's definitely "smarter".
|
|
21
|
+
|
|
22
|
+
The program supports options that allow you to adjust the weights of the metrics that are used to calculate the rank sum table. By default all metrics are given equal weight, but you can tweak this as you see fit.
|
|
23
|
+
|
|
24
|
+
The program also includes a link validation function. If you use it (--links), hiddengems.rb will first audit each gem's homepage_uri and source_code_uri links to ensure that (a) at least one of those URIs is populated and (b) that the links that are defined are valid. If both (a) and (b) fail, the gem is removed from the list of found gems before ranking is calculated.
|
|
25
|
+
|
|
26
|
+
The --links option can add a bunch of time to processing so it's not on by default, but it does a good job of weeding out dead projects.
|
|
27
|
+
|
|
28
|
+
Note: you'll almost certainly want to pipe output to a pager (e.g. "| less") as the output tends to be pretty long line-wise.
|
|
18
29
|
|
|
19
30
|
Example:
|
|
20
31
|
|
|
21
32
|
```
|
|
22
|
-
$ hiddengems.rb
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
33
|
+
$ ./hiddengems.rb --help
|
|
34
|
+
Usage:
|
|
35
|
+
hiddengems.rb [options] '<search term>'
|
|
36
|
+
|
|
37
|
+
Where [options] are:
|
|
38
|
+
-l, --links Verify gem's URIs before scoring (slow)
|
|
39
|
+
-d, --downloads=<i> Weight of download count (default: 1)
|
|
40
|
+
-v, --versions=<i> Weight of version count (default: 1)
|
|
41
|
+
-a, --latest-version-date=<i> Weight of latest version date (default: 1)
|
|
42
|
+
-g, --avg-version-date=<i> Weight of average version date (default: 1)
|
|
43
|
+
-o, --download-slope=<i> Weight of download slope (default: 1)
|
|
44
|
+
-h, --help Show this message
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
$ ./hiddengems.rb --links --avg-version-date 2 'zabbix sender'
|
|
48
|
+
Option values:
|
|
49
|
+
{
|
|
50
|
+
:links => true,
|
|
51
|
+
:downloads => 1,
|
|
52
|
+
:versions => 1,
|
|
53
|
+
:latest_version_date => 1,
|
|
54
|
+
:avg_version_date => 2,
|
|
55
|
+
:download_slope => 1,
|
|
56
|
+
:help => false,
|
|
57
|
+
:links_given => true,
|
|
58
|
+
:avg_version_date_given => true
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
Search term: zabbix sender
|
|
62
|
+
Found +6 gems (total 6)
|
|
63
|
+
|
|
64
|
+
Search term 'zabbix sender' returned 6 hits
|
|
65
|
+
|
|
66
|
+
Validating URIs...
|
|
67
|
+
Attempting https://github.com/nownabe/zabbix_sender
|
|
68
|
+
Attempting https://svdasein.gitlab.io/zabbix_sender_api
|
|
69
|
+
Attempting https://gitlab.com/svdasein/zabbix_sender_api
|
|
70
|
+
Attempting https://github.com/lehn-etracker/zabbix_sender_legacy
|
|
71
|
+
Attempting https://github.com/eduvo/zabbix-ruby-client
|
|
72
|
+
Attempting https://github.com/englishm/zabbix_send
|
|
73
|
+
Attempting https://github.com/englishm/zabbix_send
|
|
74
|
+
Rejecting zabbix_receiver due to lack of valid URIs
|
|
75
|
+
|
|
76
|
+
5 remain gems after link validation
|
|
77
|
+
|
|
78
|
+
Calculating rankings...
|
|
79
|
+
|
|
80
|
+
Individual Rankings (best -> worst)
|
|
81
|
+
+-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
|
|
82
|
+
|rank |downloads |versions |latest_version_date |avg_version_date |download_slope |
|
|
83
|
+
+-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
|
|
84
|
+
|0 |zabbix-ruby-client (51536) |zabbix-ruby-client (24) |zabbix_sender_api (2022) |zabbix_sender_api (2021) |zabbix_send (1289) |
|
|
85
|
+
|1 |zabbix_sender (35317) |zabbix_sender_api (11) |zabbix_sender (2020) |zabbix_sender (2017) |zabbix_sender (320) |
|
|
86
|
+
|2 |zabbix_send (13477) |zabbix_sender (5) |zabbix_sender_legacy (2016) |zabbix_sender_legacy (2016) |zabbix_sender_legacy (279) |
|
|
87
|
+
|3 |zabbix_sender_api (10149) |zabbix_sender_legacy (5) |zabbix-ruby-client (2016) |zabbix-ruby-client (2014) |zabbix_sender_api (100) |
|
|
88
|
+
|4 |zabbix_sender_legacy (9989) |zabbix_send (3) |zabbix_send (2013) |zabbix_send (2013) |zabbix-ruby-client (64) |
|
|
89
|
+
+-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
|
|
90
|
+
|
|
91
|
+
Rank sums (best -> worst)
|
|
92
|
+
+-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
|
|
93
|
+
|score|name |info |
|
|
94
|
+
+-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
|
|
95
|
+
|7 |zabbix_sender |Zabbix Sender gem |
|
|
96
|
+
|7 |zabbix_sender_api |This gem implements an api that abstracts the zabbix sender/trapper mechanism. It saves tons of ti… |
|
|
97
|
+
|13 |zabbix-ruby-client |A zabbix alternative to zabbix-agent using zabbix-sender. |
|
|
98
|
+
|15 |zabbix_sender_legacy |Improved zabbix_sender gem just for legacy ruby < 2.0 like in Debian Wheezy |
|
|
99
|
+
|18 |zabbix_send |Send data to Zabbix trappers from Ruby |
|
|
100
|
+
+-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
|
|
101
|
+
|
|
102
|
+
|
|
47
103
|
```
|
|
48
104
|
|
|
49
105
|
## Installation
|
data/exe/hiddengems.rb
CHANGED
|
@@ -2,12 +2,14 @@
|
|
|
2
2
|
require 'json'
|
|
3
3
|
require 'time'
|
|
4
4
|
require 'faraday'
|
|
5
|
+
require 'faraday/follow_redirects'
|
|
6
|
+
require 'optimist'
|
|
5
7
|
require 'ostruct'
|
|
6
8
|
require 'tty-table'
|
|
7
9
|
require 'pry-rescue'
|
|
8
10
|
require 'linear_regression_trend'
|
|
11
|
+
require 'awesome_print'
|
|
9
12
|
|
|
10
|
-
searchTerm = ARGV[0]
|
|
11
13
|
|
|
12
14
|
module GemEntry
|
|
13
15
|
def catalogEntry
|
|
@@ -47,6 +49,36 @@ module GemEntry
|
|
|
47
49
|
slope = LinearRegressionTrend::Calculator.new(versionDownloadHistory).slope
|
|
48
50
|
slope.nan? ? 0 : slope
|
|
49
51
|
end
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def uriScore
|
|
55
|
+
if @uriScore.nil?
|
|
56
|
+
@uriScore = 0.0
|
|
57
|
+
uriCount = 0
|
|
58
|
+
foundCount = 0
|
|
59
|
+
[:homepage_uri, :source_code_uri].each {|uri|
|
|
60
|
+
if not self[uri].nil? and (self[uri].size > 0)
|
|
61
|
+
print "Attempting #{self[uri]}"
|
|
62
|
+
uriCount += 1
|
|
63
|
+
begin
|
|
64
|
+
client = Faraday.new(url: self[uri]) do |faraday|
|
|
65
|
+
faraday.response(:follow_redirects)
|
|
66
|
+
faraday.options.timeout = 5
|
|
67
|
+
faraday.adapter(Faraday.default_adapter)
|
|
68
|
+
end
|
|
69
|
+
res = client.get
|
|
70
|
+
foundCount += 1 if (res.status == 200)
|
|
71
|
+
puts
|
|
72
|
+
rescue Exception => e
|
|
73
|
+
ap e
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
}
|
|
77
|
+
@uriScore = uriCount == 0.0 ? 0 : foundCount.to_f / uriCount.to_f
|
|
78
|
+
end
|
|
79
|
+
@uriScore
|
|
80
|
+
end
|
|
81
|
+
|
|
50
82
|
end
|
|
51
83
|
|
|
52
84
|
module GemVersion
|
|
@@ -63,12 +95,43 @@ module GemVersion
|
|
|
63
95
|
end
|
|
64
96
|
|
|
65
97
|
|
|
98
|
+
##################################################################################################################
|
|
99
|
+
#
|
|
100
|
+
|
|
101
|
+
opts = Optimist::options do
|
|
102
|
+
banner <<-EOS
|
|
103
|
+
Usage:
|
|
104
|
+
hiddengems.rb [options] '<search term>'
|
|
105
|
+
|
|
106
|
+
Where [options] are:
|
|
107
|
+
EOS
|
|
108
|
+
opt :links, "Verify gem's URIs before scoring (slow)", type: :boolean, default: false
|
|
109
|
+
opt :downloads, "Weight of download count", type: :integer, default: 1
|
|
110
|
+
opt :versions, "Weight of version count", type: :integer, default: 1
|
|
111
|
+
opt :latest_version_date, "Weight of latest version date", type: :integer, default: 1
|
|
112
|
+
opt :avg_version_date, "Weight of average version date", type: :integer, default: 1
|
|
113
|
+
opt :download_slope, "Weight of download slope", type: :integer, default: 1
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
puts "Option values:"
|
|
117
|
+
ap opts
|
|
118
|
+
|
|
119
|
+
searchTerm = ARGV[0]
|
|
120
|
+
puts "\nSearch term: #{searchTerm}\n"
|
|
121
|
+
|
|
122
|
+
|
|
66
123
|
curpage = 1
|
|
67
124
|
gems = Array.new
|
|
68
125
|
gemdict = Hash.new
|
|
69
126
|
until curpage == 0
|
|
127
|
+
dupepad = 0
|
|
70
128
|
gemPage = JSON.parse(Faraday.get("https://rubygems.org/api/v1/search.json?page=#{curpage}&query=#{searchTerm}").body).collect { |each| OpenStruct.new(each)}.collect {|gem|
|
|
71
129
|
gem.extend(GemEntry)
|
|
130
|
+
if gemdict.has_key?(gem.name)
|
|
131
|
+
puts "(rejected a dupe)"
|
|
132
|
+
dupepad += 1
|
|
133
|
+
next
|
|
134
|
+
end
|
|
72
135
|
gem[:versions] = JSON.parse(
|
|
73
136
|
Faraday.get(
|
|
74
137
|
"https://rubygems.org/api/v1/versions/#{gem.name}.json"
|
|
@@ -79,39 +142,53 @@ until curpage == 0
|
|
|
79
142
|
gem[:rank] = 0
|
|
80
143
|
gemdict[gem.name] = gem
|
|
81
144
|
gem
|
|
82
|
-
}
|
|
145
|
+
}.reject(&:nil?)
|
|
83
146
|
gems += gemPage
|
|
84
147
|
puts "Found +#{gemPage.size} gems (total #{gems.size})"
|
|
85
|
-
curpage = ((gemPage.size < 30) ? 0 : curpage.next)
|
|
148
|
+
curpage = (((gemPage.size + dupepad) < 30) ? 0 : curpage.next)
|
|
86
149
|
puts "looking for more..." if curpage != 0
|
|
87
150
|
end
|
|
88
151
|
|
|
89
152
|
|
|
90
|
-
puts "
|
|
153
|
+
puts "\nSearch term '#{searchTerm}' returned #{gems.size} hits"
|
|
154
|
+
if opts[:links]
|
|
155
|
+
puts "\nValidating URIs..."
|
|
156
|
+
gems.reject! {|each|
|
|
157
|
+
if each.uriScore == 0.0
|
|
158
|
+
puts "Rejecting #{each.name} due to lack of valid URIs"
|
|
159
|
+
true
|
|
160
|
+
else
|
|
161
|
+
false
|
|
162
|
+
end
|
|
163
|
+
}
|
|
164
|
+
puts "\n#{gems.size} remain gems after link validation"
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
puts "\nCalculating rankings..."
|
|
91
168
|
|
|
92
169
|
data = Hash.new
|
|
93
|
-
data[:
|
|
94
|
-
data[:
|
|
95
|
-
data[:
|
|
96
|
-
data[:
|
|
97
|
-
data[:
|
|
170
|
+
data[:downloads] = gems.sort {|a,b| b.downloads <=> a.downloads}.collect {|each| "#{each.name} (#{each.downloads})"}
|
|
171
|
+
data[:versions] = gems.sort {|a,b| b.totalVersions <=> a.totalVersions}.collect {|each| "#{each.name} (#{each.totalVersions})"}
|
|
172
|
+
data[:latest_version_date] = gems.sort {|a,b| b.latestVersionDate <=> a.latestVersionDate}.collect {|each| "#{each.name} (#{each.latestVersionDate.year})"}
|
|
173
|
+
data[:avg_version_date] = gems.sort {|a,b| b.avgDateOfAllVersions<=> a.avgDateOfAllVersions}.collect {|each| "#{each.name} (#{each.avgDateOfAllVersions.year})"}
|
|
174
|
+
data[:download_slope] = gems.sort {|a,b| b.downloadSlope <=> a.downloadSlope}.collect {|each| "#{each.name} (#{each.downloadSlope.round})"}
|
|
98
175
|
report = Array.new
|
|
99
176
|
rank = 0
|
|
100
177
|
gems.each_index {|index|
|
|
101
178
|
record = Hash.new
|
|
102
179
|
record[:rank] = rank
|
|
103
|
-
|
|
180
|
+
data.keys.each {|column|
|
|
104
181
|
record[column] = data[column][index]
|
|
105
|
-
gemdict[data[column][index].split(' (').first][:rank] += index
|
|
182
|
+
gemdict[data[column][index].split(' (').first][:rank] += index * opts[column]
|
|
106
183
|
}
|
|
107
184
|
report.push(record)
|
|
108
185
|
rank += 1
|
|
109
186
|
}
|
|
187
|
+
puts
|
|
188
|
+
puts "Individual Rankings (best -> worst)"
|
|
189
|
+
puts TTY::Table.new(header: [:rank]+data.keys, rows: report).render(:ascii, resize: false, column_widths: [5] + [30] * data.keys.size)
|
|
110
190
|
|
|
111
|
-
puts "
|
|
112
|
-
puts TTY::Table.new(header: [:rank]+data.keys, rows: report).render(:ascii)
|
|
113
|
-
|
|
114
|
-
puts "\nVitality Ranking:"
|
|
191
|
+
puts "\nRank sums (best -> worst)"
|
|
115
192
|
puts TTY::Table.new(header: [:score, :name,:info], rows: gems.sort {|a,b| a.rank<=>b.rank}.collect {|each| [each.rank,each.name,each.info]}).render(:ascii, column_widths: [5,35,100],resize: false)
|
|
116
193
|
|
|
117
194
|
puts
|
data/hiddengems.gemspec
CHANGED
|
@@ -33,8 +33,11 @@ Gem::Specification.new do |spec|
|
|
|
33
33
|
spec.add_dependency 'json'
|
|
34
34
|
spec.add_dependency 'time'
|
|
35
35
|
spec.add_dependency 'faraday'
|
|
36
|
+
spec.add_dependency 'faraday-follow_redirects'
|
|
37
|
+
spec.add_dependency 'optimist'
|
|
36
38
|
spec.add_dependency 'ostruct'
|
|
37
39
|
spec.add_dependency 'tty-table'
|
|
40
|
+
spec.add_dependency 'awesome_print'
|
|
38
41
|
spec.add_dependency 'pry-rescue'
|
|
39
42
|
spec.add_dependency 'linear_regression_trend'
|
|
40
43
|
|
data/lib/hiddengems/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: hiddengems
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Dave Parker
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2022-
|
|
11
|
+
date: 2022-09-04 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: json
|
|
@@ -52,6 +52,34 @@ dependencies:
|
|
|
52
52
|
- - ">="
|
|
53
53
|
- !ruby/object:Gem::Version
|
|
54
54
|
version: '0'
|
|
55
|
+
- !ruby/object:Gem::Dependency
|
|
56
|
+
name: faraday-follow_redirects
|
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
|
58
|
+
requirements:
|
|
59
|
+
- - ">="
|
|
60
|
+
- !ruby/object:Gem::Version
|
|
61
|
+
version: '0'
|
|
62
|
+
type: :runtime
|
|
63
|
+
prerelease: false
|
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
65
|
+
requirements:
|
|
66
|
+
- - ">="
|
|
67
|
+
- !ruby/object:Gem::Version
|
|
68
|
+
version: '0'
|
|
69
|
+
- !ruby/object:Gem::Dependency
|
|
70
|
+
name: optimist
|
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
|
72
|
+
requirements:
|
|
73
|
+
- - ">="
|
|
74
|
+
- !ruby/object:Gem::Version
|
|
75
|
+
version: '0'
|
|
76
|
+
type: :runtime
|
|
77
|
+
prerelease: false
|
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
79
|
+
requirements:
|
|
80
|
+
- - ">="
|
|
81
|
+
- !ruby/object:Gem::Version
|
|
82
|
+
version: '0'
|
|
55
83
|
- !ruby/object:Gem::Dependency
|
|
56
84
|
name: ostruct
|
|
57
85
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -80,6 +108,20 @@ dependencies:
|
|
|
80
108
|
- - ">="
|
|
81
109
|
- !ruby/object:Gem::Version
|
|
82
110
|
version: '0'
|
|
111
|
+
- !ruby/object:Gem::Dependency
|
|
112
|
+
name: awesome_print
|
|
113
|
+
requirement: !ruby/object:Gem::Requirement
|
|
114
|
+
requirements:
|
|
115
|
+
- - ">="
|
|
116
|
+
- !ruby/object:Gem::Version
|
|
117
|
+
version: '0'
|
|
118
|
+
type: :runtime
|
|
119
|
+
prerelease: false
|
|
120
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
121
|
+
requirements:
|
|
122
|
+
- - ">="
|
|
123
|
+
- !ruby/object:Gem::Version
|
|
124
|
+
version: '0'
|
|
83
125
|
- !ruby/object:Gem::Dependency
|
|
84
126
|
name: pry-rescue
|
|
85
127
|
requirement: !ruby/object:Gem::Requirement
|