hiddengems 0.2.3 → 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (6) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +77 -23
  3. data/exe/hiddengems.rb +84 -13
  4. data/hiddengems.gemspec +3 -0
  5. data/lib/hiddengems/version.rb +1 -1
  6. metadata +44 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 55b3634a3b5823bdd3d634dad3e013d377a7e16f3f33496bcc3859dbee798199
4
- data.tar.gz: b495911b4b3b4b6f16c43dd539a726bff9422e0c0de4812d17f44fcbab3cf31b
3
+ metadata.gz: 61ca9fbd2ce6b45ccfafd3ee33feac087833e5f4ec40b8255134ce38523b2bee
4
+ data.tar.gz: 332a46385eb03d77ee543a87792d2a5674ed2e629bc8ebdf95c863461bb59c2f
5
5
  SHA512:
6
- metadata.gz: b9e48d6fa7a5d51c3a8cbe3d15fb51abd22ed9741b2de2f76b5aaf519ec4e693755a35417e20b23378c5ddd7122cd6b7deaaeaf55b52f91eb0a6eab6c9d695f1
7
- data.tar.gz: 9e5811bc5737f1d13d524091867f3d797f76f832150cfcb36f6fe240c99e07963c00f1bbebb723acb8429b475871c01cd39c68e71cd21c202fa66eabe55911e8
6
+ metadata.gz: 267c90327e59a508b1a1d8933bd569d6aa614a2f2d0702473ef9afcb239854d92ff9a80d48fec59a7de3d6faedb47a489592f94ed43fa3a7bab57084fe3f7035
7
+ data.tar.gz: 5c3221b36b09e2d364538db18e04381f8910b920a6b94eacfc76e3f42f29f299b2d47e6f73f4eec31658767bd38f7a10b18279842c6abcce968fbe166a138a3d
data/README.md CHANGED
@@ -9,43 +9,97 @@ and sorts the search results by a number of ranking parameters:
9
9
  - average date of all released versions
10
10
  - download linear regression trend slope
11
11
 
12
- Using the above, it produces a table of found gems sorted by "vitality", which
13
- works like this:
12
+ Using the above, it produces two tables:
14
13
 
15
- The gem's ranking order for each of the above ranking parameters is summed together.
16
- The ranking sum that is lowest produces the highest "vitality" ranking.
14
+ The first is comprised of 5 lists of individual rankings sorted best -> worst for each metric.
17
15
 
18
- The rationale for this is: bestgems.org ranks only according to download counts. If developers are using just bestgems.org's results to help find libraries that they need, they'll end up with results that are heavily biased toward libraries that have simply been around for a long time; if a gem has been around for many years it will accrue lots of downloads regardless of whether the gem is being actively developed/improved.
16
+ The second table contains a list of gems and their descriptions sorted by the *sum* of all a given gems ranks from the first table. The lowest overall score is best. By default each individual ranking is given a weight of 1, but you can alter this with command line options (see below)
17
+
18
+ The rationale for this is that ranking based on download count alone (e.g. http://bestgems.org) doesn't always give you a good sense of where things stand for a given search term. Looking at download count alone, a gem that has been around for many years and has garnered many downloads but has stagnated will tend to retain its top ranking due to the fact that people will tend to download the top ranking gem first. In other words: it invites a positive feedback situation.
19
+
20
+ hiddengems.rb attempts to do a better job of ranking for a given search term. As it's grabbing all the requisite data at run time, it's not as fast as using an indexing site, but it's definitely "smarter".
21
+
22
+ The program supports options that allow you to adjust the weights of the metrics that are used to calculate the rank sum table. By default all metrics are given equal weight, but you can tweak this as you see fit.
23
+
24
+ The program also includes a link validation function. If you use it (--links), hiddengems.rb will first audit each gem's homepage_uri and source_code_uri links to ensure that (a) at least one of those URIs is populated and (b) that the links that are defined are valid. If both (a) and (b) fail, the gem is removed from the list of found gems before ranking is calculated.
25
+
26
+ The --links option can add a bunch of time to processing so it's not on by default, but it does a good job of weeding out dead projects.
27
+
28
+ Note: you'll almost certainly want to pipe output to a pager (e.g. "| less") as the output tends to be pretty long line-wise.
19
29
 
20
30
  Example:
21
31
 
22
32
  ```
23
- $ hiddengems.rb 'zabbix sender'
33
+ $ ./hiddengems.rb --help
34
+ Usage:
35
+ hiddengems.rb [options] '<search term>'
36
+
37
+ Where [options] are:
38
+ -l, --links Verify gem's URIs before scoring (slow)
39
+ -d, --downloads=<i> Weight of download count (default: 1)
40
+ -v, --versions=<i> Weight of version count (default: 1)
41
+ -a, --latest-version-date=<i> Weight of latest version date (default: 1)
42
+ -g, --avg-version-date=<i> Weight of average version date (default: 1)
43
+ -o, --download-slope=<i> Weight of download slope (default: 1)
44
+ -h, --help Show this message
45
+
46
+
47
+ $ ./hiddengems.rb --links --avg-version-date 2 'zabbix sender'
48
+ Option values:
49
+ {
50
+ :links => true,
51
+ :downloads => 1,
52
+ :versions => 1,
53
+ :latest_version_date => 1,
54
+ :avg_version_date => 2,
55
+ :download_slope => 1,
56
+ :help => false,
57
+ :links_given => true,
58
+ :avg_version_date_given => true
59
+ }
60
+
61
+ Search term: zabbix sender
24
62
  Found +6 gems (total 6)
63
+
25
64
  Search term 'zabbix sender' returned 6 hits
26
- Simple Rankings
27
- +----+---------------------------+------------------------+---------------------------+---------------------------+--------------------------+
28
- |rank|by_downloads |by_versions |by_latest_version_date |by_avg_version_date |by_download_slope |
29
- +----+---------------------------+------------------------+---------------------------+---------------------------+--------------------------+
30
- |0 |zabbix-ruby-client (51368) |zabbix-ruby-client (24) |zabbix_sender_api (2022) |zabbix_sender_api (2021) |zabbix_send (1286) |
31
- |1 |zabbix_sender (35004) |zabbix_sender_api (11) |zabbix_sender (2020) |zabbix_sender (2017) |zabbix_receiver (375) |
32
- |2 |zabbix_send (13456) |zabbix_sender (5) |zabbix_sender_legacy (2016)|zabbix_sender_legacy (2016)|zabbix_sender (319) |
33
- |3 |zabbix_receiver (10286) |zabbix_sender_legacy (5)|zabbix-ruby-client (2016) |zabbix_receiver (2015) |zabbix_sender_legacy (278)|
34
- |4 |zabbix_sender_api (10064) |zabbix_receiver (5) |zabbix_receiver (2015) |zabbix-ruby-client (2014) |zabbix_sender_api (100) |
35
- |5 |zabbix_sender_legacy (9954)|zabbix_send (3) |zabbix_send (2013) |zabbix_send (2013) |zabbix-ruby-client (64) |
36
- +----+---------------------------+------------------------+---------------------------+---------------------------+--------------------------+
37
-
38
- Vitality Ranking:
65
+
66
+ Validating URIs...
67
+ Attempting https://github.com/nownabe/zabbix_sender
68
+ Attempting https://svdasein.gitlab.io/zabbix_sender_api
69
+ Attempting https://gitlab.com/svdasein/zabbix_sender_api
70
+ Attempting https://github.com/lehn-etracker/zabbix_sender_legacy
71
+ Attempting https://github.com/eduvo/zabbix-ruby-client
72
+ Attempting https://github.com/englishm/zabbix_send
73
+ Attempting https://github.com/englishm/zabbix_send
74
+ Rejecting zabbix_receiver due to lack of valid URIs
75
+
76
+ 5 remain gems after link validation
77
+
78
+ Calculating rankings...
79
+
80
+ Individual Rankings (best -> worst)
81
+ +-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
82
+ |rank |downloads |versions |latest_version_date |avg_version_date |download_slope |
83
+ +-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
84
+ |0 |zabbix-ruby-client (51536) |zabbix-ruby-client (24) |zabbix_sender_api (2022) |zabbix_sender_api (2021) |zabbix_send (1289) |
85
+ |1 |zabbix_sender (35317) |zabbix_sender_api (11) |zabbix_sender (2020) |zabbix_sender (2017) |zabbix_sender (320) |
86
+ |2 |zabbix_send (13477) |zabbix_sender (5) |zabbix_sender_legacy (2016) |zabbix_sender_legacy (2016) |zabbix_sender_legacy (279) |
87
+ |3 |zabbix_sender_api (10149) |zabbix_sender_legacy (5) |zabbix-ruby-client (2016) |zabbix-ruby-client (2014) |zabbix_sender_api (100) |
88
+ |4 |zabbix_sender_legacy (9989) |zabbix_send (3) |zabbix_send (2013) |zabbix_send (2013) |zabbix-ruby-client (64) |
89
+ +-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
90
+
91
+ Rank sums (best -> worst)
39
92
  +-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
40
93
  |score|name |info |
41
94
  +-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
42
95
  |7 |zabbix_sender |Zabbix Sender gem |
43
- |9 |zabbix_sender_api |This gem implements an api that abstracts the zabbix sender/trapper mechanism. It saves tons of ti… |
44
- |12 |zabbix-ruby-client |A zabbix alternative to zabbix-agent using zabbix-sender. |
96
+ |7 |zabbix_sender_api |This gem implements an api that abstracts the zabbix sender/trapper mechanism. It saves tons of ti… |
97
+ |13 |zabbix-ruby-client |A zabbix alternative to zabbix-agent using zabbix-sender. |
45
98
  |15 |zabbix_sender_legacy |Improved zabbix_sender gem just for legacy ruby &lt; 2.0 like in Debian Wheezy |
46
- |15 |zabbix_receiver |Server to receive sender data from zabbix-agent. |
47
- |17 |zabbix_send |Send data to Zabbix trappers from Ruby |
99
+ |18 |zabbix_send |Send data to Zabbix trappers from Ruby |
48
100
  +-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
101
+
102
+
49
103
  ```
50
104
 
51
105
  ## Installation
data/exe/hiddengems.rb CHANGED
@@ -2,12 +2,14 @@
2
2
  require 'json'
3
3
  require 'time'
4
4
  require 'faraday'
5
+ require 'faraday/follow_redirects'
6
+ require 'optimist'
5
7
  require 'ostruct'
6
8
  require 'tty-table'
7
9
  require 'pry-rescue'
8
10
  require 'linear_regression_trend'
11
+ require 'awesome_print'
9
12
 
10
- searchTerm = ARGV[0]
11
13
 
12
14
  module GemEntry
13
15
  def catalogEntry
@@ -47,6 +49,36 @@ module GemEntry
47
49
  slope = LinearRegressionTrend::Calculator.new(versionDownloadHistory).slope
48
50
  slope.nan? ? 0 : slope
49
51
  end
52
+
53
+
54
+ def uriScore
55
+ if @uriScore.nil?
56
+ @uriScore = 0.0
57
+ uriCount = 0
58
+ foundCount = 0
59
+ [:homepage_uri, :source_code_uri].each {|uri|
60
+ if not self[uri].nil? and (self[uri].size > 0)
61
+ print "Attempting #{self[uri]}"
62
+ uriCount += 1
63
+ begin
64
+ client = Faraday.new(url: self[uri]) do |faraday|
65
+ faraday.response(:follow_redirects)
66
+ faraday.options.timeout = 5
67
+ faraday.adapter(Faraday.default_adapter)
68
+ end
69
+ res = client.get
70
+ foundCount += 1 if (res.status == 200)
71
+ puts
72
+ rescue Exception => e
73
+ ap e
74
+ end
75
+ end
76
+ }
77
+ @uriScore = uriCount == 0.0 ? 0 : foundCount.to_f / uriCount.to_f
78
+ end
79
+ @uriScore
80
+ end
81
+
50
82
  end
51
83
 
52
84
  module GemVersion
@@ -63,6 +95,31 @@ module GemVersion
63
95
  end
64
96
 
65
97
 
98
+ ##################################################################################################################
99
+ #
100
+
101
+ opts = Optimist::options do
102
+ banner <<-EOS
103
+ Usage:
104
+ hiddengems.rb [options] '<search term>'
105
+
106
+ Where [options] are:
107
+ EOS
108
+ opt :links, "Verify gem's URIs before scoring (slow)", type: :boolean, default: false
109
+ opt :downloads, "Weight of download count", type: :integer, default: 1
110
+ opt :versions, "Weight of version count", type: :integer, default: 1
111
+ opt :latest_version_date, "Weight of latest version date", type: :integer, default: 1
112
+ opt :avg_version_date, "Weight of average version date", type: :integer, default: 1
113
+ opt :download_slope, "Weight of download slope", type: :integer, default: 1
114
+ end
115
+
116
+ puts "Option values:"
117
+ ap opts
118
+
119
+ searchTerm = ARGV[0]
120
+ puts "\nSearch term: #{searchTerm}\n"
121
+
122
+
66
123
  curpage = 1
67
124
  gems = Array.new
68
125
  gemdict = Hash.new
@@ -93,31 +150,45 @@ until curpage == 0
93
150
  end
94
151
 
95
152
 
96
- puts "Search term '#{searchTerm}' returned #{gems.size} hits"
153
+ puts "\nSearch term '#{searchTerm}' returned #{gems.size} hits"
154
+ if opts[:links]
155
+ puts "\nValidating URIs..."
156
+ gems.reject! {|each|
157
+ if each.uriScore == 0.0
158
+ puts "Rejecting #{each.name} due to lack of valid URIs"
159
+ true
160
+ else
161
+ false
162
+ end
163
+ }
164
+ puts "\n#{gems.size} remain gems after link validation"
165
+ end
166
+
167
+ puts "\nCalculating rankings..."
97
168
 
98
169
  data = Hash.new
99
- data[:by_downloads] = gems.sort {|a,b| b.downloads <=> a.downloads}.collect {|each| "#{each.name} (#{each.downloads})"}
100
- data[:by_versions] = gems.sort {|a,b| b.totalVersions <=> a.totalVersions}.collect {|each| "#{each.name} (#{each.totalVersions})"}
101
- data[:by_latest_version_date] = gems.sort {|a,b| b.latestVersionDate <=> a.latestVersionDate}.collect {|each| "#{each.name} (#{each.latestVersionDate.year})"}
102
- data[:by_avg_version_date] = gems.sort {|a,b| b.avgDateOfAllVersions<=> a.avgDateOfAllVersions}.collect {|each| "#{each.name} (#{each.avgDateOfAllVersions.year})"}
103
- data[:by_download_slope] = gems.sort {|a,b| b.downloadSlope <=> a.downloadSlope}.collect {|each| "#{each.name} (#{each.downloadSlope.round})"}
170
+ data[:downloads] = gems.sort {|a,b| b.downloads <=> a.downloads}.collect {|each| "#{each.name} (#{each.downloads})"}
171
+ data[:versions] = gems.sort {|a,b| b.totalVersions <=> a.totalVersions}.collect {|each| "#{each.name} (#{each.totalVersions})"}
172
+ data[:latest_version_date] = gems.sort {|a,b| b.latestVersionDate <=> a.latestVersionDate}.collect {|each| "#{each.name} (#{each.latestVersionDate.year})"}
173
+ data[:avg_version_date] = gems.sort {|a,b| b.avgDateOfAllVersions<=> a.avgDateOfAllVersions}.collect {|each| "#{each.name} (#{each.avgDateOfAllVersions.year})"}
174
+ data[:download_slope] = gems.sort {|a,b| b.downloadSlope <=> a.downloadSlope}.collect {|each| "#{each.name} (#{each.downloadSlope.round})"}
104
175
  report = Array.new
105
176
  rank = 0
106
177
  gems.each_index {|index|
107
178
  record = Hash.new
108
179
  record[:rank] = rank
109
- [:by_downloads,:by_versions,:by_latest_version_date,:by_avg_version_date,:by_download_slope].each {|column|
180
+ data.keys.each {|column|
110
181
  record[column] = data[column][index]
111
- gemdict[data[column][index].split(' (').first][:rank] += index
182
+ gemdict[data[column][index].split(' (').first][:rank] += index * opts[column]
112
183
  }
113
184
  report.push(record)
114
185
  rank += 1
115
186
  }
187
+ puts
188
+ puts "Individual Rankings (best -> worst)"
189
+ puts TTY::Table.new(header: [:rank]+data.keys, rows: report).render(:ascii, resize: false, column_widths: [5] + [30] * data.keys.size)
116
190
 
117
- puts "Simple Rankings"
118
- puts TTY::Table.new(header: [:rank]+data.keys, rows: report).render(:ascii, resize: false, column_widths: [5,30,30,30,30,30])
119
-
120
- puts "\nVitality Ranking:"
191
+ puts "\nRank sums (best -> worst)"
121
192
  puts TTY::Table.new(header: [:score, :name,:info], rows: gems.sort {|a,b| a.rank<=>b.rank}.collect {|each| [each.rank,each.name,each.info]}).render(:ascii, column_widths: [5,35,100],resize: false)
122
193
 
123
194
  puts
data/hiddengems.gemspec CHANGED
@@ -33,8 +33,11 @@ Gem::Specification.new do |spec|
33
33
  spec.add_dependency 'json'
34
34
  spec.add_dependency 'time'
35
35
  spec.add_dependency 'faraday'
36
+ spec.add_dependency 'faraday-follow_redirects'
37
+ spec.add_dependency 'optimist'
36
38
  spec.add_dependency 'ostruct'
37
39
  spec.add_dependency 'tty-table'
40
+ spec.add_dependency 'awesome_print'
38
41
  spec.add_dependency 'pry-rescue'
39
42
  spec.add_dependency 'linear_regression_trend'
40
43
 
data/lib/hiddengems/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Hiddengems
4
- VERSION = "0.2.3"
4
+ VERSION = "0.3.0"
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: hiddengems
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.3
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Dave Parker
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2022-08-30 00:00:00.000000000 Z
11
+ date: 2022-09-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: json
@@ -52,6 +52,34 @@ dependencies:
52
52
  - - ">="
53
53
  - !ruby/object:Gem::Version
54
54
  version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: faraday-follow_redirects
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ - !ruby/object:Gem::Dependency
70
+ name: optimist
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - ">="
74
+ - !ruby/object:Gem::Version
75
+ version: '0'
76
+ type: :runtime
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - ">="
81
+ - !ruby/object:Gem::Version
82
+ version: '0'
55
83
  - !ruby/object:Gem::Dependency
56
84
  name: ostruct
57
85
  requirement: !ruby/object:Gem::Requirement
@@ -80,6 +108,20 @@ dependencies:
80
108
  - - ">="
81
109
  - !ruby/object:Gem::Version
82
110
  version: '0'
111
+ - !ruby/object:Gem::Dependency
112
+ name: awesome_print
113
+ requirement: !ruby/object:Gem::Requirement
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ version: '0'
118
+ type: :runtime
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ requirements:
122
+ - - ">="
123
+ - !ruby/object:Gem::Version
124
+ version: '0'
83
125
  - !ruby/object:Gem::Dependency
84
126
  name: pry-rescue
85
127
  requirement: !ruby/object:Gem::Requirement