RubyGems - hiddengems - Versions diffs - 0.2.3 → 0.3.2 - Mend

hiddengems 0.2.3 → 0.3.2

Files changed (10) hide show

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55b3634a3b5823bdd3d634dad3e013d377a7e16f3f33496bcc3859dbee798199
-  data.tar.gz: b495911b4b3b4b6f16c43dd539a726bff9422e0c0de4812d17f44fcbab3cf31b
+  metadata.gz: a47531cb9520bab4fcb67bb5290340088c5d11ca3099d78fa2917b8b5fa26dad
+  data.tar.gz: 357f84413e436a8fcef588a617e2feb831c7b3ce5fa7058ea80b0a8ab8f3bf78
 SHA512:
-  metadata.gz: b9e48d6fa7a5d51c3a8cbe3d15fb51abd22ed9741b2de2f76b5aaf519ec4e693755a35417e20b23378c5ddd7122cd6b7deaaeaf55b52f91eb0a6eab6c9d695f1
-  data.tar.gz: 9e5811bc5737f1d13d524091867f3d797f76f832150cfcb36f6fe240c99e07963c00f1bbebb723acb8429b475871c01cd39c68e71cd21c202fa66eabe55911e8
+  metadata.gz: 30306422b2a18cc4ac4cc7fa3d3506868392e2b600066ac94294475ed70ad741afa0df5f2e4eedbceb00a345a4b6467dea648acdbc0589b18e3dc8527a4ec15e
+  data.tar.gz: f9b61754013e77e08b766a10b8e39fe4d464b5c39631e79f5e556074884c1d8a392e5e856f2651f4ee1b388ca1d26a0fd8971ca1b523fac4fbc0144730ea079f

data/LICENSE ADDED Viewed

@@ -0,0 +1,7 @@
+Copyright 2022 David Parker
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md CHANGED Viewed

@@ -9,43 +9,106 @@ and sorts the search results by a number of ranking parameters:
 - average date of all released versions
 - download linear regression trend slope
-Using the above, it produces a table of found gems sorted by "vitality", which
-works like this:
+Using the above, it produces two tables:
-The gem's ranking order for each of the above ranking parameters is summed together.
-The ranking sum that is lowest produces the highest "vitality" ranking.
+The first is comprised of 5 lists of individual rankings sorted best -> worst for each metric.
-The rationale for this is:  bestgems.org ranks only according to download counts.  If developers are using just bestgems.org's results to help find libraries that they need, they'll end up with results that are  heavily biased toward libraries that have simply been around for a long time; if a gem has been around for many years it will accrue lots of downloads regardless of whether the gem is being actively developed/improved.
+The second table contains a list of gems and their descriptions sorted by the *sum* of all a given gems ranks from the first table.  The lowest overall score is best.   By default each individual ranking is given a weight of 1, but you can alter this with command line options (see below)
+![illustration of how scoring works by default](img/2022-09-04_13-40.png "Scoring example")
+The rationale for this is that ranking based on download count alone (e.g. http://bestgems.org) doesn't always give you a good sense of where things stand for a given search term.  Looking at download count alone, a gem that has been around for many years and has garnered many downloads but has stagnated will tend to retain its top ranking due to the fact that people will tend to download the top ranking gem first.  In other words: it invites a positive feedback situation.
+hiddengems.rb attempts to do a better job of ranking for a given search term.  As it's grabbing all the requisite data at run time, it's not as fast as using an indexing site, but it's definitely "smarter".
+The program supports options that allow you to adjust the weights of the metrics that are used to calculate the rank sum table.  By default all metrics are given equal weight, but you can tweak this as you see fit.
+Example:
+```
+hiddengems.rb 'zabbix sender' --links --downloads 0 --avg-version-date 3 --download-slope 3 --latest-version-date 2
+```
+gives:
+![illustration of how scoring works with weight adjustments](img/2022-09-04_13-51.png "Weighted Scoring example")
+The program also includes a link validation function.  If you use it (--links), hiddengems.rb will first audit each gem's homepage_uri and source_code_uri links to ensure that (a) at least one of those URIs is populated and (b) that the links that are defined are valid.  If both (a) and (b) fail, the gem is removed from the list of found gems before ranking is calculated.
+The --links option can add a bunch of time to processing so it's not on by default, but it does a good job of weeding out dead projects.
+Note: you'll almost certainly want to pipe output to a pager (e.g. "| less") as the output tends to be pretty long line-wise.
 Example:
 ```
-$ hiddengems.rb 'zabbix sender'
+$ ./hiddengems.rb --help
+Usage:
+  hiddengems.rb [options] '<search term>'
+Where [options] are:
+  -l, --links                      Verify gem's URIs before scoring (slow)
+  -d, --downloads=<i>              Weight of download count (default: 1)
+  -v, --versions=<i>               Weight of version count (default: 1)
+  -a, --latest-version-date=<i>    Weight of latest version date (default: 1)
+  -g, --avg-version-date=<i>       Weight of average version date (default: 1)
+  -o, --download-slope=<i>         Weight of download slope (default: 1)
+  -r, --rejectfilter=<s>           Regular Expression of gem names to reject (default: )
+  -h, --help                       Show this message
+$ hiddengems.rb 'zabbix sender' --links
+Option values:
+{
+                  :links => true,
+              :downloads => 1,
+               :versions => 1,
+    :latest_version_date => 1,
+       :avg_version_date => 1,
+         :download_slope => 1,
+                   :help => false,
+            :links_given => true
+}
+Search term: zabbix sender
 Found +6 gems (total 6)
 Search term 'zabbix sender' returned 6 hits
-Simple Rankings
-+----+---------------------------+------------------------+---------------------------+---------------------------+--------------------------+
-|rank|by_downloads               |by_versions             |by_latest_version_date     |by_avg_version_date        |by_download_slope         |
-+----+---------------------------+------------------------+---------------------------+---------------------------+--------------------------+
-|0   |zabbix-ruby-client (51368) |zabbix-ruby-client (24) |zabbix_sender_api (2022)   |zabbix_sender_api (2021)   |zabbix_send (1286)        |
-|1   |zabbix_sender (35004)      |zabbix_sender_api (11)  |zabbix_sender (2020)       |zabbix_sender (2017)       |zabbix_receiver (375)     |
-|2   |zabbix_send (13456)        |zabbix_sender (5)       |zabbix_sender_legacy (2016)|zabbix_sender_legacy (2016)|zabbix_sender (319)       |
-|3   |zabbix_receiver (10286)    |zabbix_sender_legacy (5)|zabbix-ruby-client (2016)  |zabbix_receiver (2015)     |zabbix_sender_legacy (278)|
-|4   |zabbix_sender_api (10064)  |zabbix_receiver (5)     |zabbix_receiver (2015)     |zabbix-ruby-client (2014)  |zabbix_sender_api (100)   |
-|5   |zabbix_sender_legacy (9954)|zabbix_send (3)         |zabbix_send (2013)         |zabbix_send (2013)         |zabbix-ruby-client (64)   |
-+----+---------------------------+------------------------+---------------------------+---------------------------+--------------------------+
-Vitality Ranking:
+Validating URIs...
+Attempting https://github.com/nownabe/zabbix_sender
+Attempting https://svdasein.gitlab.io/zabbix_sender_api
+Attempting https://gitlab.com/svdasein/zabbix_sender_api
+Attempting https://github.com/lehn-etracker/zabbix_sender_legacy
+Attempting https://github.com/eduvo/zabbix-ruby-client
+Attempting https://github.com/englishm/zabbix_send
+Attempting https://github.com/englishm/zabbix_send
+Rejecting zabbix_receiver due to lack of valid URIs
+5 remaining gems after link validation
+Calculating rankings...
+Individual Rankings (best -> worst)
++-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
+|rank |downloads (1)                 |versions (1)                  |latest_version_date (1)       |avg_version_date (1)          |download_slope (1)            |
++-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
+|0    |zabbix-ruby-client (51536)    |zabbix-ruby-client (24)       |zabbix_sender_api (2022)      |zabbix_sender_api (2021)      |zabbix_send (1289)            |
+|1    |zabbix_sender (35317)         |zabbix_sender_api (11)        |zabbix_sender (2020)          |zabbix_sender (2017)          |zabbix_sender (320)           |
+|2    |zabbix_send (13477)           |zabbix_sender (5)             |zabbix_sender_legacy (2016)   |zabbix_sender_legacy (2016)   |zabbix_sender_legacy (279)    |
+|3    |zabbix_sender_api (10149)     |zabbix_sender_legacy (5)      |zabbix-ruby-client (2016)     |zabbix-ruby-client (2014)     |zabbix_sender_api (100)       |
+|4    |zabbix_sender_legacy (9989)   |zabbix_send (3)               |zabbix_send (2013)            |zabbix_send (2013)            |zabbix-ruby-client (64)       |
++-----+------------------------------+------------------------------+------------------------------+------------------------------+------------------------------+
+Rank sums (best -> worst)
 +-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
 |score|name                               |info                                                                                                |
 +-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
-|7    |zabbix_sender                      |Zabbix Sender gem                                                                                   |
-|9    |zabbix_sender_api                  |This gem implements an api that abstracts the zabbix sender/trapper mechanism. It saves tons of ti… |
-|12   |zabbix-ruby-client                 |A zabbix alternative to zabbix-agent using zabbix-sender.                                           |
-|15   |zabbix_sender_legacy               |Improved zabbix_sender gem just for legacy ruby &lt; 2.0 like in Debian Wheezy                      |
-|15   |zabbix_receiver                    |Server to receive sender data from zabbix-agent.                                                    |
-|17   |zabbix_send                        |Send data to Zabbix trappers from Ruby                                                              |
+|6    |zabbix_sender                      |Zabbix Sender gem                                                                                   |
+|7    |zabbix_sender_api                  |This gem implements an api that abstracts the zabbix sender/trapper mechanism. It saves tons of ti… |
+|10   |zabbix-ruby-client                 |A zabbix alternative to zabbix-agent using zabbix-sender.                                           |
+|13   |zabbix_sender_legacy               |Improved zabbix_sender gem just for legacy ruby &lt; 2.0 like in Debian Wheezy                      |
+|14   |zabbix_send                        |Send data to Zabbix trappers from Ruby                                                              |
 +-----+-----------------------------------+----------------------------------------------------------------------------------------------------+
 ```
 ## Installation
@@ -76,4 +139,4 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 ## Contributing
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/hiddengems.
+Bug reports and pull requests are welcome on GitHub at https://gitlab.com/svdasein/hiddengems.

data/exe/hiddengems.rb CHANGED Viewed

@@ -2,12 +2,14 @@
 require 'json'
 require 'time'
 require 'faraday'
+require 'faraday/follow_redirects'
+require 'optimist'
 require 'ostruct'
 require 'tty-table'
 require 'pry-rescue'
 require 'linear_regression_trend'
+require 'awesome_print'
-searchTerm = ARGV[0]
 module GemEntry
   def catalogEntry
@@ -47,6 +49,36 @@ module GemEntry
     slope = LinearRegressionTrend::Calculator.new(versionDownloadHistory).slope
     slope.nan? ? 0 : slope
   end
+  def uriScore
+    if @uriScore.nil?
+      @uriScore = 0.0
+      uriCount = 0
+      foundCount = 0
+      [:homepage_uri, :source_code_uri].each {|uri|
+        if not self[uri].nil? and (self[uri].size > 0)
+          print "Attempting #{self[uri]}"
+          uriCount += 1
+          begin
+            client = Faraday.new(url: self[uri]) do |faraday|
+              faraday.response(:follow_redirects)
+              faraday.options.timeout = 5
+              faraday.adapter(Faraday.default_adapter)
+            end
+            res = client.get
+            foundCount += 1 if (res.status == 200)
+            puts
+          rescue Exception => e
+            ap e
+          end
+        end
+      }
+      @uriScore = uriCount == 0.0 ? 0 : foundCount.to_f / uriCount.to_f
+    end
+    @uriScore
+  end
 end
 module GemVersion
@@ -63,6 +95,32 @@ module GemVersion
 end
+##################################################################################################################
+#
+opts = Optimist::options do
+  banner <<-EOS
+Usage:
+  hiddengems.rb [options] '<search term>'
+Where [options] are:
+EOS
+  opt :links, "Verify gem's URIs before scoring (slow)", type: :boolean, default: false
+  opt :downloads, "Weight of download count", type: :integer, default: 1
+  opt :versions, "Weight of version count", type: :integer, default: 1
+  opt :latest_version_date, "Weight of latest version date", type: :integer, default: 1
+  opt :avg_version_date, "Weight of average version date", type: :integer, default: 1
+  opt :download_slope, "Weight of download slope", type: :integer, default: 1
+  opt :rejectfilter, "Regular Expression of gem names to reject", type: :string
+end
+puts "Option values:"
+ap opts
+searchTerm = ARGV[0]
+puts "\nSearch term: #{searchTerm}\n"
 curpage = 1
 gems = Array.new
 gemdict = Hash.new
@@ -92,32 +150,55 @@ until curpage == 0
   puts "looking for more..." if curpage != 0
 end
+puts "\nSearch term '#{searchTerm}' returned #{gems.size} hits"
+if opts[:rejectfilter_given]
+  puts "Filtering out unwanted gems with #{opts[:rejectfilter]}"
+  re = Regexp.compile(opts[:rejectfilter])
+  gems.reject! {|each|
+    re.match(each.name)
+  }
+  puts "\n#{gems.size} remaining gems after reject filter"
+end
-puts "Search term '#{searchTerm}' returned #{gems.size} hits"
+if opts[:links]
+  puts "\nValidating URIs..."
+  gems.reject! {|each|
+    if each.uriScore == 0.0
+      puts "Rejecting #{each.name} due to lack of valid URIs"
+      true
+    else
+      false
+    end
+  }
+  puts "\n#{gems.size} remaining gems after link validation"
+end
+puts "\nCalculating rankings..."
 data = Hash.new
-data[:by_downloads] = gems.sort {|a,b| b.downloads <=> a.downloads}.collect {|each| "#{each.name} (#{each.downloads})"}
-data[:by_versions] = gems.sort {|a,b| b.totalVersions <=> a.totalVersions}.collect {|each| "#{each.name} (#{each.totalVersions})"}
-data[:by_latest_version_date] = gems.sort {|a,b| b.latestVersionDate <=> a.latestVersionDate}.collect {|each| "#{each.name} (#{each.latestVersionDate.year})"}
-data[:by_avg_version_date] = gems.sort {|a,b| b.avgDateOfAllVersions<=> a.avgDateOfAllVersions}.collect {|each| "#{each.name} (#{each.avgDateOfAllVersions.year})"}
-data[:by_download_slope] = gems.sort {|a,b| b.downloadSlope <=> a.downloadSlope}.collect {|each| "#{each.name} (#{each.downloadSlope.round})"}
+data[:downloads] = gems.sort {|a,b| b.downloads <=> a.downloads}.collect {|each| "#{each.name} (#{each.downloads})"}
+data[:versions] = gems.sort {|a,b| b.totalVersions <=> a.totalVersions}.collect {|each| "#{each.name} (#{each.totalVersions})"}
+data[:latest_version_date] = gems.sort {|a,b| b.latestVersionDate <=> a.latestVersionDate}.collect {|each| "#{each.name} (#{each.latestVersionDate.year})"}
+data[:avg_version_date] = gems.sort {|a,b| b.avgDateOfAllVersions<=> a.avgDateOfAllVersions}.collect {|each| "#{each.name} (#{each.avgDateOfAllVersions.year})"}
+data[:download_slope] = gems.sort {|a,b| b.downloadSlope <=> a.downloadSlope}.collect {|each| "#{each.name} (#{each.downloadSlope.round})"}
 report = Array.new
 rank = 0
 gems.each_index {|index|
   record = Hash.new
   record[:rank] = rank
-  [:by_downloads,:by_versions,:by_latest_version_date,:by_avg_version_date,:by_download_slope].each {|column|
+  data.keys.each {|column|
     record[column] = data[column][index]
-    gemdict[data[column][index].split(' (').first][:rank] += index
+    gemdict[data[column][index].split(' (').first][:rank] += index * opts[column]
   }
   report.push(record)
   rank += 1
 }
+puts
+puts "Individual Rankings (best -> worst)"
+puts  TTY::Table.new(header: [:rank]+data.keys.collect {|each| "#{each} (#{opts[each]})" }, rows: report).render(:ascii, resize: false, column_widths: [5] + [30] * data.keys.size)
-puts "Simple Rankings"
-puts  TTY::Table.new(header: [:rank]+data.keys, rows: report).render(:ascii, resize: false, column_widths: [5,30,30,30,30,30])
-puts "\nVitality Ranking:"
+puts "\nRank sums (best -> worst)"
 puts TTY::Table.new(header: [:score, :name,:info], rows: gems.sort {|a,b| a.rank<=>b.rank}.collect {|each| [each.rank,each.name,each.info]}).render(:ascii,  column_widths: [5,35,100],resize: false)
 puts

data/hiddengems.gemspec CHANGED Viewed

@@ -9,9 +9,10 @@ Gem::Specification.new do |spec|
   spec.email         = ["dparker@svdasein.org"]
   spec.summary       = "Search for ruby gems with a search term (like bestgems.org) with better ranking"
-  spec.description   = "Provides a cli that lets you search rubygems for a search term.  Displays multiple ranking criteria with emphasis on project vitality"
+  spec.description   = "Provides a cli that lets you search rubygems for a search term.  Displays multiple ranking criteria with emphasis on project vitality. Alternative to rubygems.org"
   spec.homepage      = "https://gitlab.com/svdasein/hiddengems"
   spec.required_ruby_version = ">= 2.4.0"
+  spec.license = 'MIT'
   spec.metadata["allowed_push_host"] = "https://rubygems.org"
@@ -33,8 +34,11 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'json'
   spec.add_dependency 'time'
   spec.add_dependency 'faraday'
+  spec.add_dependency 'faraday-follow_redirects'
+  spec.add_dependency 'optimist'
   spec.add_dependency 'ostruct'
   spec.add_dependency 'tty-table'
+  spec.add_dependency 'awesome_print'
   spec.add_dependency 'pry-rescue'
   spec.add_dependency 'linear_regression_trend'

data/img/2022-09-04_12-37.png ADDED Viewed

Binary file

data/img/2022-09-04_13-40.png ADDED Viewed

Binary file

data/img/2022-09-04_13-51.png ADDED Viewed

Binary file

data/lib/hiddengems/version.rb CHANGED Viewed

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 module Hiddengems
-  VERSION = "0.2.3"
+  VERSION = "0.3.2"
 end

metadata CHANGED Viewed

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hiddengems
 version: !ruby/object:Gem::Version
-  version: 0.2.3
+  version: 0.3.2
 platform: ruby
 authors:
 - Dave Parker
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-08-30 00:00:00.000000000 Z
+date: 2022-09-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: json
@@ -52,6 +52,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: faraday-follow_redirects
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: optimist
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: ostruct
   requirement: !ruby/object:Gem::Requirement
@@ -80,6 +108,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: awesome_print
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: pry-rescue
   requirement: !ruby/object:Gem::Requirement
@@ -109,7 +151,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 description: Provides a cli that lets you search rubygems for a search term.  Displays
-  multiple ranking criteria with emphasis on project vitality
+  multiple ranking criteria with emphasis on project vitality. Alternative to rubygems.org
 email:
 - dparker@svdasein.org
 executables:
@@ -119,16 +161,21 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - Gemfile
+- LICENSE
 - README.md
 - Rakefile
 - bin/console
 - bin/setup
 - exe/hiddengems.rb
 - hiddengems.gemspec
+- img/2022-09-04_12-37.png
+- img/2022-09-04_13-40.png
+- img/2022-09-04_13-51.png
 - lib/hiddengems.rb
 - lib/hiddengems/version.rb
 homepage: https://gitlab.com/svdasein/hiddengems
-licenses: []
+licenses:
+- MIT
 metadata:
   allowed_push_host: https://rubygems.org
   homepage_uri: https://gitlab.com/svdasein/hiddengems