hubba 0.6.1 → 0.6.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 122527cdc3cddb68e2c54a6e51fcfb77c2c91c0c
-  data.tar.gz: 6bb52baff0536f55ad969885fc8500b031457e44
+  metadata.gz: 0c7c4d9d7af183756855559bc03b8910bdcb1ace
+  data.tar.gz: 4cfd725ee9c1a3138370cbcb61e1bd900d689012
 SHA512:
-  metadata.gz: df4e036a205be30371ec9de85e8085717235922fcc21e22f9f516cc40481a2a8cf6a9b0edd0a37cb24759059a08b5ea74957f41f89139bf9e1a860ed20029e3b
-  data.tar.gz: 698fc0d240aa6c6f8b125d1ae004beacf5ae97c220eb42d4fd542a49a656ece101e3a9fd9e740eaefb63e7f03c132458872f4ea5949cd30e2eceb9ccc3bb9e54
+  metadata.gz: a6cb53226a5a73777bc9ad86d82a22a39ea56710514233d5530b489d9fcce1bd9cc67b65f6d41f488d3ee4a266e0be5cbcbeef9231cef9007427928d60785214
+  data.tar.gz: 38395c69b0e45a174a1629029803f41b3e5d1ca7dba27d893dfcebd70f59ed4f17e60e6ec1715c331b61d8e49ef727de564aba14a03b0dfe75a6c15850e3bdc8

data/Manifest.txt

@@ -3,12 +3,14 @@ Manifest.txt
 README.md
 Rakefile
 lib/hubba.rb
-lib/hubba/client.rb
 lib/hubba/config.rb
+lib/hubba/folio.rb
 lib/hubba/github.rb
+lib/hubba/hubba.rb
 lib/hubba/reports.rb
-lib/hubba/reposet.rb
 lib/hubba/stats.rb
+lib/hubba/update.rb
+lib/hubba/update_traffic.rb
 lib/hubba/version.rb
 test/helper.rb
 test/stats/jekyll~minima.json

data/README.md

@@ -10,8 +10,255 @@ hubba gem - (yet) another (lite) GitHub HTTP API client / library
 
 ## Usage
 
-TBD
 
+### Step 0:  Secrets, Secrets, Secrets - Your Authentication Token
+
+Note: Set your GitHub env credentials (personal access token preferred) e.g.
+
+```
+set HUBBA_TOKEN=abcdef0123456789
+#   - or -
+set HUBBA_USER=you
+set HUBBA_PASSWORD=topsecret
+```
+
+
+### Step 1: Get a list of all your repos
+
+Use the GitHub API to get a list of all your repos:
+
+``` ruby
+require 'hubba'
+
+h = Hubba.reposet( 'geraldb' )
+pp h
+
+File.open( './repos.yml', 'w' ) { |f| f.write( h.to_yaml ) }
+```
+
+resulting in:
+
+``` yaml
+geraldb (11):
+- austria
+- catalog
+- chelitas
+- geraldb.github.io
+- logos
+- sandbox
+- talks
+- web-proxy-win
+- webcomponents
+- webpub-reader
+- wine.db.tools
+
+openfootball (41):
+- africa-cup
+- austria
+- club-world-cup
+- clubs
+- confed-cup
+- copa-america
+- copa-libertadores
+- copa-sudamericana
+- deutschland
+# ...
+```
+
+
+Note: If you have more than one account (e.g. an extra robot account or such)
+you can add them along e.g.
+
+
+``` ruby
+h = Hubba.reposet( 'geraldb', 'yorobot' )
+pp h
+```
+
+
+Note: By default all your repos from organizations get auto-included -
+use the `orgs: false` option to turn off auto-inclusion.
+
+Note: By default all (personal) repos, that is, repos in your primary (first)
+account that are forks get auto-excluded.
+
+
+
+### Step 2: Update repo statistics (daily / weekly / monthly)
+
+
+Use `update_stats` to
+to get the latest commit, star count and more for all your repos
+listed in `./repos.yml` via the GitHub API:
+
+``` ruby
+Hubba.update_stats( './repos.yml' )
+```
+
+Note: By default the datafiles (one per repo)
+get stored in the `./data` directory.
+
+
+
+
+### Step 3: Generate some statistics / reports
+
+
+Hubba has four built-in reports (for now):
+
+- `ReportSummary`  - A-Z list of your repos by orgs with stars and size in kb
+- `ReportStars`    - your repos ranked by stars
+- `ReportTimeline` - your repos in reverse chronological order by creation
+- `ReportUpdates`  - your repos in reverse chronological order by last commit
+
+
+If you only generate a single report, use:
+
+``` ruby
+report = Hubba::ReportSummary.new( './repos.yml' )
+report.save( './SUMMARY.md' )
+```
+
+
+If you generate more reports, (re)use the in-memory statistics cache / object:
+
+``` ruby
+stats = Hubba.stats( './repos.yml' )
+
+report = Hubba::ReportSummary.new( stats )
+report.save( './SUMMARY.md' )
+
+report = Hubba::ReportStars.new( stats )
+report.save( './STARS.md' )
+
+report = Hubba::ReportTimeline.new( stats )
+report.save( './TIMELINE.md' )
+
+report = Hubba::ReportUpdates.new( stats )
+report.save( './UPDATES.md' )
+```
+
+
+### Report Examples
+
+#### Report Example - Summary
+
+A-Z list of your repos by orgs with stars and size in kb.
+Results in:
+
+---
+
+> 593 repos @ 83 orgs
+>
+> ### geraldb _(11)_
+>
+> **austria** ★1 (552 kb) · **catalog** ★3 (156 kb) · **chelitas** ★1 (168 kb) · **geraldb.github.io** ★1 (520 kb) · **logos** ★1 (363 kb) · **sandbox** ★2 (529 kb) · **talks** ★200 (16203 kb) · **web-proxy-win** ★8 (152 kb) · **webcomponents** ★1 (164 kb) · **webpub-reader** ★3 (11 kb) · **wine.db.tools** ★1 (252 kb)
+>
+> ...
+
+---
+
+(Live Example - [`SUMMARY.md`](https://github.com/yorobot/backup/blob/master/SUMMARY.md))
+
+
+#### Report Example - Stars
+
+Your repos ranked by stars. Results in:
+
+---
+
+> 593 repos @ 83 orgs
+>
+> 1. ★2936 **openblockchains/awesome-blockchains** (2514 kb)
+> 2. ★851 **planetjekyll/awesome-jekyll-plugins** (148 kb)
+> 3. ★604 **factbook/factbook.json** (7355 kb)
+> 4. ★593 **openfootball/football.json** (2135 kb)
+> 5. ★570 **openmundi/world.db** (1088 kb)
+> 6. ★552 **openblockchains/programming-blockchains** (552 kb)
+> 7. ★547 **mundimark/awesome-markdown** (83 kb)
+> 8. ★532 **planetjekyll/awesome-jekyll** (110 kb)
+> 9. ★489 **cryptocopycats/awesome-cryptokitties** (4154 kb)
+> 10. ★445 **openfootball/world-cup** (638 kb)
+>
+> ...
+
+---
+
+(Live Example: [`STARS.md`](https://github.com/yorobot/backup/blob/master/STARS.md))
+
+
+#### Report Example - Timeline
+
+Your repos in reverse chronological order by creation.
+Results in:
+
+---
+
+> 593 repos @ 83 orgs
+>
+> ## 2020
+>
+> ### 9
+>
+> - 2020-09-18 ★1 **yorobot/workflow** (83 kb)
+>
+> ### 6
+>
+> - 2020-06-27 ★2 **yorobot/sport.db.more** (80 kb)
+> - 2020-06-24 ★1 **yorobot/stage** (554 kb)
+> - 2020-06-11 ★1 **yorobot/cache.csv** (336 kb)
+>
+> ...
+
+---
+
+(Live Example: [`TIMELINE.md`](https://github.com/yorobot/backup/blob/master/TIMELINE.md))
+
+
+
+#### Report Example - Updates
+
+Your repos in reverse chronological order by last commit. Results in:
+
+---
+
+> 593 repos @ 83 orgs
+>
+> committed / pushed / updated / created
+>
+> - (1d) **yorobot/backup** ★4 - 2020-10-08 (=/=) / 2020-10-08 (=) / 2020-10-08 / 2015-04-04 - ‹› (1595 kb)
+> - (1d) **yorobot/logs** ★1 - 2020-10-08 (=/=) / 2020-10-08 (=) / 2020-10-08 / 2016-09-13 - ‹› (172 kb)
+> - (1d) **rubycoco/git** ★9 - 2020-10-08 (=/=) / 2020-10-08 (=) / 2020-10-08 / 2015-11-16 - ‹› (88 kb)
+> - (1d) **openfootball/football.json** ★593 - 2020-10-08 (=/=) / 2020-10-08 (=) / 2020-10-08 / 2015-09-17 - ‹› (2135 kb)
+> - (2d) **yorobot/workflow** ★1 - 2020-10-07 (=/=) / 2020-10-07 (=) / 2020-10-07 / 2020-09-18 - ‹› (83 kb)
+> - (2d) **rubycoco/webclient** ★5 - 2020-10-07 (=/=) / 2020-10-07 (=) / 2020-10-07 / 2012-06-02 - ‹› (39 kb)
+> - (3d) **footballcsv/belgium** ★1 - 2020-10-06 (=/=) / 2020-10-06 (=) / 2020-10-06 / 2014-07-25 - ‹› (314 kb)
+> - (3d) **footballcsv/england** ★105 - 2020-10-06 (=/=) / 2020-10-06 (=) / 2020-10-06 / 2014-07-23 - ‹› (8666 kb)
+> - (3d) **footballcsv/austria** ★1 - 2020-10-06 (=/=) / 2020-10-06 (=) / 2020-10-06 / 2018-07-16 - ‹› (91 kb)
+> - (3d) **footballcsv/espana** ★15 - 2020-10-06 (=/=) / 2020-10-06 (=) / 2020-10-06 / 2014-07-23 - ‹› (1107 kb)
+> - (3d) **footballcsv/deutschland** ★5 - 2020-10-06 (=/=) / 2020-10-06 (=) / 2020-10-06 / 2014-07-25 - ‹› (1343 kb)
+>
+> ...
+
+---
+
+(Live Example: [`UPDATES.md`](https://github.com/yorobot/backup/blob/master/UPDATES.md))
+
+
+
+That's all for now.
+
+
+
+## Installation
+
+Use
+
+    gem install hubba
+
+or add the gem to your Gemfile
+
+    gem 'hubba'
 
 
 ## License

data/lib/hubba/folio.rb

@@ -0,0 +1,60 @@
+module Hubba
+
+class Folio    # todo/check: use a different name e.g (Port)Folio, Cache, Summary, (Data)Base, Census, Catalog, Collection, Index, Register or such???
+class Repo  ## (nested) class
+
+  attr_reader :owner,
+              :name
+
+  def initialize( owner, name )
+    @owner = owner   ## rename to login, username - why? why not?
+    @name  = name    ## rename to reponame ??
+  end
+
+  def full_name() "#{owner}/#{name}"; end
+
+  def stats
+    ## note: load stats on demand only (first access) for now - why? why not?
+    @stats ||= begin
+                 stats = Stats.new( full_name )
+                 stats.read
+                 stats
+               end
+  end
+
+  def diff
+    @diff ||= stats.calc_diff_stars( samples: 3, days: 30 )
+  end
+end  # (nested) class Repo
+
+
+attr_reader :orgs, :repos
+
+def initialize( h )
+  @orgs     = []    # orgs and users -todo/check: use better name - logins or owners? why? why not?
+  @repos    = []
+  add( h )
+
+  puts "#{@repos.size} repos @ #{@orgs.size} orgs"
+end
+
+#############
+## private helpes
+def add( h )   ## add repos.yml set
+  h.each do |org_with_counter, names|
+    ## remove optional number from key e.g.
+    ##   mrhydescripts (3)    =>  mrhydescripts
+    ##   footballjs (4)       =>  footballjs
+    ##   etc.
+    org = org_with_counter.sub( /\([0-9]+\)/, '' ).strip
+    repos = []
+    names.each do |name|
+      repo = Repo.new( org, name )
+      repos << repo
+    end
+    @orgs << [org, repos]
+    @repos += repos
+  end
+end
+end  # class Folio
+end  # module Hubba

data/lib/hubba/github.rb

@@ -1,45 +1,59 @@
 module Hubba
 
 
-class Resource
-  attr_reader :data
-  def initialize( data )
-    @data = data
+class Github
+  BASE_URL = 'https://api.github.com'
+
+  ###############
+  ## (nested) classes for "wrapped" response (parsed json body)
+  class Resource
+    attr_reader :data
+    def initialize( data )
+      @data = data
+    end
   end
-end
 
-class Repos < Resource
-  def names
-    ## sort by name
-    data.map { |item| item['name'] }.sort
+  class Repos < Resource
+    def names
+      ## sort by name
+      data.map { |item| item['name'] }.sort
+    end
   end
-end
 
-class Orgs < Resource
-  def logins
-    ## sort by name
-    data.map { |item| item['login'] }.sort
+  class Orgs < Resource
+    def logins
+      ## sort by name
+      data.map { |item| item['login'] }.sort
+    end
+    alias_method :names, :logins   ## add name alias - why? why not?
   end
-  alias_method :names, :logins   ## add name alias - why? why not?
-end
-
 
 
-class Github
 
-def initialize
-   @client = if Hubba.configuration.token
-               Client.new( token: Hubba.configuration.token )
-             elsif Hubba.configuration.user &&
-                   Hubba.configuration.password
-               Client.new( user:     Hubba.configuration.user,
-                           password: Hubba.configuration.password )
-             else
-               Client.new
-             end
+def initialize( token: nil,
+                user:  nil,
+                password: nil )
+   @token    = nil
+   @user     = nil
+   @password = nil
+
+   if token                         ## 1a) give preference to passed in token
+     @token     = token
+   elsif user && password           ## 1b)  or passed in user/password credentials
+     @user      = user
+     @password  = password
+   elsif Hubba.config.token         ## 2a) followed by configured (or env) token
+     @token     = Hubba.config.token
+   elsif Hubba.config.user && Hubba.config.password   ## 2b)
+     @user      = Hubba.config.user
+     @password  = Hubba.config.password
+   else                             ## 3)
+     ## no token or credentials passed in or configured
+   end
 end
 
 
+
 def user( name )
   Resource.new( get "/users/#{name}" )
 end
@@ -77,61 +91,120 @@ def repo( full_name )   ## full_name (handle) e.g. henrythemes/jekyll-starter-th
   Resource.new( get "/repos/#{full_name}" )
 end
 
+def repo_languages( full_name )
+  Resource.new( get "/repos/#{full_name}/languages" )
+end
+
+def repo_topics( full_name )
+  ## note: requires "api preview" accept headers (overwrites default v3+json)
+  ##  e.g. application/vnd.github.mercy-preview+json
+  Resource.new( get( "/repos/#{full_name}/topics", preview: 'mercy' ) )
+end
+
+
 def repo_commits( full_name )
   Resource.new( get "/repos/#{full_name}/commits" )
 end
 
 
+def repo_traffic_clones( full_name )
+ # Get repository clones
+ # Get the total number of clones and breakdown per day or week
+ #   for the last 14 days.
+ # Timestamps are aligned to UTC midnight of the beginning of the day or week.
+ # Week begins on Monday.
+ Resource.new( get "/repos/#{full_name}/traffic/clones" )
+end
 
-####
-# more
-def update( obj )
-  if obj.is_a?( Stats )
-    stats = obj
-    full_name = stats.full_name
-    puts "[update 1/2] fetching repo >#{full_name}<..."
-    repo    = repo( full_name )
-    puts "[update 2/2] fetching repo >#{full_name}< commits ..."
-    commits = repo_commits( full_name )
-
-    stats.update( repo, commits )
-  else
-    raise ArgumentError, "unknown source object passed in - expected Hubba::Stats; got #{obj.class.name}"
-  end
+def repo_traffic_views( full_name )
+ # Get page views
+ # Get the total number of views and breakdown per day or week
+ #  for the last 14 days.
+ # Timestamps are aligned to UTC midnight of the beginning of the day or week.
+ # Week begins on Monday.
+ Resource.new( get "/repos/#{full_name}/traffic/views" )
 end
 
 
-def update_stats( h )    ## todo/fix: change to Reposet - why? why not???
-    h.each do |org_with_counter,names|
+def repo_traffic_popular_paths( full_name )
+ # Get top referral paths
+ # Get the top 10 popular contents over the last 14 days.
+ Resource.new( get "/repos/#{full_name}/traffic/popular/paths" )
+end
 
-      ## remove optional number from key e.g.
-      ##   mrhydescripts (3)    =>  mrhydescripts
-      ##   footballjs (4)       =>  footballjs
-      ##   etc.
+def repo_traffic_popular_referrers( full_name )
+ # Get top referral sources
+ # Get the top 10 referrers over the last 14 days.
+ Resource.new( get "/repos/#{full_name}/traffic/popular/referrers" )
+end
 
-      org = org_with_counter.sub( /\([0-9]+\)/, '' ).strip
 
-      ## puts "  -- #{key_with_counter} [#{key}] --"
 
-      names.each do |name|
-        full_name = "#{org}/#{name}"
 
-        ## puts "  fetching stats #{count+1}/#{repo_count} - >#{full_name}<..."
-        stats = Stats.new( full_name )
-        stats.read
+private
+def get( request_uri, preview: nil )
+
+  puts "GET #{request_uri}"
+
+  ## note: request_uri ALWAYS starts with leading /, thus use + for now!!!
+  #          e.g. /users/geraldb
+  #               /users/geraldb/repos
+  url = BASE_URL + request_uri
+
+
+  headers = {}
+  ## add default headers if nothing (custom) set / passed-in
+  headers['User-Agent'] = "ruby/hubba v#{VERSION}"          ## required by GitHub API
+  headers['Accept']     =  if preview   # e.g. mercy or ???
+                             "application/vnd.github.#{preview}-preview+json"
+                           else
+                             'application/vnd.github.v3+json'  ## default - recommend by GitHub API
+                           end
+
+  auth = []
+  ## check if credentials (user/password) present - if yes, use basic auth
+  if @token
+    puts "  using (personal access) token - starting with: #{@token[0..6]}**********"
+    headers['Authorization'] = "token #{@token}"
+    ## token works like:
+    ##  curl -H 'Authorization: token my_access_token' https://api.github.com/user/repos
+  elsif @user && @password
+    puts "  using basic auth - user: #{@user}, password: ***"
+    ## use credential auth "tuple" (that is, array with two string items) for now
+    ##  or use Webclient::HttpBasicAuth or something - why? why not?
+    auth = [@user, @password]
+    # req.basic_auth( @user, @password )
+  else
+    puts "  using no credentials (no token, no user/password)"
+  end
 
-        update( stats )   ## fetch & update stats
+  res = Webclient.get( url,
+                       headers: headers,
+                       auth:    auth )
 
-        stats.write
-      end
-    end
-end
+  # Get specific header
+  # response["content-type"]
+  # => "text/html; charset=UTF-8"
 
+  # Iterate all response headers.
+  # puts "HTTP HEADERS:"
+  # res.headers.each do |key, value|
+  #  puts "  #{key}: >#{value}<"
+  # end
+  # puts
 
-private
-def get( request_uri )
-  @client.get( request_uri )
-end
+  # => "location => http://www.google.com/"
+  # => "content-type => text/html; charset=UTF-8"
+  # ...
+
+  if res.status.ok?
+    res.json
+  else
+    puts "!! HTTP ERROR: #{res.status.code} #{res.status.message}:"
+    pp res.raw
+    exit 1
+  end
+end  # method get
 
 end  # class Github
 end  # module Hubba

data/lib/hubba/{reposet.rb → hubba.rb}

@@ -1,100 +1,13 @@
 module Hubba
 
 
-def self.update_stats( host_or_path='./repos.yml' )  ## move to reposet e.g. Reposet#update_status!!!!
-  h = if hash_or_path.is_a?( String )    ## assume it is a file path!!!
-        path = hash_or_path
-        YAML.load_file( path )
-      else
-        hash_or_path  # assume its a hash / reposet already!!!
-      end
-
-  gh = Github.new
-  gh.update_stats( h )
-end
-
-
-def self.stats( hash_or_path='./repos.yml' )   ## use read_stats or such - why? why not?
-  h = if hash_or_path.is_a?( String )    ## assume it is a file path!!!
-        path = hash_or_path
-        YAML.load_file( path )
-      else
-        hash_or_path  # assume its a hash / reposet already!!!
-      end
-
-  Summary.new( h )  ## wrap in "easy-access" facade / wrapper
-end
-
-
-
-class Summary    # todo/check: use a different name e.g (Data)Base, Census, Catalog, Collection, Index, Register or such???
-
-class Repo  ## (nested) class
-
-  attr_reader :owner,
-              :name
-
-  def initialize( owner, name )
-    @owner = owner   ## rename to login, username - why? why not?
-    @name  = name    ## rename to reponame ??
-  end
-
-  def full_name() "#{owner}/#{name}"; end
-
-  def stats
-    ## note: load stats on demand only (first access) for now - why? why not?
-    @stats ||= begin
-                 stats = Stats.new( full_name )
-                 stats.read
-                 stats
-               end
-  end
-
-  def diff
-    @diff ||= stats.calc_diff_stars( samples: 3, days: 30 )
-  end
-end  # (nested) class Repo
-
-
-attr_reader :orgs, :repos
-
-def initialize( hash )
-  @orgs     = []    # orgs and users -todo/check: use better name - logins or owners? why? why not?
-  @repos    = []
-  add( hash )
-
-  puts "#{@repos.size} repos @ #{@orgs.size} orgs"
-end
-
-#############
-## private helpes
-def add( hash )   ## add repos.yml set
-  hash.each do |org_with_counter, names|
-    ## remove optional number from key e.g.
-    ##   mrhydescripts (3)    =>  mrhydescripts
-    ##   footballjs (4)       =>  footballjs
-    ##   etc.
-    org = org_with_counter.sub( /\([0-9]+\)/, '' ).strip
-    repos = []
-    names.each do |name|
-      repo = Repo.new( org, name )
-      repos << repo
-    end
-    @orgs << [org, repos]
-    @repos += repos
-  end
-end
-end  # class Summary
-
-
-
 ## orgs  - include repos form org(anizations) too
 ## cache - save json response to cache_dir   - change to/use debug/tmp_dir? - why? why not?
 def self.reposet( *users, orgs: true,
                           cache: false )
   # users = [users]   if users.is_a?( String )  ### wrap in array if single user
 
-  gh = Hubba::Github.new
+  gh = Github.new
 
   forks = []
 
@@ -166,5 +79,19 @@ def self.reposet( *users, orgs: true,
 
   h
 end  ## method reposet
+
+
+def self.stats( hash_or_path='./repos.yml' )   ## use read_stats or such - why? why not?
+  h = if hash_or_path.is_a?( String )    ## assume it is a file path!!!
+        path = hash_or_path
+        YAML.load_file( path )
+      else
+        hash_or_path  # assume its a hash / reposet already!!!
+      end
+
+  Folio.new( h )  ## wrap in "easy-access" facade / wrapper
+end  ## method stats
+
+
 end # module Hubba
 

data/lib/hubba/stats.rb

@@ -223,12 +223,105 @@ module Hubba
 
 
 
-###############################
-## fetch / read / write methods
+##################
+## update
+
+def update_traffic( clones:    nil,
+                    views:     nil,
+                    paths:     nil,
+                    referrers: nil )
+
+  traffic = @data[ 'traffic' ] ||= {}
+
+  summary = traffic['summary'] ||= {}
+  history = traffic['history'] ||= {}
+
+
+  if views
+    raise ArgumentError, "Github::Resource expected; got #{views.class.name}"    unless views.is_a?( Github::Resource )
+=begin
+{"count"=>1526,
+ "uniques"=>287,
+ "views"=>
+[{"timestamp"=>"2020-09-27T00:00:00Z", "count"=>52, "uniques"=>13},
+ {"timestamp"=>"2020-09-28T00:00:00Z", "count"=>108, "uniques"=>28},
+ ...
+]}>
+=end
+
+    ## keep lastest (summary) record of last two weeks (14 days)
+    summary['views'] = { 'count'   => views.data['count'],
+                         'uniques' => views.data['uniques'] }
+
+    ## update history / day-by-day items / timeline
+    views.data['views'].each do |view|
+       # e.g. "2020-09-27T00:00:00Z"
+       timestamp = DateTime.strptime( view['timestamp'], '%Y-%m-%dT%H:%M:%S%z' )
+
+       item = history[ timestamp.strftime( '%Y-%m-%d' ) ] ||= {}   ## e.g. 2016-09-27
+       ## note: merge "in-place"
+       item.merge!( { 'views' => { 'count'   => view['count'],
+                                   'uniques' => view['uniques'] }} )
+    end
+  end
+
+  if clones
+    raise ArgumentError, "Github::Resource expected; got #{clones.class.name}"    unless clones.is_a?( Github::Resource )
+=begin
+ {"count"=>51,
+   "uniques"=>17,
+   "clones"=>
+    [{"timestamp"=>"2020-09-26T00:00:00Z", "count"=>1, "uniques"=>1},
+     {"timestamp"=>"2020-09-27T00:00:00Z", "count"=>2, "uniques"=>1},
+     ...
+    ]}
+=end
+
+    ## keep lastest (summary) record of last two weeks (14 days)
+    summary['clones'] = { 'count'   => clones.data['count'],
+                          'uniques' => clones.data['uniques'] }
+
+    ## update history / day-by-day items / timeline
+    clones.data['clones'].each do |clone|
+       # e.g. "2020-09-27T00:00:00Z"
+       timestamp = DateTime.strptime( clone['timestamp'], '%Y-%m-%dT%H:%M:%S%z' )
+
+       item = history[ timestamp.strftime( '%Y-%m-%d' ) ] ||= {}   ## e.g. 2016-09-27
+       ## note: merge "in-place"
+       item.merge!( { 'clones' => { 'count'   => clone['count'],
+                                    'uniques' => clone['uniques'] }} )
+    end
+  end
 
-    def update( repo, commits )   ## update stats / fetch data from github via api
-      raise ArgumentError, "Hubba::Resource expected; got #{repo.class.name}"      unless repo.is_a?( Resource )
-      raise ArgumentError, "Hubba::Resource expected; got #{commits.class.name}"   unless commits.is_a?( Resource )
+  if paths
+    raise ArgumentError, "Github::Resource expected; got #{paths.class.name}"    unless paths.is_a?( Github::Resource )
+=begin
+  [{"path"=>"/openfootball/england",
+  "title"=>
+   "openfootball/england: Free open public domain football data for England (and ...",
+  "count"=>394,
+  "uniques"=>227},
+=end
+   summary['paths'] = paths.data
+  end
+
+  if referrers
+    raise ArgumentError, "Github::Resource expected; got #{referrers.class.name}"    unless referrers.is_a?( Github::Resource )
+=begin
+  [{"referrer"=>"github.com", "count"=>327, "uniques"=>198},
+  {"referrer"=>"openfootball.github.io", "count"=>71, "uniques"=>54},
+  {"referrer"=>"Google", "count"=>5, "uniques"=>5},
+  {"referrer"=>"reddit.com", "count"=>4, "uniques"=>4}]
+=end
+    summary['referrers'] = referrers.data
+  end
+end  # method update_traffic
+
+
+    def update( repo,
+                 commits: nil,
+                 topics:  nil )   ## update stats / fetch data from github via api
+      raise ArgumentError, "Github::Resource expected; got #{repo.class.name}"      unless repo.is_a?( Github::Resource )
 
       ## e.g. 2015-05-11T20:21:43Z
       ## puts Time.iso8601( repo.data['created_at'] )
@@ -236,42 +329,82 @@ module Hubba
       @data['updated_at'] = repo.data['updated_at']
       @data['pushed_at']  = repo.data['pushed_at']
 
-      @data['size']       = repo.data['size']  # size in kb (kilobyte)
+      @data['size']       = repo.data['size']  # note: size in kb (kilobyte)
+
+      @data['description'] = repo.data['description']
+      @data['language']    = repo.data['language']  ## note: might be nil!!!
 
+
+
+      ########################################
+      ####  history / by date record
       rec = {}
 
-      puts "stargazers_count"
-      puts repo.data['stargazers_count']
       rec['stargazers_count'] = repo.data['stargazers_count']
+      rec['forks_count']      = repo.data['forks_count']
+
 
       today = Date.today.strftime( '%Y-%m-%d' )   ## e.g. 2016-09-27
       puts "add record #{today} to history..."
       pp rec      # check if stargazers_count is a number (NOT a string)
 
-      @data[ 'history' ] ||= {}
-      @data[ 'history' ][ today ] = rec
+      history = @data[ 'history' ] ||= {}
+      item    = history[ today ]   ||= {}
+      ## note: merge "in-place" (overwrite with new - but keep other key/value pairs if any e.g. pageviews, clones, etc.)
+      item.merge!( rec )
+
+
 
       ##########################
       ## also check / keep track of (latest) commit
-      puts "last commit/update:"
-      ## pp commits
-      commit = {
-        'committer' => {
-          'date' => commits.data[0]['commit']['committer']['date'],
-          'name' => commits.data[0]['commit']['committer']['name']
-        },
-        'author' => {
-          'date' => commits.data[0]['commit']['author']['date'],
-          'name' => commits.data[0]['commit']['author']['name']
-        },
-        'message' => commits.data[0]['commit']['message']
-      }
-
-      ## for now store only the latest commit (e.g. a single commit in an array)
-      @data[ 'commits' ] = [commit]
+      if commits
+        raise ArgumentError, "Github::Resource expected; got #{commits.class.name}"   unless commits.is_a?( Github::Resource )
+
+        puts "update - last commit:"
+        ## pp commits
+        commit = {
+          'committer' => {
+            'date' => commits.data[0]['commit']['committer']['date'],
+            'name' => commits.data[0]['commit']['committer']['name']
+          },
+          'author' => {
+            'date' => commits.data[0]['commit']['author']['date'],
+            'name' => commits.data[0]['commit']['author']['name']
+          },
+          'message' => commits.data[0]['commit']['message']
+        }
+
+        ## for now store only the latest commit (e.g. a single commit in an array)
+        @data[ 'commits' ] = [commit]
+      end
+
+      if topics
+        raise ArgumentError, "Github::Resource expected; got #{topics.class.name}"   unless topics.is_a?( Github::Resource )
+
+        puts "update - topics:"
+        ## e.g.
+        # {"names"=>
+        #   ["opendata",
+        #    "football",
+        #    "seriea",
+        #    "italia",
+        #    "italy",
+        #    "juve",
+        #    "inter",
+        #    "napoli",
+        #    "roma",
+        # "sqlite"]}
+        #
+        #  {"names"=>[]}
+
+        @data[ 'topics' ] = topics.data['names']
+      end
+
 
       pp @data
 
+
+
       ## reset (invalidate) cached values from data hash
       ##   use after reading or fetching
       @cache = {}
@@ -280,6 +413,8 @@ module Hubba
     end
 
 
+########################################
+## read / write methods / helpers
 
     def write
       basename = full_name.gsub( '/', '~' )   ## e.g. poole/hyde become poole~hyde

data/lib/hubba/update.rb

@@ -0,0 +1,44 @@
+module Hubba
+
+def self.update_stats( hash_or_path='./repos.yml' )  ## move to reposet e.g. Reposet#update_status!!!!
+  h = if hash_or_path.is_a?( String )    ## assume it is a file path!!!
+        path = hash_or_path
+        YAML.load_file( path )
+      else
+        hash_or_path  # assume its a hash / reposet already!!!
+      end
+
+    gh = Github.new
+
+    h.each do |org_with_counter,names|
+
+      ## remove optional number from key e.g.
+      ##   mrhydescripts (3)    =>  mrhydescripts
+      ##   footballjs (4)       =>  footballjs
+      ##   etc.
+      org = org_with_counter.sub( /\([0-9]+\)/, '' ).strip
+
+      ## puts "  -- #{key_with_counter} [#{key}] --"
+
+      names.each do |name|
+        full_name = "#{org}/#{name}"
+
+        ## puts "  fetching stats #{count+1}/#{repo_count} - >#{full_name}<..."
+        stats = Stats.new( full_name )
+        stats.read
+
+        puts "update >#{full_name}< [1/3] - fetching repo..."
+        repo    = gh.repo( full_name )
+        puts "update >#{full_name}< [2/3] - fetching repo commits ..."
+        commits = gh.repo_commits( full_name )
+        puts "update >#{full_name}< [3/3] - fetching repo topics ..."
+        topics  = gh.repo_topics( full_name )
+
+        stats.update( repo,
+                       commits: commits,
+                       topics:  topics )
+        stats.write
+      end
+    end
+end
+end # module Hubba

data/lib/hubba/update_traffic.rb

@@ -0,0 +1,52 @@
+module Hubba
+
+
+###
+## note: keep update traffic separate from update (basic) stats
+##         traffic stats require (personal access) token with push access!!
+
+def self.update_traffic( hash_or_path='./repos.yml' )  ## move to reposet e.g. Reposet#update_status!!!!
+  h = if hash_or_path.is_a?( String )    ## assume it is a file path!!!
+        path = hash_or_path
+        YAML.load_file( path )
+      else
+        hash_or_path  # assume its a hash / reposet already!!!
+      end
+
+    gh = Github.new
+
+    h.each do |org_with_counter,names|
+
+      ## remove optional number from key e.g.
+      ##   mrhydescripts (3)    =>  mrhydescripts
+      ##   footballjs (4)       =>  footballjs
+      ##   etc.
+      org = org_with_counter.sub( /\([0-9]+\)/, '' ).strip
+
+      ## puts "  -- #{key_with_counter} [#{key}] --"
+
+      names.each do |name|
+        full_name = "#{org}/#{name}"
+
+        ## puts "  fetching stats #{count+1}/#{repo_count} - >#{full_name}<..."
+        stats = Stats.new( full_name )
+        stats.read
+
+        puts "update >#{full_name}< [1/4] - fetching repo traffic clones..."
+        clones    = gh.repo_traffic_clones( full_name )
+        puts "update >#{full_name}< [2/4] - fetching repo traffic views..."
+        views     = gh.repo_traffic_views( full_name )
+        puts "update >#{full_name}< [3/4] - fetching repo traffic popular paths..."
+        paths     = gh.repo_traffic_popular_paths( full_name )
+        puts "update >#{full_name}< [4/4] - fetching repo traffic popular referrers..."
+        referrers = gh.repo_traffic_popular_referrers( full_name )
+
+        stats.update_traffic( clones:    clones,
+                              views:     views,
+                              paths:     paths,
+                              referrers: referrers )
+        stats.write
+      end
+    end
+end
+end # module Hubba

data/lib/hubba/version.rb

@@ -1,7 +1,7 @@
 module Hubba
   MAJOR = 0    ## todo: namespace inside version or something - why? why not??
   MINOR = 6
-  PATCH = 1
+  PATCH = 2
   VERSION = [MAJOR,MINOR,PATCH].join('.')
 
   def self.version

data/lib/hubba.rb

@@ -22,12 +22,14 @@ end
 # our own code
 require 'hubba/version'   # note: let version always go first
 require 'hubba/config'
-require 'hubba/client'
 require 'hubba/github'
 require 'hubba/stats'
 
 ## "higher level" porcelain services / helpers for easy (re)use
-require 'hubba/reposet'
+require 'hubba/folio'     ## "access layer" for reports
+require 'hubba/hubba'
+require 'hubba/update'
+require 'hubba/update_traffic'
 
 require 'hubba/reports'
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hubba
 version: !ruby/object:Gem::Version
-  version: 0.6.1
+  version: 0.6.2
 platform: ruby
 authors:
 - Gerald Bauer
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-10-09 00:00:00.000000000 Z
+date: 2020-10-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: webclient
@@ -72,12 +72,14 @@ files:
 - README.md
 - Rakefile
 - lib/hubba.rb
-- lib/hubba/client.rb
 - lib/hubba/config.rb
+- lib/hubba/folio.rb
 - lib/hubba/github.rb
+- lib/hubba/hubba.rb
 - lib/hubba/reports.rb
-- lib/hubba/reposet.rb
 - lib/hubba/stats.rb
+- lib/hubba/update.rb
+- lib/hubba/update_traffic.rb
 - lib/hubba/version.rb
 - test/helper.rb
 - test/stats/jekyll~minima.json

data/lib/hubba/client.rb

@@ -1,82 +0,0 @@
-module Hubba
-
-
-class Client
-
-BASE_URL = 'https://api.github.com'
-
-
-def initialize( token: nil,
-                user:  nil, password: nil )
-  ## add support for (personal access) token
-  @token     = token
-
-  ## add support for basic auth - defaults to no auth (nil/nil)
-  ##   remove - deprecated (use token) - why? why not?
-  @user      = user    ## use login like Oktokit - why? why not?
-  @password  = password
-end # method initialize
-
-
-
-def get( request_uri )
-  puts "GET #{request_uri}"
-
-  ## note: request_uri ALWAYS starts with leading /, thus use + for now!!!
-  #          e.g. /users/geraldb
-  #               /users/geraldb/repos
-  url = BASE_URL + request_uri
-
-  headers = {}
-  headers['User-Agent'] = 'ruby/hubba'                      ## required by GitHub API
-  headers['Accept']     = 'application/vnd.github.v3+json'  ## recommend by GitHub API
-
-  auth = []
-  ## check if credentials (user/password) present - if yes, use basic auth
-  if @token
-    puts "  using (personal access) token - starting with: #{@token[0..6]}**********"
-    headers['Authorization'] = "token #{@token}"
-    ## token works like:
-    ##  curl -H 'Authorization: token my_access_token' https://api.github.com/user/repos
-  elsif @user && @password
-    puts "  using basic auth - user: #{@user}, password: ***"
-    ## use credential auth "tuple" (that is, array with two string items) for now
-    ##  or use Webclient::HttpBasicAuth or something - why? why not?
-    auth = [@user, @password]
-    # req.basic_auth( @user, @password )
-  else
-    puts "  using no credentials (no token, no user/password)"
-  end
-
-  res = Webclient.get( url,
-                       headers: headers,
-                       auth:    auth )
-
-  # Get specific header
-  # response["content-type"]
-  # => "text/html; charset=UTF-8"
-
-  # Iterate all response headers.
-  # puts "HTTP HEADERS:"
-  # res.headers.each do |key, value|
-  #  puts "  #{key}: >#{value}<"
-  # end
-  # puts
-
-  # => "location => http://www.google.com/"
-  # => "content-type => text/html; charset=UTF-8"
-  # ...
-
-  if res.status.ok?
-    res.json
-  else
-    puts "!! HTTP ERROR: #{res.status.code} #{res.status.message}:"
-    pp res.raw
-    exit 1
-  end
-end  # methdo get
-
-end  ## class Client
-
-
-end # module Hubba