football-sources 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: b068838936cfdba0eae277fe5f44c957f2281655
-  data.tar.gz: da3b4be77ac9661aaaa469580d38d37e68f3ae08
+SHA256:
+  metadata.gz: 42b02267a8c37a2b3a38662941d13ac8e61e622c8b45dc0c06a28453e272b2dc
+  data.tar.gz: e26f5cf47b6c0390b4f5d99b85919852eb46e7822201e4d6eba52315dc683e44
 SHA512:
-  metadata.gz: 9179c04e869df95a9b473aeabe1397f19c29108a1eac6e6d3fe149f2cb3b5f6967002d459e4ba013bfae26a22748724030805ebc591c333a389e53eb5d05576a
-  data.tar.gz: 31c7a7a673ea694996ddcf4ae550aa0904c2ba5391f5af3bfe09c9bf1f27f2af2ba0d8ddee0389da0b11a739b7bedb9e35055486a834a338072049d5a0f758e8
+  metadata.gz: 24038df8393a7ab1635cd6a5a9376e20c29ba077fbe3db63a917a5f59154aec50c1f6f99c68e1f51cbbd1c28ce07412e89387ae9576e8fc22c2bc872df689ae0
+  data.tar.gz: d27328142ffed18c30078e66c5a313a9df7a12be714401cba36cfa6752c08d0f1c158a796e87c50202ba08ae5ab2212061db494982656b378c3374c3e1cd23c5

data/CHANGELOG.md

@@ -1,4 +1,6 @@
-### 0.0.1 / 2020-11-09
-
-* Everything is new. First release.
-
+### 0.2.0
+
+### 0.0.1 / 2020-11-09
+
+* Everything is new. First release.
+

data/Manifest.txt

@@ -2,26 +2,8 @@ CHANGELOG.md
 Manifest.txt
 README.md
 Rakefile
+bin/fbgen
 lib/football-sources.rb
-lib/football-sources/apis.rb
-lib/football-sources/apis/config.rb
-lib/football-sources/apis/convert.rb
-lib/football-sources/apis/convert_cl.rb
-lib/football-sources/apis/mods.rb
-lib/football-sources/apis/stat.rb
-lib/football-sources/fbref.rb
-lib/football-sources/fbref/build.rb
-lib/football-sources/fbref/config.rb
-lib/football-sources/fbref/convert.rb
+lib/football-sources/process.rb
 lib/football-sources/version.rb
-lib/football-sources/worldfootball.rb
-lib/football-sources/worldfootball/build.rb
-lib/football-sources/worldfootball/config.rb
-lib/football-sources/worldfootball/convert.rb
-lib/football-sources/worldfootball/convert_reports.rb
-lib/football-sources/worldfootball/jobs.rb
-lib/football-sources/worldfootball/mods.rb
-lib/football-sources/worldfootball/vacuum.rb
 lib/football/sources.rb
-test/helper.rb
-test/test_version.rb

data/README.md

@@ -1,169 +1,30 @@
-# football-sources - get football data via web pages or web api (json) calls
-
-
-* home  :: [github.com/sportdb/sport.db](https://github.com/sportdb/sport.db)
-* bugs  :: [github.com/sportdb/sport.db/issues](https://github.com/sportdb/sport.db/issues)
-* gem   :: [rubygems.org/gems/football-sources](https://rubygems.org/gems/football-sources)
-* rdoc  :: [rubydoc.info/gems/football-sources](http://rubydoc.info/gems/football-sources)
-* forum :: [groups.google.com/group/opensport](https://groups.google.com/group/opensport)
-
-
-
-## Usage
-
-
-### Source #1 - `football-data.org` - ur src for machine readable football data
-
-[Daniel Freitag](https://www.football-data.org/about)'s dev-friendly football API
-offers a free to use plan
-for 12 leagues (API key sign-up and use required).
-See [`football-data.org` »](https://www.football-data.org)
-
-
-**Step 0 - Setup Secrets**
-
-Set the API key / token in the env(ironement).
-Example:
-
-```
-set FOOTBALLDATA=1234567890abcdef1234567890abcdef
-```
-
-
-**Step 1 - Download Match Schedules**
-
-Download the match schedules (in json) via api calls
-to your (local) web cache (in `~/.cache`).
-Note: The free trier has a 10 request/minute limit,
-thus, sleep/wait 10 secs after every request
-(should result in ~6 requests/minute).
-
-
-``` ruby
-require 'football/sources'
-
-
-# download up (ongoing) 2020 or 2020/21 seasons
-Webget.config.sleep  = 10
-
-Footballdata.schedule( league: 'eng.1', season: '2020/21' )
-Footballdata.schedule( league: 'eng.2', season: '2020/21' )
-
-Footballdata.schedule( league: 'de.1', season: '2020/21' )
-Footballdata.schedule( league: 'es.1', season: '2020/21' )
-
-Footballdata.schedule( league: 'fr.1', season: '2020/21' )
-Footballdata.schedule( league: 'it.1', season: '2020/21' )
-
-Footballdata.schedule( league: 'nl.1', season: '2020/21' )
-Footballdata.schedule( league: 'pt.1', season: '2020/21' )
-
-Footballdata.schedule( league: 'cl',   season: '2020/21' )
-
-# note: Brasileirão - season is a calendar year (NOT an academic year)
-Footballdata.schedule( league: 'br.1', season: '2020' )
-```
-
-Note: You can find all downloaded match schedules
-in your (local) web cache (in `~/.cache/api.football-data.org`) as pretty printed json documents.
-
-
-
-
-**Step 2 - Convert (Cached) Match Schedules to Records**
-
-Convert the (cached) match schedules
-in JSON to the one-line, one-match & one-file, one-season
-"standard" [Football.CSV format](https://github.com/footballcsv). Example:
-
-``` ruby
-require 'football/sources'
-
-['eng.1', 'eng.2',
- 'de.1',
- 'es.1',
- 'fr.1',
- 'it.1',
- 'nl.1',
- 'pt.1',
- 'cl',
-].each do |league|
-  Footballdata.convert( league: league, season: '2020/21' )
-end
-
-Footballdata.convert( league: 'br.1', season: '2020' )
-```
-
-Note: By default all datasets get written into the `./o`
-directory.  Use `Footballdata.config.convert.out_dir`
-to change the output directory.
-
-The English Premier League (`eng.1`) results in `./o/2020-21/eng.1.csv`:
-
-```
-Matchday,Date,Team 1,FT,HT,Team 2,Comments
-1,Sun Sep 13 2020,Manchester City FC,(*),,Aston Villa FC,postponed
-1,Sun Sep 13 2020,Burnley FC,(*),,Manchester United FC,postponed
-1,Sat Sep 12 2020,Fulham FC,0-3,0-1,Arsenal FC,
-1,Sat Sep 12 2020,Crystal Palace FC,1-0,1-0,Southampton FC,
-1,Sat Sep 12 2020,Liverpool FC,4-3,3-2,Leeds United FC,
-1,Sat Sep 12 2020,West Ham United FC,0-2,0-0,Newcastle United FC,
-1,Sun Sep 13 2020,West Bromwich Albion FC,0-3,0-0,Leicester City FC,
-1,Sun Sep 13 2020,Tottenham Hotspur FC,0-1,0-0,Everton FC,
-1,Mon Sep 14 2020,Sheffield United FC,0-2,0-2,Wolverhampton Wanderers FC,
-1,Mon Sep 14 2020,Brighton & Hove Albion FC,1-3,0-1,Chelsea FC,
-2,Sat Sep 19 2020,Everton FC,5-2,2-1,West Bromwich Albion FC,
-2,Sat Sep 19 2020,Leeds United FC,4-3,2-1,Fulham FC,
-2,Sat Sep 19 2020,Manchester United FC,1-3,0-1,Crystal Palace FC,
-2,Sat Sep 19 2020,Arsenal FC,2-1,1-1,West Ham United FC,
-2,Sun Sep 20 2020,Southampton FC,2-5,1-1,Tottenham Hotspur FC,
-2,Sun Sep 20 2020,Newcastle United FC,0-3,0-2,Brighton & Hove Albion FC,
-2,Sun Sep 20 2020,Chelsea FC,0-2,0-0,Liverpool FC,
-2,Sun Sep 20 2020,Leicester City FC,4-2,1-1,Burnley FC,
-2,Mon Sep 21 2020,Aston Villa FC,1-0,0-0,Sheffield United FC,
-2,Mon Sep 21 2020,Wolverhampton Wanderers FC,1-3,0-2,Manchester City FC,
-...
-```
-
-Or the Brasileirão (`br.1`) in  `./o/2020/br.1.csv`:
-
-```
-Matchday,Date,Team 1,FT,HT,Team 2,Comments
-1,Sat Aug 8 2020,Fortaleza EC,0-2,0-2,CA Paranaense,
-1,Sat Aug 8 2020,Coritiba FBC,0-1,0-0,SC Internacional,
-1,Sun Aug 9 2020,SC Recife,3-2,3-1,Ceará SC,
-1,Sun Aug 9 2020,Santos FC,1-1,0-0,RB Bragantino,
-1,Sun Aug 9 2020,CR Flamengo,0-1,0-1,CA Mineiro,
-1,Sun Aug 9 2020,Goiás EC,(*),,São Paulo FC,postponed
-1,Sun Aug 9 2020,Grêmio FBPA,1-0,1-0,Fluminense FC,
-1,Sun Aug 9 2020,SE Palmeiras,(*),,CR Vasco da Gama,postponed
-2,Wed Aug 12 2020,CA Mineiro,3-2,0-2,SC Corinthians Paulista,
-2,Wed Aug 12 2020,CA Paranaense,2-1,1-1,Goiás EC,
-2,Wed Aug 12 2020,RB Bragantino,1-1,1-0,Botafogo FR,
-2,Wed Aug 12 2020,AC Goianiense,3-0,2-0,CR Flamengo,
-2,Wed Aug 12 2020,EC Bahia,1-0,1-0,Coritiba FBC,
-2,Thu Aug 13 2020,Fluminense FC,1-1,1-1,SE Palmeiras,
-2,Thu Aug 13 2020,Ceará SC,1-1,1-0,Grêmio FBPA,
-2,Thu Aug 13 2020,São Paulo FC,1-0,1-0,Fortaleza EC,
-2,Thu Aug 13 2020,SC Internacional,2-0,0-0,Santos FC,
-2,Thu Aug 13 2020,CR Vasco da Gama,2-0,2-0,SC Recife,
-...
-```
-
-
-
-That's it for now. More sources upcoming.
-
-## License
-
-![](https://publicdomainworks.github.io/buttons/zero88x31.png)
-
-The `football-sources` scripts are dedicated to the public domain.
-Use it as you please with no restrictions whatsoever.
-
-
-## Questions? Comments?
-
-Send them along to the
-[Open Sports & Friends Forum/Mailing List](http://groups.google.com/group/opensport).
-Thanks!
+# football-sources - get football match data (leagues, cups & more) via web pages or web api (json) calls
+
+
+* home  :: [github.com/sportdb/sport.db](https://github.com/sportdb/sport.db)
+* bugs  :: [github.com/sportdb/sport.db/issues](https://github.com/sportdb/sport.db/issues)
+* gem   :: [rubygems.org/gems/football-sources](https://rubygems.org/gems/football-sources)
+* rdoc  :: [rubydoc.info/gems/football-sources](http://rubydoc.info/gems/football-sources)
+* forum :: [groups.google.com/group/opensport](https://groups.google.com/group/opensport)
+
+
+
+## Usage
+
+
+To be done.
+
+
+
+## License
+
+
+The `football-sources` scripts are dedicated to the public domain.
+Use it as you please with no restrictions whatsoever.
+
+
+
+## Questions? Comments?
+
+Yes, you can. More than welcome.
+See [Help & Support »](https://github.com/openfootball/help)

data/Rakefile

@@ -1,31 +1,36 @@
-require 'hoe'
-require './lib/football-sources/version.rb'
-
-Hoe.spec 'football-sources' do
-
-  self.version = FootballSources::VERSION
-
-  self.summary = 'football-sources - get football data via web pages or web api (json) calls'
-  self.description = summary
-
-  self.urls = { home: 'https://github.com/sportdb/sport.db' }
-
-  self.author = 'Gerald Bauer'
-  self.email  = 'opensport@googlegroups.com'
-
-  # switch extension to .markdown for gihub formatting
-  self.readme_file  = 'README.md'
-  self.history_file = 'CHANGELOG.md'
-
-  self.extra_deps = [
-    ['webget-football',  '>= 0.1.1'],
-    ['sportdb-catalogs', '>= 1.0.0'],
-  ]
-
-  self.licenses = ['Public Domain']
-
-  self.spec_extras = {
-   required_ruby_version: '>= 2.2.2'
-  }
-
-end
+require 'hoe'
+require './lib/football-sources/version.rb'
+
+Hoe.spec 'football-sources' do
+
+  self.version = FootballSources::VERSION
+
+  self.summary = 'football-sources - get football match data (leagues, cups & more) via web pages or web api (json) calls'
+  self.description = summary
+
+
+
+  self.urls = { home: 'https://github.com/sportdb/sport.db' }
+
+  self.author = 'Gerald Bauer'
+  self.email  = 'gerald.bauer@gmail.com'
+
+  # switch extension to .markdown for gihub formatting
+  self.readme_file  = 'README.md'
+  self.history_file = 'CHANGELOG.md'
+
+  self.extra_deps = [
+    ['sportdb-catalogs'],   #, '>= 1.0.0'],
+    ['sportdb-writers'],
+    ##  footballdata sources
+    ['footballdata-api'],
+    ['worldfootball'],
+  ]
+
+  self.licenses = ['Public Domain']
+
+  self.spec_extras = {
+   required_ruby_version: '>= 2.2.2'
+  }
+
+end

data/bin/fbgen

@@ -0,0 +1,131 @@
+#!/usr/bin/env ruby
+
+## tip: to test run:
+##   ruby -I ./lib bin/fbgen
+
+require 'football/sources'
+
+
+load_env   ## use dotenv (.env)
+
+
+Webcache.root =  if File.exist?( '/sports/cache' )
+                     puts "  setting web cache to >/sports/cache<"
+                     '/sports/cache'
+                 else 
+                     './cache'
+                 end
+
+
+
+## set openfootball github org root - note - NOT monorepo (github) root dirc
+SportDb::GitHubSync.root = '/sports/openfootball'   
+
+
+## add convenience shortcuts
+Country = Sports::Country
+League  = Sports::League
+Club    = Sports::Club
+
+# Worldfootball.config.convert.out_dir = './o/aug29'
+# Worldfootball.config.convert.out_dir = './o'
+# Worldfootball.config.convert.out_dir = './o'
+Worldfootball.config.convert.out_dir = if File.exist?( '/sports/cache.wfb' )
+                                  puts "  setting wfb stage/cache to >/sports/cache.wfb<"
+                                          '/sports/cache.wfb'
+                                       else
+                                         './stage/wfb'
+                                       end
+#########
+##  staging cache settings
+Footballdata.config.convert.out_dir =  if File.exist?( '/sports/cache.api.fbdat' )
+                                  puts "  setting fbdat stage/cache to >/sports/cache.api.fbdat<"
+                                           '/sports/cache.api.fbdat'
+                                        else
+                                          './stage/fbdat'
+                                       end
+
+pp File.expand_path( Footballdata.config.convert.out_dir )
+
+
+## note -  free tier (tier one) plan - 10 requests/minute   
+##              (one request every 6 seconds 6*10=60 secs)
+##     10 API calls per minute max.
+##  note - default sleep (delay in secs) is 3 sec(s)
+Webget.config.sleep  = 10
+
+
+
+
+require 'optparse'
+
+
+
+args=ARGV 
+
+
+opts = {
+    type: 'fbdat',
+    download: false,
+    push:     false,
+}
+
+parser = OptionParser.new do |parser|
+  parser.banner = "Usage: #{$PROGRAM_NAME} [options]"
+
+#  parser.on( "--cache", "--cached", "--offline",
+#               "use cached data in #{Webcache.root}" ) do |cached|
+#    opts[:cached] = cached
+#  end
+
+  parser.on( "-t NAME", "--type=NAME", 
+               "use source type - default is (#{opts[:type]})" ) do |type|
+     opts[:type] = type
+  end
+#  end
+
+
+# add download and push ???
+
+end
+parser.parse!( args )
+
+puts "OPTS:"
+p opts
+puts "ARGV:"
+p args
+
+
+
+source = if ['fbdat'].include?( opts[:type] )
+               Footballdata
+        elsif ['wfb'].include?( opts[:type] )
+               Worldfootball
+        else
+              raise ArgumentError, "unknown source type - expected fbdata|wfb; got #{opts[:type]}"
+        end
+
+puts "source:"
+pp source
+
+
+league      =  (args[0] || 'eng.1' ).downcase
+season      =  (args[1] || '2024/25')
+
+datasets = [
+  [league, [season]]  
+]
+
+puts "datasets:"
+pp datasets
+
+
+
+Fbgen.process( datasets,
+                  source:   source,    
+                  download: opts[:download],
+                  push:     opts[:push] )
+
+
+puts "bye"
+

data/lib/football/sources.rb

@@ -1,6 +1,6 @@
-# note: allow require 'football/sources' too
-#        (in addition to require 'football-sources')
-
-require_relative '../football-sources'
-
-
+# note: allow require 'football/sources' too
+#        (in addition to require 'football-sources')
+
+require_relative '../football-sources'
+
+

data/lib/football-sources/process.rb

@@ -0,0 +1,68 @@
+
+
+module Fbgen
+
+
+class Job     ## todo/check: use a module (NOT a class) - why? why not?
+  ## note - source expected module/class e.g. Footballdata/Worldfootball e.g.    
+def self.download( datasets, source: )
+  datasets.each_with_index do |dataset,i|
+    league  = dataset[0]
+    seasons = dataset[1]
+
+    puts "downloading [#{i+1}/#{datasets.size}] #{league}..."
+    seasons.each_with_index do |season,j|
+      puts "  season [#{j+1}/#{season.size}] #{league} #{season}..."
+      source.schedule( league: league,
+                       season: season )
+    end
+  end
+end
+
+def self.convert( datasets, source: )
+  datasets.each_with_index do |dataset,i|
+    league  = dataset[0]
+    seasons = dataset[1]
+
+    puts "converting [#{i+1}/#{datasets.size}] #{league}..."
+    seasons.each_with_index do |season,j|
+      puts "  season [#{j+1}/#{season.size}] #{league} #{season}..."
+      source.convert( league: league,
+                      season: season )
+    end
+  end
+end
+end  # class Job
+
+
+## change download? to cache - true/false - why? why not?   
+##  change push: to sync - true/false - why? why not?
+def self.process( datasets,
+                     source:,
+                     download: false,
+                     push:     false )
+
+  Job.download( datasets, source: source )   if download
+
+  ## always pull before push!! (use fast_forward)
+  gh = SportDb::GitHubSync.new( datasets )
+  gh.git_fast_forward_if_clean    if push
+
+
+  Job.convert( datasets, source: source )
+
+  if push
+    Writer.config.out_dir = SportDb::GitHubSync.root   # e.g. "/sports/openfootball"
+  else
+    ##  fix/fix - use default - do not (re)set here - why? why not?
+    Writer.config.out_dir = './tmp'
+  end
+
+  Writer::Job.write( datasets,
+                     source: source.config.convert.out_dir )
+
+  ## todo/fix: add a getch or something to hit return before commiting pushing - why? why not?
+  gh.git_push_if_changes     if push
+end  
+
+end  # module Fbgen

data/lib/football-sources/version.rb

@@ -1,19 +1,19 @@
-
-module FootballSources
-  MAJOR = 0    ## todo: namespace inside version or something - why? why not??
-  MINOR = 1
-  PATCH = 0
-  VERSION = [MAJOR,MINOR,PATCH].join('.')
-
-  def self.version
-    VERSION
-  end
-
-  def self.banner
-    "football-sources/#{VERSION} on Ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
-  end
-
-  def self.root
-    File.expand_path( File.dirname(File.dirname(File.dirname(__FILE__))) )
-  end
-end   # module FootballSources
+
+module FootballSources
+  MAJOR = 0    ## todo: namespace inside version or something - why? why not??
+  MINOR = 2
+  PATCH = 0
+  VERSION = [MAJOR,MINOR,PATCH].join('.')
+
+  def self.version
+    VERSION
+  end
+
+  def self.banner
+    "football-sources/#{VERSION} on Ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}] in (#{root})"
+  end
+
+  def self.root
+    File.expand_path( File.dirname(File.dirname(File.dirname(__FILE__))) )
+  end
+end   # module FootballSources

data/lib/football-sources.rb

@@ -1,57 +1,21 @@
-require 'webget/football'
-
-
-# require 'sportdb/formats'   ## for Season etc.
-require 'sportdb/catalogs'    ## note: incl. deps csvreader etc.
-
-
-
-#############
-## todo/fix:  reuse  a "original" CsvMatchWriter
-##                  how? why? why not?
-###############
-module Cache
-class CsvMatchWriter
-
-  def self.csv_encode( values )
-    ## quote values that incl. a comma
-    values.map do |value|
-      if value.index(',')
-        puts "** rec with field with comma >#{value}< in:"
-        pp values
-        %Q{"#{value}"}
-      else
-        value
-      end
-    end.join( ',' )
-  end
-
-  def self.write( path, recs, headers: )
-    ## for convenience - make sure parent folders/directories exist
-    FileUtils.mkdir_p( File.dirname( path ))  unless Dir.exist?( File.dirname( path ))
-
-    File.open( path, 'w:utf-8' ) do |f|
-      f.write( headers.join(','))   ## e.g. Date,Team 1,FT,HT,Team 2
-      f.write( "\n" )
-      recs.each do |values|
-        f.write( csv_encode( values ))
-        f.write( "\n" )
-      end
-    end
-  end
-end # class CsvMatchWriter
-end # module Cache
-
-
-
-
-###
-# our own code
-require 'football-sources/version' # let version always go first
-
-require 'football-sources/apis'
-require 'football-sources/worldfootball'
-require 'football-sources/fbref'
-
-
-puts FootballSources.banner   # say hello
+
+
+# require 'sportdb/formats'   ## for Season etc.
+require 'sportdb/catalogs'    ## note: incl. deps csvreader etc.
+require 'sportdb/writers'
+
+
+require 'footballdata'      # e.g. football-data.org (api)
+require 'worldfootball'     # e.g. weltfussball.de (web)
+
+
+
+###
+# our own code
+require_relative 'football-sources/version' # let version always go first
+require_relative 'football-sources/process'
+
+
+
+
+puts FootballSources.banner   # say hello