footballdata-12xpert 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 53d9f93e00ee1b85e5848b4dee48988b16e659c3
+  data.tar.gz: 60068169b3aaab76e84d4875a00db617e4e6211d
+SHA512:
+  metadata.gz: fedfd89358237ef25ba0083c8d1f21c7ec13df976e70ea8460f9682ba5966b718dcfbbf6d4b3b92d1fb9310dbb3739b9f62972f8fce5d43d3c2afcfdaf98e2b0
+  data.tar.gz: 36691c7a8a0e4ebfd70aaeecba2d8e363977a9ced0ebb05d1550b5cf3b33bb3e93dc31e76ffb5d4f512165a75149c70edb9f0c76e96277c31ee034199bfd0c73

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+### 0.0.1 / 2019-08-19
+
+* Everything is new. First release.

data/Manifest.txt

@@ -0,0 +1,13 @@
+CHANGELOG.md
+Manifest.txt
+README.md
+Rakefile
+lib/footballdata-12xpert.rb
+lib/footballdata-12xpert/config.rb
+lib/footballdata-12xpert/convert.rb
+lib/footballdata-12xpert/download.rb
+lib/footballdata-12xpert/version.rb
+lib/footballdata/12xpert.rb
+test/helper.rb
+test/test_download.rb
+test/test_import.rb

data/README.md

@@ -0,0 +1,306 @@
+# footballdata-12xpert - download, convert & import 22+ top football leagues from 25 seasons back to 1993/94 from Joseph Buchdahl (12Xpert)'s Football Data website (football-data.co.uk) up and running since 2001 (and updated twice a week)
+
+
+* home  :: [github.com/sportdb/sport.db.sources](https://github.com/sportdb/sport.db.sources)
+* bugs  :: [github.com/sportdb/sport.db.sources/issues](https://github.com/sportdb/sport.db.sources/issues)
+* gem   :: [rubygems.org/gems/footballdata-12xpert](https://rubygems.org/gems/footballdata-12xpert)
+* rdoc  :: [rubydoc.info/gems/footballdata-12xpert](http://rubydoc.info/gems/footballdata-12xpert)
+* forum :: [opensport](http://groups.google.com/group/opensport)
+
+
+
+## What's Joseph Buchdahl's Football Data?
+
+[Joseph Buchdahl (12Xpert)](https://twitter.com/12Xpert) has been publishing football data
+at the [`football-data.co.uk`](https://www.football-data.co.uk/data.php) website
+in the world's most popular tabular data interchange format in text, that is,
+comma-separated value (.csv) records for (bulk) download (and offline usage) since 2001 (!).
+
+The main top football leagues include:
+
+- England (`E0`, `E1`, `E2`, `E3` & `EC`) - Premiership & Divs 1, 2, 3 & Conference
+- Scotland  (`SC0`, `SC1`, `SC2` & `SC3`) - Premiership & Divs 1, 2 & 3
+- Germany (`D1` & `D2`) - Bundesligas 1 & 2
+- Italy (`I1` & `I2`) - Serie A & B
+- Spain (`SP1` & `SP2`) - La Liga (Primera & Segunda)
+- France (`F1` & `F2`) -  Le Championnat & Division 2
+- Netherlands (`N1`) - Eredivisie
+- Belgium (`B1`) - Pro League
+- Portugal (`P1`) - Liga I
+- Turkey (`T1`) - Ligi 1
+- Greece (`G1`) -  Ethniki Katigoria
+
+And the extra leagues include:
+
+- Argentina (`ARG`) - Primera Division
+- Austria (`AUT`) -  Bundesliga
+- Brazil (`BRA`) -  Serie A
+- China (`CHN`) - Super League
+- Denmark (`DNK`) -  Superliga
+- Finland (`FIN`) -  Veikkausliiga
+- Ireland (`IRL`) - Premier Division
+- Japan (`JPN`) -  J-League
+- Mexico (`MEX`) -  Liga MX
+- Norway (`NOR`) -  Eliteserien
+- Poland (`POL`)-  Ekstraklasa
+- Romania (`ROU`) -  Liga 1
+- Russia (`RUS`) - Premier League
+- Sweden (`SWE`) -   Allsvenskan
+- Switzerland (`SWZ`) -  Super League
+- USA (`USA`) -  Major League Soccer (MLS)
+
+
+The top football leagues include 25 seasons back to 1993/94
+and get at least updated twice weekly
+(Sunday nights and Wednesday nights).
+
+
+## Usage
+
+Let's download all datasets (about 570+) for offline usage into the
+default web cache directory (that is, `~/.cache/www.football-data.co.uk`):
+
+
+``` ruby
+require 'footballdata/12xpert'
+
+Footballdata12xpert.download
+```
+
+Note: You can use `Footballdata12Xpert`,
+`Footballdata_12xpert`, or `Footballdata_12Xpert`
+as alternate alias names for `Footballdata12xpert`.
+
+
+
+Stand back ten feet. Resulting in:
+
+```
+~/.cache/www.football-data.co.uk
+│   ARG.csv
+│   AUT.csv
+│   BRA.csv
+│   CHN.csv
+│   DNK.csv
+│   FIN.csv
+│   IRL.csv
+│   JPN.csv
+│   MEX.csv
+│   NOR.csv
+│   POL.csv
+│   ROU.csv
+│   RUS.csv
+│   SWE.csv
+│   SWZ.csv
+│   USA.csv
+│
+├───9394
+│       D1.csv
+│       D2.csv
+│       E0.csv
+│       E1.csv
+│       E2.csv
+│       E3.csv
+│       F1.csv
+│       I1.csv
+│       N1.csv
+│       SP1.csv
+│
+├───9495
+│       D1.csv
+│       D2.csv
+│       E0.csv
+│       E1.csv
+│       E2.csv
+│       E3.csv
+│       F1.csv
+│       G1.csv
+│       I1.csv
+│       N1.csv
+│       P1.csv
+│       SC0.csv
+│       SC1.csv
+│       SP1.csv
+│       T1.csv
+...
+├───1819
+│       B1.csv
+│       D1.csv
+│       D2.csv
+│       E0.csv
+│       E1.csv
+│       E2.csv
+│       E3.csv
+│       EC.csv
+│       F1.csv
+│       F2.csv
+│       G1.csv
+│       I1.csv
+│       I2.csv
+│       N1.csv
+│       P1.csv
+│       SC0.csv
+│       SC1.csv
+│       SC2.csv
+│       SC3.csv
+│       SP1.csv
+│       SP2.csv
+│       T1.csv
+│
+└───1920
+        B1.csv
+        D1.csv
+        D2.csv
+        E0.csv
+        E1.csv
+        E2.csv
+        E3.csv
+        EC.csv
+        F1.csv
+        F2.csv
+        N1.csv
+        P1.csv
+        SC0.csv
+        SC1.csv
+        SC2.csv
+        SC3.csv
+        SP1.csv
+        SP2.csv
+        T1.csv
+```
+
+The football datasets come in two flavors / formats.
+The main leagues use season-by-season datafiles.
+For example, `E0.csv`, `E1.csv`, `E2.csv`, `E3.csv` & `E4.csv` in the `1920`
+season directory hold the matches for the English Premiership & Divs 1, 2, 3 & Conference;
+`D1.csv` & `D2.csv` for the Bundesligas 1 & 2 and so on
+for the 2019/20 season.
+
+The extra leagues use an all-seasons-in-one datafile.
+For example, `ARG.csv`
+holds all seasons of the Argentinian Primera Division;
+`AUT.csv` for the Austrian Bundesliga and so on.
+
+
+Note: The datasets character encoding gets converted from
+[Windows-1252 (8-bit)](https://en.wikipedia.org/wiki/Windows-1252) to UTF-8 (Unicode multi-byte).
+
+
+Less is More?
+
+You can download datasets for selected countries only. Pass in
+the country keys. Let's download only England (`eng`)'s leagues:
+
+``` ruby
+Footballdata.download( 'eng' )
+```
+
+Or let's download only the top five leagues, that is,
+England (`eng`), Spain (`es`), Germany (`de`), France (`fr`)
+and Italy (`it`):
+
+``` ruby
+Footballdata.download( 'eng', 'es', 'de', 'fr', 'it' )
+```
+
+
+Now what? Let's import all football datasets from the web cache
+into an SQL database.
+
+
+``` ruby
+SportDb.connect( adapter:  'sqlite3',
+                 database: './football.db' )
+
+SportDb.create_all   ## build database schema / tables
+
+
+Footballdata.import
+```
+
+Note: Depending on your computing processing power the import might take
+10+ minutes.
+
+
+Done. Let's try some database (SQL) queries (using the sport.db ActiveRecord models):
+
+``` ruby
+## ActiveRecord model (convenience) shortcuts
+Team   = SportDb::Model::Team
+Game   = SportDb::Model::Game
+League = SportDb::Model::League
+Event  = SportDb::Model::Event
+
+
+## Let's query for some stats  - How many teams? How many games / matches? etc.
+
+puts Team.count   #=> 1143
+# SELECT COUNT(*) FROM teams
+
+puts Game.count   #=> 227_142
+# SELECT COUNT(*) FROM games
+
+puts League.count  #=> 38
+# SELECT COUNT(*) FROM leagues
+```
+
+Note: See the [SUMMARY.md](SUMMARY.md) page for a list of all 1000+ (canonical)
+club names by country.
+
+``` ruby
+## Let's query for the Real Madrid football club from Spain
+
+madrid = Team.find_by( title: 'Real Madrid' )
+# SELECT * FROM teams WHERE title = 'Real Madrid' LIMIT 1
+
+puts madrid.games.count   #=> 1023
+# SELECT COUNT(*) FROM games WHERE (team1_id = 380 or team2_id = 380)
+g = madrid.games.first
+# SELECT * FROM "games" WHERE (team1_id = 380 or team2_id = 380) LIMIT 1
+
+puts g.team1.title #=> CA Osasuna
+puts g.team2.title #=> Real Madrid
+puts g.score_str   #=> 1 - 4
+
+
+## Or let's query for the Liverpool football club from England
+
+liverpool = Team.find_by( title: 'Liverpool FC' )
+
+puts liverpool.games.count  #=> 1025
+
+g = liverpool.games.first
+puts g.team1.title  #=> Liverpool FC
+puts g.team2.title  #=> Sheffield Wednesday FC
+puts g.score_str    #=> 2 - 0
+
+
+## Let's try the English Premier League 2019/20
+
+pl = Event.find_by( key: 'eng.1.2019/20' )
+
+puts pl.games.count  #=> 288
+
+g = pl.games.first
+puts g.team1.title  #=> Liverpool FC
+puts g.team2.title  #=> Norwich City FC
+puts g.score_str    #=> 4 - 1
+
+# and so on
+```
+
+That's it. Enjoy the beautiful game.
+
+
+
+## License
+
+The `sportdb-source-footballdata` scripts are dedicated to the public domain.
+Use it as you please with no restrictions whatsoever.
+
+
+## Questions? Comments?
+
+Send them along to the
+[Open Sports & Friends Forum/Mailing List](http://groups.google.com/group/opensport).
+Thanks!

data/Rakefile

@@ -0,0 +1,31 @@
+require 'hoe'
+require './lib/footballdata-12xpert/version.rb'
+
+Hoe.spec 'footballdata-12xpert' do
+
+  self.version = Footballdata12xpert::VERSION
+
+  self.summary = "footballdata-12xpert - download, convert & import 22+ top football leagues from 25 seasons back to 1993/94 from Joseph Buchdahl (12Xpert)'s Football Data website (football-data.co.uk) up and running since 2001 (and updated twice a week)"
+  self.description = summary
+
+  self.urls = { home: 'https://github.com/sportdb/sport.db.sources'}
+
+  self.author = 'Gerald Bauer'
+  self.email  = 'opensport@googlegroups.com'
+
+  # switch extension to .markdown for gihub formatting
+  self.readme_file  = 'README.md'
+  self.history_file = 'CHANGELOG.md'
+
+  self.licenses = ['Public Domain']
+
+  self.extra_deps = [
+    ['webget',            '>= 0.2.1'],
+    ['footballdb-clubs',  '>= 2020.9.15'],
+    ['sportdb-importers', '>= 1.1.1'],
+  ]
+
+  self.spec_extras = {
+   required_ruby_version: '>= 2.2.2'
+  }
+end

data/lib/footballdata-12xpert.rb

@@ -0,0 +1,168 @@
+
+## 3rd party libs / gems
+require 'webget'
+
+## sportdb libs / gems
+require 'sportdb/importers'
+
+
+
+###
+# our own code
+require 'footballdata-12xpert/version' # let version always go first
+
+require 'footballdata-12xpert/config'
+require 'footballdata-12xpert/download'
+require 'footballdata-12xpert/convert'
+
+###
+##  add alternate aliases - why? why not?
+Footballdata12Xpert   = Footballdata12xpert
+Footballdata_12xpert  = Footballdata12xpert
+Footballdata_12Xpert  = Footballdata12xpert
+
+
+
+
+module Footballdata12xpert
+
+
+def self.import( *args, dir: './dl' )
+
+  country_keys = args  ## countries to include / fetch - optinal
+
+  FOOTBALLDATA_SOURCES.each do |country_key, country_sources|
+    if country_keys.empty? || country_keys.include?( country_key )
+      Footballdata.import_season_by_season( country_key, country_sources, dir: dir )
+    else
+      ## skipping country
+    end
+  end
+
+  FOOTBALLDATA_SOURCES_II.each do |country_key, country_basename|
+    if country_keys.empty? || country_keys.include?( country_key )
+      Footballdata.import_all_seasons( country_key, country_basename, dir: dir )
+    else
+      ## skipping country
+    end
+  end
+end # method import
+
+class << self
+  alias_method :load,     :import
+end
+
+
+
+def self.import_season_by_season( country_key, sources, dir: )
+
+  ## todo/check: make sure timezones entry for country_key exists!!! what results with nil/24.0 ??
+  fix_date_converter = ->(row) { fix_date( row, FOOTBALLDATA_TIMEZONES[country_key]/24.0 ) }
+
+  sources.each do |rec|
+    season_key  = rec[0]   ## note: dirname is season_key e.g. 2011-12 etc.
+    basenames  = rec[1]   ## e.g. E1,E2,etc.
+
+    basenames.each do |basename|
+
+      path = "#{dir}/#{season_key}/#{basename}.csv"
+
+      league_key = FOOTBALLDATA_LEAGUES[basename]  ## e.g.: eng.1, fr.1, fr.2 etc.
+      if league_key.nil?
+        puts "** !!! ERROR !!! league key missing for >#{basename}<; sorry - please add"
+        exit 1
+      end
+
+      country, league = find_or_create_country_and_league( league_key )
+
+      season = SportDb::Importer::Season.find_or_create_builtin( season_key )
+
+      puts "path: #{path}"
+
+      matches = CsvMatchReader.read( path, converters: fix_date_converter )
+
+      update_matches_txt( matches,
+                            league:  league,
+                            season:  season )
+    end
+  end
+end  # method import_season_by_season
+
+
+
+def self.import_all_seasons( country_key, basename, dir: )
+
+  col  = 'Season'
+  path = "#{dir}/#{basename}.csv"
+
+  season_keys = CsvMatchSplitter.find_seasons( path, col: col )
+  pp season_keys
+
+  ## note: assume always first level/tier league for now
+  league_key = "#{country_key}.1"
+  country, league = find_or_create_country_and_league( league_key )
+
+  ## todo/check: make sure timezones entry for country_key exists!!! what results with nil/24.0 ??
+  fix_date_converter = ->(row) { fix_date( row, FOOTBALLDATA_TIMEZONES[country_key]/24.0 ) }
+
+  season_keys.each do |season_key|
+    season = SportDb::Importer::Season.find_or_create_builtin( season_key )
+
+    matches = CsvMatchReader.read( path, filters: { col => season_key },
+                                         converters: fix_date_converter )
+
+    pp matches[0..2]
+    pp matches.size
+
+    update_matches_txt( matches,
+                          league:  league,
+                          season:  season )
+  end
+end  # method import_all_seasons
+
+
+###
+## helper for country and league db record
+def self.find_or_create_country_and_league( league_key )
+  country_key, level = league_key.split( '.' )
+  country = SportDb::Importer::Country.find_or_create_builtin!( country_key )
+
+  league_auto_name = "#{country.name} #{level}"   ## "fallback" auto-generated league name
+  pp league_auto_name
+  league = SportDb::Importer::League.find_or_create( league_key,
+                                                       name:       league_auto_name,
+                                                       country_id: country.id )
+
+  [country, league]
+end
+
+## helper to fix dates to use local timezone (and not utc/london time)
+def self.fix_date( row, offset )
+  return row    if row['Time'].nil?   ## note: time (column) required for fix
+
+  col = row['Date']
+  if col =~ /^\d{2}\/\d{2}\/\d{4}$/
+    date_fmt = '%d/%m/%Y'   # e.g. 17/08/2002
+  elsif col =~ /^\d{2}\/\d{2}\/\d{2}$/
+    date_fmt = '%d/%m/%y'   # e.g. 17/08/02
+  else
+    puts "*** !!! wrong (unknown) date format >>#{col}<<; cannot continue; fix it; sorry"
+    ## todo/fix: add to errors/warns list - why? why not?
+    exit 1
+  end
+
+  date = DateTime.strptime( "#{row['Date']} #{row['Time']}", "#{date_fmt} %H:%M" )
+  date = date + offset
+
+  row['Date'] = date.strftime( date_fmt )  ## overwrite "old"
+  row['Time'] = date.strftime( '%H:%M' )
+  row   ## return row for possible pipelining - why? why not?
+end
+
+
+
+end ## module Footballdata12xpert
+
+
+
+puts Footballdata12xpert.banner   # say hello