rsssf 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 0e916c424ef3d9ebd62d5c2fb1e09122806c5765
-  data.tar.gz: 232f06dea9fd3f878e554b1d4a2029531468f842
+  metadata.gz: 9c33c4bef56bf6e79c9339b5010a3129b98dc9fa
+  data.tar.gz: 4d97b1862dfbaab48282593947c6d7b28d047dfe
 SHA512:
-  metadata.gz: 898399cdcedd7890145278b17d6d7ab5d8f305160513def86152b69a345de5c19ddb4e2e770753c091216c203bf31a3bd2bd70ca33b0087a8d003292cdd97acf
-  data.tar.gz: 0b76813d36aee3c59a83c2dfce1d916ee4677da585d52464a531dc2563e5bf96d33b345afe766bb4af962c03f4abcfed0a5b8523e485c85c7e85e406fd9ab8c6
+  metadata.gz: 8da437c2a72c364f81d53ca1491cdfca4c11c0f187912702e653523cb54f636ec910f8690f9af690cf632f0c98335b9ab10983676d0e67e1a59f3c7d7a5919c7
+  data.tar.gz: b8a13466f301863e6ae0e4e449f1da08d5ae17a7481bf2055e12976f6f14fda79a8ebd38aa5cfb5f097d21b36e43bc322f362483d1c522a005c9cf49e576f732

data/Manifest.txt

@@ -3,4 +3,15 @@ Manifest.txt
 README.md
 Rakefile
 lib/rsssf.rb
+lib/rsssf/fetch.rb
+lib/rsssf/html2txt.rb
+lib/rsssf/page.rb
+lib/rsssf/patch.rb
+lib/rsssf/repo.rb
+lib/rsssf/reports/page.rb
+lib/rsssf/reports/schedule.rb
+lib/rsssf/schedule.rb
+lib/rsssf/utils.rb
 lib/rsssf/version.rb
+test/helper.rb
+test/test_utils.rb

data/README.md

@@ -1,16 +1,172 @@
 # rsssf - tools 'n' scripts for RSSSF (Rec.Sport.Soccer Statistics Foundation) archive data
 
 
-* home  :: [github.com/sportdb/rsssf](https://github.com/sportdb/mrhyde)
+* home  :: [github.com/sportdb/rsssf](https://github.com/sportdb/rsssf)
 * bugs  :: [github.com/sportdb/rsssf/issues](https://github.com/sportdb/rsssf/issues)
 * gem   :: [rubygems.org/gems/rsssf](https://rubygems.org/gems/rsssf)
 * rdoc  :: [rubydoc.info/gems/rsssf](http://rubydoc.info/gems/rsssf)
 * forum :: [opensport](http://groups.google.com/group/opensport)
 
 
+## What's the Rec.Sport.Soccer Statistics Foundation (RSSSF)? 
+
+The RSSSF collects and offers football (soccer) league tables, match results and more
+from all over the world online in plain text.
+
+Example:
+
+```
+Round 1
+[May 25]
+Vasco da Gama   1-0 Portuguesa
+ [Carlos Tenório 47']
+Vitória         2-2 Internacional
+ [Maxi Biancucchi 2', Gabriel Paulista 11'; Diego Forlán 29', Fred 63']
+Corinthians     1-1 Botafogo
+ [Paulinho 73'; Rafael Marques 24']
+[May 26]
+Grêmio          2-0 Náutico         [played in Caxias do Sul-RS]
+ [Zé Roberto 15', Elano 70']
+Ponte Preta     0-2 São Paulo
+ [Lúcio 9', Jádson 44'p]
+Criciúma        3-1 Bahia
+ [Matheus Ferraz 45'+1', Lins 46', João Vítor 82'; Diones 72']
+Santos          0-0 Flamengo        [played in Brasília-DF]
+Fluminense      2-1 Atlético/PR     [played in Macaé-RJ]
+ [Rafael Sóbis 15'p, Samuel 53'; Manoel 28']
+Cruzeiro        5-0 Goiás
+ [Diego Souza 5', Bruno Rodrigo 30', Nílton 40',79', Borges 42']
+Coritiba        2-1 Atlético/MG
+ [Deivid 53', Arthur 90'+1'; Diego Tardelli 51']
+```
+
+[Find out more about the Rec.Sport.Soccer Statistics Foundation (RSSSF) »](http://www.rsssf.com)
+
+
+
 ## Usage
 
-To be done
+### Working with Pages
+
+To fetch pages from the world wide web use:
+
+``` ruby
+page = RsssfPage.from_url( 'http://www.rsssf.com/tablese/eng2015.html')
+```
+
+Note: The `RsssfPageFetcher` will convert the rsssf archive page
+from hypertext (HTML) to plain text e.g.
+
+```
+<hr>
+<a href="#premier">Premier League</A><br>
+<a href="#cups">Cup Tournaments</A><br>
+<a href="#champ">Championship</A><br>
+<a href="#first">Division 1</A><br>
+<a href="#second">Division 2</A><br>
+<a href="#conf">Conference</A>
+<hr>
+<h4><a name="premier">Premier League</A></h4>
+<pre>
+Final Table:
+
+ 1.Chelsea                 38  26  9  3  73-32  87  Champions
+ 2.Manchester City         38  24  7  7  83-38  79
+ 3.Arsenal                 38  22  9  7  71-36  75
+...
+```
+
+will become
+
+```
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+‹Premier League›
+‹Cup Tournaments›
+‹Championship›
+‹Division 1›
+‹Division 2›
+‹Conference›
+=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+
+#### Premier League
+
+
+Final Table:
+
+ 1.Chelsea                 38  26  9  3  73-32  87  Champions
+ 2.Manchester City         38  24  7  7  83-38  79
+ 3.Arsenal                 38  22  9  7  71-36  75
+...
+```
+
+
+### Working with Repos
+
+To fetch pages from the world wide web for many seasons in batch setup and use a repo.
+
+Step 1: List all archive pages
+
+In the `tables/config.yml` list all archive pages to fetch. Example:
+
+``` yaml
+2010-11: tablese/eng2011.html
+2011-12: tablese/eng2012.html
+2012-13: tablese/eng2013.html
+2013-14: tablese/eng2014.html
+2014-15: tablese/eng2015.html
+```
+
+Step 2: Fetch all archive pages
+
+Use:
+
+``` ruby
+repo = RsssfRepo.new( './eng-england', title: 'England (and Wales)' )
+repo.fetch_pages
+```
+
+Bonus: To create a summary of all pages fetched (e.g. authors, last_updated, sections, etc.).
+Use:
+
+``` ruby
+repo.make_pages_report
+```
+
+Example - `tables/README.md`:
+
+
+football.db RSSSF Archive Data Summary for England (and Wales)
+
+_Last Update: 2015-11-26 18:22:22 +0200_
+
+| Season  | File   | Authors  | Last Updated | Lines (Chars) | Sections | 
+| :------ | :------ | :------- | :----------- | ------------: | :------- |
+| 2014-15 | [eng2015.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2015.txt) | Ian King and Karel Stokkermans | 4 Jun 2015 | 1249 (34138) | Premier League, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+| 2013-14 | [eng2014.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2014.txt) | Ian King and Karel Stokkermans | 5 Feb 2015 | 1254 (34294) | Premier League, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+| 2012-13 | [eng2013.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2013.txt) | Karel Stokkermans | 5 Feb 2015 | 1269 (34531) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+| 2011-12 | [eng2012.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2012.txt) | Karel Stokkermans | 5 Feb 2015 | 691 (21925) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+| 2010-11 | [eng2011.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2011.txt) | Ian King, Karel Stokkermans and Jan Schoenmakers | 5 Feb 2015 | 959 (37393) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+
+
+That's it.
+
+
+### Preparing Archive Pages for SQL Database Imports (e.g. football.db)
+
+To import match schedules (fixtures and results) and more using the football.db machinery
+prepare "simple" single league (or cup) pages with standings tables etc. stripped out.
+For example, to break-out the Premier League and FA Cup from the `eng2015.txt`
+archive page use:
+
+``` ruby
+page = RsssfPage.from_url( 'http://www.rsssf.com/tablese/eng2015.html')
+
+schedule = page.find_schedule( header: 'Premier League')   ## returns RsssfSchedule obj
+schedule.save( './1-premierleague.txt' )
+
+schedule = page.find_schedule( header: 'FA Cup', cup: true )
+schedule.save( './facup.txt' )
+```
 
 
 
@@ -21,6 +177,19 @@ Just install the gem:
     $ gem install rsssf
 
 
+
+## RSSSF Datasets
+
+See the rsssf github org for pre-processed ready-to-import datasets. Prepared repos include:
+
+- [`eng-england`](https://github.com/rsssf/eng-england)    - rsssf archive data for England - Premier League, Championship, FA Cup etc.
+- [`de-deutschland`](https://github.com/rsssf/de-deutschland) - rsssf archive data for Germany (Deutschland) - Deutsche Bundesliga, 2. Bundesliga, 3. Liga, DFB Pokal etc.
+- [`es-espana`](https://github.com/rsssf/es-espana)      - rsssf archive data for España (Spain) - Primera División / La Liga, Copa de Rey, etc.
+- [`at-austria`](https://github.com/rsssf/at-austria)     - rsssf archive data for Austria (Österreich) - Österr. Bundesliga, Erste Liga, ÖFB Pokal etc.
+- [`br-brazil`](https://github.com/rsssf/br-brazil)      - rsssf archive data for Brazil (Brasil) - Campeonato Brasileiro Série A / Brasileirão etc.
+- and more
+
+
 ## License
 
 The `rsssf` scripts are dedicated to the public domain.

data/lib/rsssf.rb

@@ -14,6 +14,19 @@ require 'fetcher'        ## used for Fetcher::Worker.new.fetch etc.
 ## our own code
 require 'rsssf/version'    # note: let version always go first
 
+require 'rsssf/utils'      # include Utils - goes first
+require 'rsssf/html2txt'   # include Filters - goes first
+
+require 'rsssf/fetch'
+require 'rsssf/page'
+require 'rsssf/schedule'
+require 'rsssf/patch'
+
+require 'rsssf/reports/schedule'
+require 'rsssf/reports/page'
+
+require 'rsssf/repo'
+
 
 
 

data/lib/rsssf/fetch.rb

@@ -0,0 +1,80 @@
+# encoding: utf-8
+
+module Rsssf
+
+class PageFetcher
+
+  include Filters   # e.g. html2text, sanitize etc.
+
+
+def initialize
+  @worker = Fetcher::Worker.new
+end
+  
+def fetch( src_url )
+
+  ## note: assume plain 7-bit ascii for now
+  ##  -- assume rsssf uses ISO_8859_15 (updated version of ISO_8859_1) -- does NOT use utf-8 character encoding!!!
+  html = @worker.read( src_url )  
+
+  ### todo/fix: first check if html is all ascii-7bit e.g.
+  ## includes only chars from 64 to 127!!!
+
+  ## normalize newlines
+  ##   remove \r (form feed) used by Windows; just use \n (new line)
+  html = html.gsub( "\r", '' )
+
+  ## note:
+  ##   assume (default) to ISO 3166-15 (an updated version of ISO 3166-1) for now
+  ##
+  ##  other possible alternatives - try:
+  ##  - Windows CP 1562  or
+  ##  - ISO 3166-2  (for eastern european languages )
+  ##
+  ## note: german umlaut use the same code (int)
+  ##    in ISO 3166-1/15 and 2 and Windows CP1562  (other chars ARE different!!!)
+
+  html = html.force_encoding( Encoding::ISO_8859_15 )
+  html = html.encode( Encoding::UTF_8 )    # try conversion to utf-8
+
+  ## check for html entities
+  html = html.gsub( "ä", 'ä' )
+  html = html.gsub( "ö", 'ö' )
+  html = html.gsub( "ü", 'ü' )
+  html = html.gsub( "Ä", 'Ä' )
+  html = html.gsub( "Ö", 'Ö' )
+  html = html.gsub( "Ü", 'Ü' )
+  html = html.gsub( "ß", 'ß' )
+
+  html = html.gsub( "&oulm;", 'ö' )    ## support typo in entity (ö)
+  html = html.gsub( "&slig;", "ß" )    ## support typo in entity (ß)
+  
+  html = html.gsub( "É", 'É' )
+  html = html.gsub( "ø", 'ø' )
+  
+  ## check for more entities
+  html = html.gsub( /&[^;]+;/) do |match|
+    puts "*** found unencoded html entity #{match}"
+    match   ## pass through as is (1:1)
+  end
+  ## todo/fix: add more entities
+
+
+  txt   = html_to_txt( html )
+
+  header = <<EOS
+<!--
+   source: #{src_url}
+  -->
+
+EOS
+
+  header+txt  ## return txt w/ header
+end  ## method fetch
+
+end  ## class PageFetcher
+end  ## module Rsssf
+
+## add (shortcut) alias
+RsssfPageFetcher = Rsssf::PageFetcher
+

data/lib/rsssf/html2txt.rb

@@ -0,0 +1,157 @@
+# encoding: utf-8
+
+module Rsssf
+module Filters
+
+def html_to_txt( html )
+
+###
+#   todo: check if any tags (still) present??
+
+
+  ## cut off everything before body
+  html = html.sub( /.+?<BODY>\s*/im, '' )
+
+  ## cut off everything after body (closing)
+  html = html.sub( /<\/BODY>.*/im, '' )
+
+
+  ## remove cite
+  html = html.gsub( /<CITE>([^<]+)<\/CITE>/im ) do |_|
+    puts " remove cite >#{$1}<"
+    "#{$1}"
+  end
+
+  html = html.gsub( /\s*<HR>\s*/im ) do |match|
+    match = match.gsub( "\n", '$$' )  ## make newlines visible for debugging
+    puts " replace horizontal rule (hr) - >#{match}<"
+    "\n=-=-=-=-=-=-=-=-=-=-=-=-=-=-=\n"    ## check what hr to use use  - . - . - or =-=-=-= or somehting distinct?
+  end
+
+  ## replace break (br)
+  ## note: do NOT use m/multiline for now - why? why not??
+  html = html.gsub( /<BR>\s*/i ) do |match|    ## note: include (swallow) "extra" newline
+    match = match.gsub( "\n", '$$' )  ## make newlines visible for debugging
+    puts " replace break (br) - >#{match}<"
+    "\n"
+  end
+
+  ## remove anchors (a name)
+  html = html.gsub( /<A NAME[^>]*>(.+?)<\/A>/im ) do |match|   ## note: use .+? non-greedy match
+    title = $1.to_s   ## note: "save" caputure first; gets replaced by gsub (next regex call)
+    match = match.gsub( "\n", '$$' )  ## make newlines visible for debugging
+    puts " replace anchor (a) name >#{title}< - >#{match}<"
+    "#{title}"
+  end
+
+  ## remove anchors (a href)
+  #    note: heading 4 includes anchor (thus, let anchors go first)
+  #  note: <a \newline href is used for authors email - thus incl. support for newline as space
+  html = html.gsub( /<A\s+HREF[^>]*>(.+?)<\/A>/im ) do |_|   ## note: use .+? non-greedy match
+    puts " replace anchor (a) href >#{$1}<"
+    "‹#{$1}›"
+  end
+
+  ## replace paragrah (p)
+  html = html.gsub( /\s*<P>\s*/im ) do |match|    ## note: include (swallow) "extra" newline
+    match = match.gsub( "\n", '$$' )  ## make newlines visible for debugging
+    puts " replace paragraph (p) - >#{match}<"
+    "\n\n"
+  end
+  html = html.gsub( /<\/P>/i, '' )  ## replace paragraph (p) closing w/ nothing for now
+
+  ## remove i
+  html = html.gsub( /<I>([^<]+)<\/I>/im ) do |_|
+    puts " remove italic (i) >#{$1}<"
+    "#{$1}"
+  end
+
+
+  ## heading 2
+  html = html.gsub( /\s*<H2>([^<]+)<\/H2>\s*/im ) do |_|
+    puts " replace heading 2 (h2) >#{$1}<"
+    "\n\n## #{$1}\n\n"    ## note: make sure to always add two newlines
+  end
+
+  ## heading 4
+  html = html.gsub( /\s*<H4>([^<]+)<\/H4>\s*/im ) do |_|
+    puts " replace heading 4 (h4) >#{$1}<"
+    "\n\n#### #{$1}\n\n"    ## note: make sure to always add two newlines
+  end
+
+
+  ## remove b   - note: might include anchors (thus, call after anchors)
+  html = html.gsub( /<B>([^<]+)<\/B>/im ) do |_|
+    puts " remove bold (b) >#{$1}<"
+    "**#{$1}**"
+  end
+
+  ## replace preformatted (pre)
+  html = html.gsub( /<PRE>|<\/PRE>/i ) do |_|
+    puts " replace preformatted (pre)"
+    ''  # replace w/ nothing for now (keep surrounding newlines)
+  end
+
+=begin
+  puts
+  puts
+  puts "html:"
+  puts html[0..2000]
+  puts "-- snip --"
+  puts html[-1000..-1]   ## print last hundred chars
+=end
+
+
+  ## cleanup whitespaces
+  ##   todo/fix:  convert newline in space first
+  ##                and than collapse spaces etc.!!!
+  txt = ''
+  html.each_line do |line|
+     line = line.gsub( "\t", '  ' ) # replace all tabs w/ two spaces for nwo
+     line = line.rstrip             # remove trailing whitespace (incl. newline/formfeed)
+
+     txt << line
+     txt << "\n"
+  end
+
+  ### remove emails etc.
+  txt = sanitize( txt )
+
+  txt
+end # method html_to_text
+
+
+
+def sanitize( txt )
+  ### remove emails for (spam/privacy) protection
+  ## e.g. (selamm@example.es)
+  ##      (buuu@mscs.dal.ca)
+  ##      (kaxx@rsssf.com)
+  ##      (Manu_Maya@yakoo.co)
+
+  ##   note add support for optional ‹› enclosure (used by html2txt converted a href :mailto links)
+  ##   e.g. (‹selamm@example.es›)
+
+  email_pattern = "\\(‹?[a-z][a-z0-9_]+@[a-z]+(\\.[a-z]+)+›?\\)"   ## note: just a string; needs to escape \\ twice!!!
+
+  ## check for "free-standing e.g. on its own line" emails only for now
+  txt = txt.gsub( /\n#{email_pattern}\n/i ) do |match|
+    puts "removing (free-standing) email >#{match}<"
+    "\n"   # return empty line
+  end
+
+  txt = txt.gsub( /#{email_pattern}/i ) do |match|
+    puts "remove email >#{match}<"
+    ''
+  end
+
+  txt  
+end # method sanitize
+
+end # module Filters
+end # module Rsssf
+
+## add (shortcut) alias
+RsssfFilters = Rsssf::Filters
+
+