rsssf 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 9c33c4bef56bf6e79c9339b5010a3129b98dc9fa
-  data.tar.gz: 4d97b1862dfbaab48282593947c6d7b28d047dfe
+SHA256:
+  metadata.gz: 77eac4fd6af25c171a9367afb06db3461e923c25f567222638d96f0665ca242e
+  data.tar.gz: b996e7cb14c9fb945c7138d3bd92e6214f922bcb2baf5e35da41b8ea957ca6f6
 SHA512:
-  metadata.gz: 8da437c2a72c364f81d53ca1491cdfca4c11c0f187912702e653523cb54f636ec910f8690f9af690cf632f0c98335b9ab10983676d0e67e1a59f3c7d7a5919c7
-  data.tar.gz: b8a13466f301863e6ae0e4e449f1da08d5ae17a7481bf2055e12976f6f14fda79a8ebd38aa5cfb5f097d21b36e43bc322f362483d1c522a005c9cf49e576f732
+  metadata.gz: ccc33fe06442b36b88724328ee1d5de5259bd4a8283b640b9e2372caea8895447a3074d14e0aaf49f752e47054dab5466b594f69e796d05b48b4b855ee2d2afe
+  data.tar.gz: 41c5fa46b0d0562ca026dfe8d9fb2448957db2a86fadfee856f6ff853b930fe8e8b8659438f32619d880836d0ee32533a4e1a07d35fcaea3edd41f2e93173157

data/{HISTORY.md → CHANGELOG.md}

@@ -1,3 +1,7 @@
+### 0.3.0
+
+### 0.2.0
+
 ### 0.0.1 / 2015-09-15
 
 * Everything is new. First release

data/Manifest.txt

@@ -1,17 +1,51 @@
-HISTORY.md
+CHANGELOG.md
 Manifest.txt
 README.md
 Rakefile
+config/groups_en.txt
+config/rounds_en.txt
+config/rounds_es.txt
+config/rounds_misc.txt
+lib/_cocos_.rb
 lib/rsssf.rb
-lib/rsssf/fetch.rb
-lib/rsssf/html2txt.rb
+lib/rsssf/convert/convert.rb
+lib/rsssf/convert/errata.rb
+lib/rsssf/convert/html_entities.rb
+lib/rsssf/convert/html_to_txt.rb
+lib/rsssf/convert/html_to_txt/beautify_anchors.rb
+lib/rsssf/convert/html_to_txt/make_heading.rb
+lib/rsssf/convert/html_to_txt/remove_emails.rb
+lib/rsssf/convert/html_to_txt/replace_a_href.rb
+lib/rsssf/convert/html_to_txt/replace_a_name.rb
+lib/rsssf/convert/html_to_txt/replace_heading.rb
+lib/rsssf/convert/html_to_txt/replace_hr.rb
+lib/rsssf/download.rb
+lib/rsssf/fmtfix/dates.rb
+lib/rsssf/fmtfix/dates_helpers.rb
+lib/rsssf/fmtfix/errata.rb
+lib/rsssf/fmtfix/fmtfix-base.rb
+lib/rsssf/fmtfix/fmtfix.rb
+lib/rsssf/fmtfix/goals.rb
+lib/rsssf/fmtfix/headers.rb
+lib/rsssf/fmtfix/outline.rb
+lib/rsssf/fmtfix/patch_headings.rb
+lib/rsssf/fmtfix/rounds.rb
+lib/rsssf/fmtfix/score.rb
+lib/rsssf/fmtfix/tables.rb
+lib/rsssf/fmtfix/topscorers.rb
+lib/rsssf/page-find_schedule.rb
+lib/rsssf/page-meta.rb
 lib/rsssf/page.rb
-lib/rsssf/patch.rb
-lib/rsssf/repo.rb
+lib/rsssf/parse_schedules.rb
+lib/rsssf/prepare/convert-links.rb
+lib/rsssf/prepare/convert-meta.rb
+lib/rsssf/prepare/convert-navlines.rb
+lib/rsssf/prepare/convert-postproc.rb
+lib/rsssf/prepare/convert.rb
+lib/rsssf/prepare/download.rb
+lib/rsssf/project.rb
 lib/rsssf/reports/page.rb
 lib/rsssf/reports/schedule.rb
 lib/rsssf/schedule.rb
 lib/rsssf/utils.rb
 lib/rsssf/version.rb
-test/helper.rb
-test/test_utils.rb

data/README.md

@@ -1,19 +1,18 @@
 # rsssf - tools 'n' scripts for RSSSF (Rec.Sport.Soccer Statistics Foundation) archive data
 
 
-* home  :: [github.com/sportdb/rsssf](https://github.com/sportdb/rsssf)
-* bugs  :: [github.com/sportdb/rsssf/issues](https://github.com/sportdb/rsssf/issues)
+* home  :: [github.com/rsssf/scripts](https://github.com/rsssf/scripts)
+* bugs  :: [github.com/rsssf/scripts/issues](https://github.com/rsssf/scripts/issues)
 * gem   :: [rubygems.org/gems/rsssf](https://rubygems.org/gems/rsssf)
 * rdoc  :: [rubydoc.info/gems/rsssf](http://rubydoc.info/gems/rsssf)
-* forum :: [opensport](http://groups.google.com/group/opensport)
 
 
-## What's the Rec.Sport.Soccer Statistics Foundation (RSSSF)? 
 
-The RSSSF collects and offers football (soccer) league tables, match results and more
-from all over the world online in plain text.
 
-Example:
+## What's the Rec.Sport.Soccer Statistics Foundation (RSSSF)?
+
+The RSSSF collects and offers football (soccer) league tables, match results and more
+from all over the world online in plain text. Example:
 
 ```
 Round 1
@@ -46,16 +45,44 @@ Coritiba        2-1 Atlético/MG
 
 ## Usage
 
-### Working with Pages
 
-To fetch pages from the world wide web use:
+### Download (and Cache)  Pages
+
+To download (and cache) pages from the world wide web use:
 
 ``` ruby
-page = RsssfPage.from_url( 'http://www.rsssf.com/tablese/eng2015.html')
+Rsssf.download_page( 'https://rsssf.org/tablese/eng2024.html',
+                      encoding: 'Windows-1252' )
+
+Rsssf.download_page( 'https://rsssf.org/tablesb/braz2024.html',
+                      encoding: 'Windows-1252' )
 ```
 
-Note: The `RsssfPageFetcher` will convert the rsssf archive page
-from hypertext (HTML) to plain text e.g.
+Note:  Most pages on rsssf.org use the Windows-1252 (character) encoding.
+To "auto-magically" convert to unicode (utf-8)
+add the encoding option (default is `UTF-8`).
+
+<!--
+Or as a convenience shortcut download (pre-configured table) pages by country code (e.g `eng` - England, `es` - Spain (España), `de` - Germany (Deutschland), `br` - Brazil (Brasil) etc.)
+and season (e.g. `2023/24` or `2024` etc.)
+
+``` ruby
+Rsssf.download_table( 'eng', season: '2023/24' )
+
+Rsssf.download_table( 'br', season: '2024' )
+```
+-->
+
+
+Note: The rsssf machinery uses a built-in (local) web cache.  All downloads get "auto-magically" cached (in `./cache/rsssf.org`).
+
+
+
+### Working with Pages  (from .HTML to .TXT)
+
+
+Note: The `RsssfPage` machinery lets you convert rsssf archive pages
+from hypertext (.html) to plain text (.txt) e.g.
 
 ```
 <hr>
@@ -88,7 +115,7 @@ will become
 ‹Conference›
 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 
-#### Premier League
+==== Premier League
 
 
 Final Table:
@@ -100,105 +127,100 @@ Final Table:
 ```
 
 
-### Working with Repos
 
-To fetch pages from the world wide web for many seasons in batch setup and use a repo.
+Tip - See <https://github.com/rsssf/tables>
+for a public online copy / mirror of converted
+tables in .txt (preserving the original "ad-hoc" formats).
+
+
+
+
+To fetch pages from the world wide web for many seasons in batch use
+a (tabular) datafile in the comma-separted values (.csv) format.
 
 Step 1: List all archive pages
 
-In the `tables/config.yml` list all archive pages to fetch. Example:
+In `eng.csv` list all archive pages to fetch. Example:
+
 
-``` yaml
-2010-11: tablese/eng2011.html
-2011-12: tablese/eng2012.html
-2012-13: tablese/eng2013.html
-2013-14: tablese/eng2014.html
-2014-15: tablese/eng2015.html
+``` csv
+season,  page,                 encoding
+2020/21, tablese/eng2021.html, windows-1252
+2021/22, tablese/eng2022.html, windows-1252
+2022/23, tablese/eng2023.html, windows-1252
+2023/24, tablese/eng2024.html, windows-1252
+2024/25, tablese/eng2025.html, windows-1252
 ```
 
-Step 2: Fetch all archive pages
+Step 2: Download all archive pages
 
 Use:
 
 ``` ruby
-repo = RsssfRepo.new( './eng-england', title: 'England (and Wales)' )
-repo.fetch_pages
-```
+pages = read_csv( './eng.csv')
 
-Bonus: To create a summary of all pages fetched (e.g. authors, last_updated, sections, etc.).
-Use:
+Rsssf::Prep.download_pages( pages )
 
-``` ruby
-repo.make_pages_report
-```
+## convert from .html to .txt
 
-Example - `tables/README.md`:
+Rsssf::Prep.convert_pages( pages, outdir: './tables' )
+```
 
 
-football.db RSSSF Archive Data Summary for England (and Wales)
 
-_Last Update: 2015-11-26 18:22:22 +0200_
 
-| Season  | File   | Authors  | Last Updated | Lines (Chars) | Sections | 
-| :------ | :------ | :------- | :----------- | ------------: | :------- |
-| 2014-15 | [eng2015.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2015.txt) | Ian King and Karel Stokkermans | 4 Jun 2015 | 1249 (34138) | Premier League, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2013-14 | [eng2014.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2014.txt) | Ian King and Karel Stokkermans | 5 Feb 2015 | 1254 (34294) | Premier League, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2012-13 | [eng2013.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2013.txt) | Karel Stokkermans | 5 Feb 2015 | 1269 (34531) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2011-12 | [eng2012.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2012.txt) | Karel Stokkermans | 5 Feb 2015 | 691 (21925) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2010-11 | [eng2011.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2011.txt) | Ian King, Karel Stokkermans and Jan Schoenmakers | 5 Feb 2015 | 959 (37393) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+For more and the latest news 'n' updates
+incl. how to mirror the rsssf.org website (all 40000+ pages),
+see the [`/scripts` source repo »](https://github.com/rsssf/scripts)
 
 
 That's it.
 
 
-### Preparing Archive Pages for SQL Database Imports (e.g. football.db)
 
-To import match schedules (fixtures and results) and more using the football.db machinery
-prepare "simple" single league (or cup) pages with standings tables etc. stripped out.
-For example, to break-out the Premier League and FA Cup from the `eng2015.txt`
-archive page use:
 
-``` ruby
-page = RsssfPage.from_url( 'http://www.rsssf.com/tablese/eng2015.html')
 
-schedule = page.find_schedule( header: 'Premier League')   ## returns RsssfSchedule obj
-schedule.save( './1-premierleague.txt' )
+### Preparing Archive Pages for SQL Database Imports
 
-schedule = page.find_schedule( header: 'FA Cup', cup: true )
-schedule.save( './facup.txt' )
-```
+Note:  The rsssf machinery includes helpers
+that try the best to convert the "ad-hoc" rsssf formats into
+the structured Football.TXT format but expect no miracles
+and fix any remaining errors "by hand"
+or with your own little search & replace scripts
+or why not with
+the help of the latest and greatest large language models (LLMs)?
 
 
+To split-up the all-in-one page and break-out sections
+such as the  Premier League or FA Cup from the `eng2015.txt`, for example,
+use:
 
-## Install
+``` ruby
+page = RsssfPage.read_txt( './pages/eng2015.txt')
 
-Just install the gem:
+schedule = page.find_schedule( header: 'Premier League')   ## returns RsssfSchedule obj
+schedule.save( './2014-15/1-premierleague.txt' )
 
-    $ gem install rsssf
+schedule = page.find_schedule( header: 'FA Cup' )
+schedule.save( './2014-15/facup.txt' )
+```
 
 
 
-## RSSSF Datasets
 
-See the rsssf github org for pre-processed ready-to-import datasets. Prepared repos include:
+Tip: See the rsssf github org for sample pages
+with format autofixes applied including:
 
-- [`eng-england`](https://github.com/rsssf/eng-england)    - rsssf archive data for England - Premier League, Championship, FA Cup etc.
-- [`de-deutschland`](https://github.com/rsssf/de-deutschland) - rsssf archive data for Germany (Deutschland) - Deutsche Bundesliga, 2. Bundesliga, 3. Liga, DFB Pokal etc.
-- [`es-espana`](https://github.com/rsssf/es-espana)      - rsssf archive data for España (Spain) - Primera División / La Liga, Copa de Rey, etc.
-- [`at-austria`](https://github.com/rsssf/at-austria)     - rsssf archive data for Austria (Österreich) - Österr. Bundesliga, Erste Liga, ÖFB Pokal etc.
-- [`br-brazil`](https://github.com/rsssf/br-brazil)      - rsssf archive data for Brazil (Brasil) - Campeonato Brasileiro Série A / Brasileirão etc.
+- [`/england`](https://github.com/rsssf/clubs/tree/master/england)    - rsssf archive data for England - Premier League, Championship, FA Cup etc.
+- [`/germany`](https://github.com/rsssf/clubs/tree/master/germany) - rsssf archive data for Germany (Deutschland) - Deutsche Bundesliga, 2. Bundesliga, 3. Liga, DFB Pokal etc.
+- [`/spain`](https://github.com/rsssf/clubs/tree/master/spain)      - rsssf archive data for España (Spain) - Primera División / La Liga, Copa de Rey, etc.
+- [`/austria`](https://github.com/rsssf/clubs/tree/master/austria)     - rsssf archive data for Austria (Österreich) - Österr. Bundesliga, Erste Liga, ÖFB Pokal etc.
+- [`/brazil`](https://github.com/rsssf/clubs/tree/master/brazil)      - rsssf archive data for Brazil (Brasil) - Campeonato Brasileiro Série A / Brasileirão etc.
 - and more
 
 
+
 ## License
 
 The `rsssf` scripts are dedicated to the public domain.
 Use it as you please with no restrictions whatsoever.
-
-
-## Questions? Comments?
-
-Send them along to the
-[Open Sports & Friends Forum/Mailing List](http://groups.google.com/group/opensport).
-Thanks!
-

data/Rakefile

@@ -8,25 +8,26 @@ Hoe.spec 'rsssf' do
   self.summary = "rsssf - tools 'n' scripts for RSSSF (Rec.Sport.Soccer Statistics Foundation) archive data"
   self.description = summary
 
-  self.urls    = ['https://github.com/sportdb/rsssf']
+  self.urls    = { home: 'https://github.com/rsssf/scripts' }
 
   self.author  = 'Gerald Bauer'
-  self.email   = 'opensport@googlegroups.com'
+  self.email   = 'gerald.bauer@gmail.com'
 
   # switch extension to .markdown for gihub formatting
   self.readme_file  = 'README.md'
-  self.history_file = 'HISTORY.md'
+  self.history_file = 'CHANGELOG.md'
 
   self.extra_deps = [
-    ['logutils'],
-    ['textutils'],
-    ['fetcher'],
+    ['cocos'],
+    ['season-formats'],
+    ['webget'],
   ]
 
+
   self.licenses = ['Public Domain']
 
   self.spec_extras = {
-    required_ruby_version: '>= 1.9.2'
+    required_ruby_version: '>= 2.2.2'
   }
 
 end

data/config/groups_en.txt

@@ -0,0 +1,44 @@
+##
+#  groups e.g.
+#  zones e.g.
+#  sections e.g.
+#  conferences e.g.
+##    
+
+## fix - move to groups.txt !!!
+## groups
+
+## e.g  group a, group b,
+##      group 1, group 2,
+##      group i, group ii, etc.
+##
+##  add a1,b1, etc too - why? why not?
+
+
+## make build-in - why? why not?
+
+Group [ ] (   [a-z]               \         
+            | \d{1,2}             \         
+            | i | ii | iii | iv   \ 
+          )
+
+## for groups and legs  
+##   use   ▪▪ (double) - why? why not? 
+  or zones (like groups)
+
+
+###
+## mls/usa
+Eastern·conference
+Central·conference
+Western·conference
+
+
+######
+## sco(tland)
+
+North·section 
+South·section
+Western·Isles·section
+Shetland·Island·section
+Orkney·Isles·section

data/config/rounds_en.txt

@@ -0,0 +1,283 @@
+##
+# rounds in english (en)
+#
+#
+#  note:  limited support for regex syntax supported
+#        use ? 
+#        use () for groups
+#    and use |
+
+#    note - leading and trailing spaces get removed
+#              and inline spaces squished (reduced to one)
+
+##
+##  note - inline spaces get translated to [] e.g.
+###        UNLESS (IF NOT) [] present in line!!!
+##   1/16 finals   =>  1/16 [ ] finals
+##   1/8 finals?   =>  1/8 [ ] finals? 
+
+
+##
+##   note - to processing modes
+##    if regex metachars found - line is regex!!
+##    if no regex metachars found - line is plain and get preprocesed
+   
+
+##
+##  check how to support here
+##               Round [ ]{1,2} \d{1,2}     ## e.g. round 1, round 2, etc.
+     
+
+##
+##  note - always remove leading & trailing spaces
+##              and squish inline spaces 
+##
+##  note -  group (...) gets changed to
+##                       non-capturing group (?: ...) 
+##
+##   use  \ for line-continuation!!!
+#
+#
+##  optional shortcut
+##     replace  Middle Dot (·)  Unicode: U+00B7 or
+##              Open Box (␣)    Unicode: U+2423 or
+##               with [ ]
+
+
+$ordinal    =  (    First | Second | Third | Fourth | Fifth     \
+                  | 1st   | 2nd    | 3rd   | 4th    | 5th   )   
+
+$opt_replay =  ( [ ] replays? )?
+
+
+## e.g. Playoff, Playoffs, Play-off, Play-offs 
+$playoff    =   play-?offs?
+
+## e.g. Match, Matches
+$match      =  match(es)?
+
+
+
+
+Preliminary·round
+
+## e.g. First round, Second round, ...
+$ordinal [ ] (   round   $opt_replay   \
+               | phase ) 
+
+
+Round·one
+
+
+
+## e.g. 1/8 finals, 8th finals...
+(   1/32            \
+  | 1/16 | 16th     \       
+  | 1/8  |  8th     \
+  | 1/4             \
+  | 1/2 ) [ ] finals?   $opt_replay
+
+
+## e.g. Eight-finals, Eight finals, Eightfinals, ...
+(   Eight    \
+  | Quarter  \
+  | Semi ) [ -]? finals?   $opt_replay
+   
+## short-forms
+Semis 
+Quarters
+
+
+## e.g. Round of 16, Last 16, ...
+(Round·of | Last ) [ ] ( 4 | 8 | 16 | 32)  
+
+
+
+  
+## e.g. fifth place, fifth-place, 5th place
+$fifthplace  =  ( fifth | 5th ) [ -] place
+
+$fifthplace [ ]  ( match | final | play[ -]?off )
+Match·for [ ] $fifthplace 
+
+
+## e.g. third place, third-place, 3rd place
+$thirdplace   =   ( third | 3rd ) [ -] place 
+
+$thirdplace [ ]  ( match | final | play[ -]?off ) 
+Match·for [ ] $thirdplace 
+
+          
+Finals?  $opt_replay
+
+
+###
+# more
+
+Final [ ] (round | group | pool)
+ 
+
+
+
+
+## e.g. Minor Semifinals
+(Minor | Major) [ ] Semi [ -]? finals?
+
+Preliminary·final
+Final·series        
+Grand·final           
+
+
+
+
+Conference·wild·card·round
+Wild·card·games
+
+(Eastern | Western) [ ] conference [ ] quarter [ -]? finals  
+(Eastern | Western) [ ] semi [ -]? finals
+(Eastern | Western) [ ] conference·final
+
+Conference·quarterfinals
+Conference·semifinals
+Conference·finals
+
+Divisional·finals
+Championship·final
+
+## used in msl/usa for "grand final"
+##  e.g. MLS cup, MLS cup final, MLS cup 2010, ...
+MLS·cup ([ ] final)? ([ ] \d{4})?           
+
+###   move to stage?  
+##     use/replace w/ generic Qualifaction <name> - why? why not?
+Qualification·MLS
+
+
+Super [ ] semi [ -]? final 
+
+Super [ ]? cup [ ] final
+
+
+Deciding·match
+Decider
+
+
+
+## in jp
+$playoff [ ] round
+Prime [ ] round 
+
+
+## in it (cup)
+##    with (no extra time) flag/note 
+## e.g.  Round 1 (no extra time)
+
+Round [ ] (1|2|3)       [ ] \( no [ ] extra [ ] time \)  
+Quarter [ -]? finals?   [ ] \( no [ ] extra [ ] time \)
+
+
+
+
+
+###
+#  moved matches  - find a better name?
+#   nachtragspiele (de) in en ??
+
+## e.g. Moved matches, 
+##     Moved match, ..
+Moved·$match
+
+## e.g. Match from round 26,
+##      Matches from round 26
+$match·from   [ ] round [ ] \d{1,2}
+   
+
+## standalone replay
+Replays?
+
+## e.g. Replayed match, 
+##      Replayed matches
+Replayed·$match 
+
+
+
+
+
+#############
+## add stages
+
+## e.g. First stage, Second stage, ... 
+$ordinal [ ] stage       
+
+(   Regular       \
+  | Group         \ 
+  | League        \
+  | Playoff ) [ ] stage
+
+Knock-?out·stage
+
+Regular·season
+
+Tournament·proper
+
+
+
+#####
+## play-offs & play-ins
+
+Championship       [ ] $playoff
+Europa·league      [ ] $playoff 
+Conference·league  [ ] $playoff
+
+Promotion [ ]             ($playoff | $match)
+Relegation [ ]            ($playoff | $match)
+Promotion/relegation [ ]  ($playoff | $match)
+Promotion/relegation [ ]  ($playoff | $match) [ ]  \
+                                  (   1st/2nd      \
+                                    | 2nd/3rd      \
+                                    | 3rd/4th)  [ ] level
+
+
+Relegation·playout
+Play-?out
+
+
+## e.g. Play-in round A, Play-in round B, Play-in round A-B
+##  note - escape for literal () required!!
+
+Play-in·round  [ ] ( A | B | A-B )
+
+
+## standalone playoff
+## e.g Playoff, Playoff 1, Playoff 2
+
+$playoff
+$playoff [ ] (1 | 2)?
+
+
+
+
+#############     
+## standalone legs
+
+## e.g. First legs
+##      Second leg replay
+$ordinal  [ ] legs?     $opt_replay   
+
+
+Third·leg·minigame            # in us          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+