rsssf 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 92c74803ad71cb9cac8376ef3e0e01890352fc0a0edb01208eab3a9c41f60767
-  data.tar.gz: 874bdc292143352c88e23b44ed23abb98312d4afd2fd2cc797d078bce1eef0ed
+  metadata.gz: 77eac4fd6af25c171a9367afb06db3461e923c25f567222638d96f0665ca242e
+  data.tar.gz: b996e7cb14c9fb945c7138d3bd92e6214f922bcb2baf5e35da41b8ea957ca6f6
 SHA512:
-  metadata.gz: 0cc04f3a78663d870ed8a4d5b77813a1601e49a67fe3ea9265972f8da5b43b88adde78d184bedc320edb29439d7abb6439e6b045181bedea48c6d6be037d1a86
-  data.tar.gz: e92c7acc956eacd756665e83baee9373cf2af422da5deddd51e91e1266d7498e005a8d0b2e88868a23b3929c7b1900ef5bf20c7bd02e260bd07152e3e53d850a
+  metadata.gz: ccc33fe06442b36b88724328ee1d5de5259bd4a8283b640b9e2372caea8895447a3074d14e0aaf49f752e47054dab5466b594f69e796d05b48b4b855ee2d2afe
+  data.tar.gz: 41c5fa46b0d0562ca026dfe8d9fb2448957db2a86fadfee856f6ff853b930fe8e8b8659438f32619d880836d0ee32533a4e1a07d35fcaea3edd41f2e93173157

data/CHANGELOG.md

@@ -1,3 +1,5 @@
+### 0.3.0
+
 ### 0.2.0
 
 ### 0.0.1 / 2015-09-15

data/Manifest.txt

@@ -2,11 +2,48 @@ CHANGELOG.md
 Manifest.txt
 README.md
 Rakefile
+config/groups_en.txt
+config/rounds_en.txt
+config/rounds_es.txt
+config/rounds_misc.txt
+lib/_cocos_.rb
 lib/rsssf.rb
-lib/rsssf/convert.rb
+lib/rsssf/convert/convert.rb
+lib/rsssf/convert/errata.rb
+lib/rsssf/convert/html_entities.rb
+lib/rsssf/convert/html_to_txt.rb
+lib/rsssf/convert/html_to_txt/beautify_anchors.rb
+lib/rsssf/convert/html_to_txt/make_heading.rb
+lib/rsssf/convert/html_to_txt/remove_emails.rb
+lib/rsssf/convert/html_to_txt/replace_a_href.rb
+lib/rsssf/convert/html_to_txt/replace_a_name.rb
+lib/rsssf/convert/html_to_txt/replace_heading.rb
+lib/rsssf/convert/html_to_txt/replace_hr.rb
 lib/rsssf/download.rb
+lib/rsssf/fmtfix/dates.rb
+lib/rsssf/fmtfix/dates_helpers.rb
+lib/rsssf/fmtfix/errata.rb
+lib/rsssf/fmtfix/fmtfix-base.rb
+lib/rsssf/fmtfix/fmtfix.rb
+lib/rsssf/fmtfix/goals.rb
+lib/rsssf/fmtfix/headers.rb
+lib/rsssf/fmtfix/outline.rb
+lib/rsssf/fmtfix/patch_headings.rb
+lib/rsssf/fmtfix/rounds.rb
+lib/rsssf/fmtfix/score.rb
+lib/rsssf/fmtfix/tables.rb
+lib/rsssf/fmtfix/topscorers.rb
+lib/rsssf/page-find_schedule.rb
+lib/rsssf/page-meta.rb
 lib/rsssf/page.rb
-lib/rsssf/repo.rb
+lib/rsssf/parse_schedules.rb
+lib/rsssf/prepare/convert-links.rb
+lib/rsssf/prepare/convert-meta.rb
+lib/rsssf/prepare/convert-navlines.rb
+lib/rsssf/prepare/convert-postproc.rb
+lib/rsssf/prepare/convert.rb
+lib/rsssf/prepare/download.rb
+lib/rsssf/project.rb
 lib/rsssf/reports/page.rb
 lib/rsssf/reports/schedule.rb
 lib/rsssf/schedule.rb

data/README.md

@@ -1,8 +1,8 @@
 # rsssf - tools 'n' scripts for RSSSF (Rec.Sport.Soccer Statistics Foundation) archive data
 
 
-* home  :: [github.com/sportdb/sport.db.sources](https://github.com/sportdb/sport.db.sources)
-* bugs  :: [github.com/sportdb/sport.db.sources/issues](https://github.com/sportdb/sport.db.sources/issues)
+* home  :: [github.com/rsssf/scripts](https://github.com/rsssf/scripts)
+* bugs  :: [github.com/rsssf/scripts/issues](https://github.com/rsssf/scripts/issues)
 * gem   :: [rubygems.org/gems/rsssf](https://rubygems.org/gems/rsssf)
 * rdoc  :: [rubydoc.info/gems/rsssf](http://rubydoc.info/gems/rsssf)
 
@@ -46,7 +46,7 @@ Coritiba        2-1 Atlético/MG
 ## Usage
 
 
-### Download (and Cache ) Pages
+### Download (and Cache)  Pages
 
 To download (and cache) pages from the world wide web use:
 
@@ -59,9 +59,10 @@ Rsssf.download_page( 'https://rsssf.org/tablesb/braz2024.html',
 ```
 
 Note:  Most pages on rsssf.org use the Windows-1252 (character) encoding.
-To "auto-magically" convert to unicode (utf-8) 
-add the encoding option (default is `UTF-8`). 
+To "auto-magically" convert to unicode (utf-8)
+add the encoding option (default is `UTF-8`).
 
+<!--
 Or as a convenience shortcut download (pre-configured table) pages by country code (e.g `eng` - England, `es` - Spain (España), `de` - Germany (Deutschland), `br` - Brazil (Brasil) etc.)
 and season (e.g. `2023/24` or `2024` etc.)
 
@@ -70,17 +71,18 @@ Rsssf.download_table( 'eng', season: '2023/24' )
 
 Rsssf.download_table( 'br', season: '2024' )
 ```
+-->
 
 
-Note: The rsssf machinery uses a built-in web cache.  All downloads get "auto-magically" cached (in `./cache/rsssf.org`).
+Note: The rsssf machinery uses a built-in (local) web cache.  All downloads get "auto-magically" cached (in `./cache/rsssf.org`).
 
 
 
-### Working with Pages
+### Working with Pages  (from .HTML to .TXT)
 
 
-Note: The `RsssfPage` machinery will convert the rsssf archive page
-from hypertext (HTML) to plain text e.g.
+Note: The `RsssfPage` machinery lets you convert rsssf archive pages
+from hypertext (.html) to plain text (.txt) e.g.
 
 ```
 <hr>
@@ -113,7 +115,7 @@ will become
 ‹Conference›
 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 
-#### Premier League
+==== Premier League
 
 
 Final Table:
@@ -125,97 +127,100 @@ Final Table:
 ```
 
 
-### Working with Repos
 
-To fetch pages from the world wide web for many seasons in batch setup and use a repo.
+Tip - See <https://github.com/rsssf/tables>
+for a public online copy / mirror of converted
+tables in .txt (preserving the original "ad-hoc" formats).
+
+
+
+
+To fetch pages from the world wide web for many seasons in batch use
+a (tabular) datafile in the comma-separted values (.csv) format.
 
 Step 1: List all archive pages
 
-In the `tables/config.yml` list all archive pages to fetch. Example:
+In `eng.csv` list all archive pages to fetch. Example:
+
 
-``` yaml
-2010-11: tablese/eng2011.html
-2011-12: tablese/eng2012.html
-2012-13: tablese/eng2013.html
-2013-14: tablese/eng2014.html
-2014-15: tablese/eng2015.html
+``` csv
+season,  page,                 encoding
+2020/21, tablese/eng2021.html, windows-1252
+2021/22, tablese/eng2022.html, windows-1252
+2022/23, tablese/eng2023.html, windows-1252
+2023/24, tablese/eng2024.html, windows-1252
+2024/25, tablese/eng2025.html, windows-1252
 ```
 
-Step 2: Fetch all archive pages
+Step 2: Download all archive pages
 
 Use:
 
 ``` ruby
-repo = RsssfRepo.new( './england', title: 'England (and Wales)' )
-repo.fetch_pages
-```
+pages = read_csv( './eng.csv')
 
-Bonus: To create a summary of all pages fetched (e.g. authors, last_updated, sections, etc.).
-Use:
+Rsssf::Prep.download_pages( pages )
 
-``` ruby
-repo.make_pages_report
-```
+## convert from .html to .txt
 
-Example - `tables/README.md`:
+Rsssf::Prep.convert_pages( pages, outdir: './tables' )
+```
 
 
-football.db RSSSF Archive Data Summary for England (and Wales)
 
-_Last Update: 2015-11-26 18:22:22 +0200_
 
-| Season  | File   | Authors  | Last Updated | Lines (Chars) | Sections |
-| :------ | :------ | :------- | :----------- | ------------: | :------- |
-| 2014-15 | [eng2015.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2015.txt) | Ian King and Karel Stokkermans | 4 Jun 2015 | 1249 (34138) | Premier League, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2013-14 | [eng2014.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2014.txt) | Ian King and Karel Stokkermans | 5 Feb 2015 | 1254 (34294) | Premier League, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2012-13 | [eng2013.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2013.txt) | Karel Stokkermans | 5 Feb 2015 | 1269 (34531) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2011-12 | [eng2012.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2012.txt) | Karel Stokkermans | 5 Feb 2015 | 691 (21925) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
-| 2010-11 | [eng2011.txt](https://github.com/rsssf/eng-england/blob/master/tables/eng2011.txt) | Ian King, Karel Stokkermans and Jan Schoenmakers | 5 Feb 2015 | 959 (37393) | Premiership, Cup Tournaments, Championship, Division 1, Division 2, Conference |
+For more and the latest news 'n' updates
+incl. how to mirror the rsssf.org website (all 40000+ pages),
+see the [`/scripts` source repo »](https://github.com/rsssf/scripts)
 
 
 That's it.
 
 
-### Preparing Archive Pages for SQL Database Imports (e.g. football.db)
 
-To import match schedules (fixtures and results) and more using the football.db machinery
-prepare "simple" single league (or cup) pages with standings tables etc. stripped out.
-For example, to break-out the Premier League and FA Cup from the `eng2015.txt`
-archive page use:
+
+
+### Preparing Archive Pages for SQL Database Imports
+
+Note:  The rsssf machinery includes helpers
+that try the best to convert the "ad-hoc" rsssf formats into
+the structured Football.TXT format but expect no miracles
+and fix any remaining errors "by hand"
+or with your own little search & replace scripts
+or why not with
+the help of the latest and greatest large language models (LLMs)?
+
+
+To split-up the all-in-one page and break-out sections
+such as the  Premier League or FA Cup from the `eng2015.txt`, for example,
+use:
 
 ``` ruby
-page = RsssfPage.from_url( 'http://www.rsssf.com/tablese/eng2015.html')
+page = RsssfPage.read_txt( './pages/eng2015.txt')
 
 schedule = page.find_schedule( header: 'Premier League')   ## returns RsssfSchedule obj
-schedule.save( './1-premierleague.txt' )
+schedule.save( './2014-15/1-premierleague.txt' )
 
-schedule = page.find_schedule( header: 'FA Cup', cup: true )
-schedule.save( './facup.txt' )
+schedule = page.find_schedule( header: 'FA Cup' )
+schedule.save( './2014-15/facup.txt' )
 ```
 
 
 
-## RSSSF Datasets
 
-See the rsssf github org for pre-processed ready-to-import datasets. Prepared repos include:
+Tip: See the rsssf github org for sample pages
+with format autofixes applied including:
 
-- [`england`](https://github.com/rsssf/england)    - rsssf archive data for England - Premier League, Championship, FA Cup etc.
-- [`deutschland`](https://github.com/rsssf/deutschland) - rsssf archive data for Germany (Deutschland) - Deutsche Bundesliga, 2. Bundesliga, 3. Liga, DFB Pokal etc.
-- [`espana`](https://github.com/rsssf/espana)      - rsssf archive data for España (Spain) - Primera División / La Liga, Copa de Rey, etc.
-- [`austria`](https://github.com/rsssf/austria)     - rsssf archive data for Austria (Österreich) - Österr. Bundesliga, Erste Liga, ÖFB Pokal etc.
-- [`brazil`](https://github.com/rsssf/brazil)      - rsssf archive data for Brazil (Brasil) - Campeonato Brasileiro Série A / Brasileirão etc.
+- [`/england`](https://github.com/rsssf/clubs/tree/master/england)    - rsssf archive data for England - Premier League, Championship, FA Cup etc.
+- [`/germany`](https://github.com/rsssf/clubs/tree/master/germany) - rsssf archive data for Germany (Deutschland) - Deutsche Bundesliga, 2. Bundesliga, 3. Liga, DFB Pokal etc.
+- [`/spain`](https://github.com/rsssf/clubs/tree/master/spain)      - rsssf archive data for España (Spain) - Primera División / La Liga, Copa de Rey, etc.
+- [`/austria`](https://github.com/rsssf/clubs/tree/master/austria)     - rsssf archive data for Austria (Österreich) - Österr. Bundesliga, Erste Liga, ÖFB Pokal etc.
+- [`/brazil`](https://github.com/rsssf/clubs/tree/master/brazil)      - rsssf archive data for Brazil (Brasil) - Campeonato Brasileiro Série A / Brasileirão etc.
 - and more
 
 
+
 ## License
 
 The `rsssf` scripts are dedicated to the public domain.
 Use it as you please with no restrictions whatsoever.
-
-
-## Questions? Comments?
-
-Send them along to the
-[Open Sports & Friends Forum/Mailing List](http://groups.google.com/group/opensport).
-Thanks!
-

data/Rakefile

@@ -8,7 +8,7 @@ Hoe.spec 'rsssf' do
   self.summary = "rsssf - tools 'n' scripts for RSSSF (Rec.Sport.Soccer Statistics Foundation) archive data"
   self.description = summary
 
-  self.urls    = { home: 'https://github.com/sportdb/sport.db.sources' }
+  self.urls    = { home: 'https://github.com/rsssf/scripts' }
 
   self.author  = 'Gerald Bauer'
   self.email   = 'gerald.bauer@gmail.com'
@@ -20,7 +20,7 @@ Hoe.spec 'rsssf' do
   self.extra_deps = [
     ['cocos'],
     ['season-formats'],
-    ['rsssf-parser'],   ## add rsssf parser machinery & tool
+    ['webget'],
   ]
 
 

data/config/groups_en.txt

@@ -0,0 +1,44 @@
+##
+#  groups e.g.
+#  zones e.g.
+#  sections e.g.
+#  conferences e.g.
+##    
+
+## fix - move to groups.txt !!!
+## groups
+
+## e.g  group a, group b,
+##      group 1, group 2,
+##      group i, group ii, etc.
+##
+##  add a1,b1, etc too - why? why not?
+
+
+## make build-in - why? why not?
+
+Group [ ] (   [a-z]               \         
+            | \d{1,2}             \         
+            | i | ii | iii | iv   \ 
+          )
+
+## for groups and legs  
+##   use   ▪▪ (double) - why? why not? 
+  or zones (like groups)
+
+
+###
+## mls/usa
+Eastern·conference
+Central·conference
+Western·conference
+
+
+######
+## sco(tland)
+
+North·section 
+South·section
+Western·Isles·section
+Shetland·Island·section
+Orkney·Isles·section

data/config/rounds_en.txt

@@ -0,0 +1,283 @@
+##
+# rounds in english (en)
+#
+#
+#  note:  limited support for regex syntax supported
+#        use ? 
+#        use () for groups
+#    and use |
+
+#    note - leading and trailing spaces get removed
+#              and inline spaces squished (reduced to one)
+
+##
+##  note - inline spaces get translated to [] e.g.
+###        UNLESS (IF NOT) [] present in line!!!
+##   1/16 finals   =>  1/16 [ ] finals
+##   1/8 finals?   =>  1/8 [ ] finals? 
+
+
+##
+##   note - to processing modes
+##    if regex metachars found - line is regex!!
+##    if no regex metachars found - line is plain and get preprocesed
+   
+
+##
+##  check how to support here
+##               Round [ ]{1,2} \d{1,2}     ## e.g. round 1, round 2, etc.
+     
+
+##
+##  note - always remove leading & trailing spaces
+##              and squish inline spaces 
+##
+##  note -  group (...) gets changed to
+##                       non-capturing group (?: ...) 
+##
+##   use  \ for line-continuation!!!
+#
+#
+##  optional shortcut
+##     replace  Middle Dot (·)  Unicode: U+00B7 or
+##              Open Box (␣)    Unicode: U+2423 or
+##               with [ ]
+
+
+$ordinal    =  (    First | Second | Third | Fourth | Fifth     \
+                  | 1st   | 2nd    | 3rd   | 4th    | 5th   )   
+
+$opt_replay =  ( [ ] replays? )?
+
+
+## e.g. Playoff, Playoffs, Play-off, Play-offs 
+$playoff    =   play-?offs?
+
+## e.g. Match, Matches
+$match      =  match(es)?
+
+
+
+
+Preliminary·round
+
+## e.g. First round, Second round, ...
+$ordinal [ ] (   round   $opt_replay   \
+               | phase ) 
+
+
+Round·one
+
+
+
+## e.g. 1/8 finals, 8th finals...
+(   1/32            \
+  | 1/16 | 16th     \       
+  | 1/8  |  8th     \
+  | 1/4             \
+  | 1/2 ) [ ] finals?   $opt_replay
+
+
+## e.g. Eight-finals, Eight finals, Eightfinals, ...
+(   Eight    \
+  | Quarter  \
+  | Semi ) [ -]? finals?   $opt_replay
+   
+## short-forms
+Semis 
+Quarters
+
+
+## e.g. Round of 16, Last 16, ...
+(Round·of | Last ) [ ] ( 4 | 8 | 16 | 32)  
+
+
+
+  
+## e.g. fifth place, fifth-place, 5th place
+$fifthplace  =  ( fifth | 5th ) [ -] place
+
+$fifthplace [ ]  ( match | final | play[ -]?off )
+Match·for [ ] $fifthplace 
+
+
+## e.g. third place, third-place, 3rd place
+$thirdplace   =   ( third | 3rd ) [ -] place 
+
+$thirdplace [ ]  ( match | final | play[ -]?off ) 
+Match·for [ ] $thirdplace 
+
+          
+Finals?  $opt_replay
+
+
+###
+# more
+
+Final [ ] (round | group | pool)
+ 
+
+
+
+
+## e.g. Minor Semifinals
+(Minor | Major) [ ] Semi [ -]? finals?
+
+Preliminary·final
+Final·series        
+Grand·final           
+
+
+
+
+Conference·wild·card·round
+Wild·card·games
+
+(Eastern | Western) [ ] conference [ ] quarter [ -]? finals  
+(Eastern | Western) [ ] semi [ -]? finals
+(Eastern | Western) [ ] conference·final
+
+Conference·quarterfinals
+Conference·semifinals
+Conference·finals
+
+Divisional·finals
+Championship·final
+
+## used in msl/usa for "grand final"
+##  e.g. MLS cup, MLS cup final, MLS cup 2010, ...
+MLS·cup ([ ] final)? ([ ] \d{4})?           
+
+###   move to stage?  
+##     use/replace w/ generic Qualifaction <name> - why? why not?
+Qualification·MLS
+
+
+Super [ ] semi [ -]? final 
+
+Super [ ]? cup [ ] final
+
+
+Deciding·match
+Decider
+
+
+
+## in jp
+$playoff [ ] round
+Prime [ ] round 
+
+
+## in it (cup)
+##    with (no extra time) flag/note 
+## e.g.  Round 1 (no extra time)
+
+Round [ ] (1|2|3)       [ ] \( no [ ] extra [ ] time \)  
+Quarter [ -]? finals?   [ ] \( no [ ] extra [ ] time \)
+
+
+
+
+
+###
+#  moved matches  - find a better name?
+#   nachtragspiele (de) in en ??
+
+## e.g. Moved matches, 
+##     Moved match, ..
+Moved·$match
+
+## e.g. Match from round 26,
+##      Matches from round 26
+$match·from   [ ] round [ ] \d{1,2}
+   
+
+## standalone replay
+Replays?
+
+## e.g. Replayed match, 
+##      Replayed matches
+Replayed·$match 
+
+
+
+
+
+#############
+## add stages
+
+## e.g. First stage, Second stage, ... 
+$ordinal [ ] stage       
+
+(   Regular       \
+  | Group         \ 
+  | League        \
+  | Playoff ) [ ] stage
+
+Knock-?out·stage
+
+Regular·season
+
+Tournament·proper
+
+
+
+#####
+## play-offs & play-ins
+
+Championship       [ ] $playoff
+Europa·league      [ ] $playoff 
+Conference·league  [ ] $playoff
+
+Promotion [ ]             ($playoff | $match)
+Relegation [ ]            ($playoff | $match)
+Promotion/relegation [ ]  ($playoff | $match)
+Promotion/relegation [ ]  ($playoff | $match) [ ]  \
+                                  (   1st/2nd      \
+                                    | 2nd/3rd      \
+                                    | 3rd/4th)  [ ] level
+
+
+Relegation·playout
+Play-?out
+
+
+## e.g. Play-in round A, Play-in round B, Play-in round A-B
+##  note - escape for literal () required!!
+
+Play-in·round  [ ] ( A | B | A-B )
+
+
+## standalone playoff
+## e.g Playoff, Playoff 1, Playoff 2
+
+$playoff
+$playoff [ ] (1 | 2)?
+
+
+
+
+#############     
+## standalone legs
+
+## e.g. First legs
+##      Second leg replay
+$ordinal  [ ] legs?     $opt_replay   
+
+
+Third·leg·minigame            # in us          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

data/config/rounds_es.txt

@@ -0,0 +1,20 @@
+
+
+
+## used in mexico
+
+Playoffs [ ]   \( Liguilla \)
+
+Recalificación
+
+
+## used in argentina
+
+Round·of·64·-·32·avos·de·final
+Round·of·32·-·16·avos·de·final
+Round·of·16·-·Octavos·de·Final
+Quarter·finals·-·Cuartos·de·final
+Semi·finals·-·Semifinales
+
+Primera·fase·de·zonas·-·Phase·of·groups
+

data/config/rounds_misc.txt

@@ -0,0 +1,7 @@
+
+
+
+# in cz
+
+Play-off·o·umístění    
+Skupina·o·záchranu