nhkore 0.3.3 → 0.3.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0ca67a215cda7c49a82aa824c1322b49285abe332f627c9ad4fae774043cbfc9
-  data.tar.gz: b62a7e518787e89a3a54bcc66c191b4d3f005a911ab76861e3b118258f31b85f
+  metadata.gz: c63efbc2f65cfe83c7b55e53a0dfca329c2aded4c22ae05c2fb50583876452b4
+  data.tar.gz: 87c5116e11cb7e2dd4a5cdb86d6fc1a80ea58dd4efa7bc27ad448c25c4fad724
 SHA512:
-  metadata.gz: b4e84a07685c71400bd50b270c4ae662e6885f7149fc7ec3dec9476bf9b6b80f402d7f874ddcbef920c2b5034a1d39b44fbcb7e9ece06f3a2d517ca89e37de3d
-  data.tar.gz: 2527b477b7b7088f2612e4a05e0369b60cacb34bedb6ac59a3296643b6f59fcfce0c054ede67c68e0f4299864795bd79f04a85020d8f4c87b67f56c5a5dbeb77
+  metadata.gz: 68eb93da6d8f5c8ba3c4c58e0a9a71803dd4eefc6063df4ead9f0d06c0f1ba59892f5ddb43a9735c30ceaf85db63ed80c1b155bac1d5f0daf73f9cebbc7f6c6e
+  data.tar.gz: 33e9f4f770bceb2c0eb5d6d62781af400bc2b66f4ba4d4092b01b224bc365edef98cc47032f4b2389e04664f6cabcd2c02024139971bee207a121570805a6015

data/.yardopts

@@ -0,0 +1,3 @@
+--files 'CHANGELOG.md,LICENSE.txt'
+--protected
+--readme 'README.md'

data/CHANGELOG.md

@@ -1,8 +1,96 @@
 # Changelog | NHKore
 
-Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+All notable changes to this project will be documented in this file.
+
+Format is based on [Keep a Changelog v1.0.0](https://keepachangelog.com/en/1.0.0),
+and this project adheres to [Semantic Versioning v2.0.0](https://semver.org/spec/v2.0.0.html).
+
+## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.8...HEAD)
+-
+
+
+## [v0.3.8] - 2021-06-26
+
+### Fixed
+- Fixed `App#refresh_cmd()` to also copy Cri's `default_proc` to the new Hash for the command options.
+- Fixed to check for non-strings for JSON & URI.
+    - For JSON, convert `StringIO` to string in `DictScraper.scrape()`.
+    - For URL, convert URL using `URI()` because `URI.parse()` will crash with a non-string (URI object) in `Scraper.open_url()`.
+- Fixed to scrape multiple HTML Ruby tag words (instead of just 1).
+    - I thought multiple Ruby bases/texts (`<rb>`/`<rt>`) were invalid, but after running into the article below and checking the HTML with a validator, it's actually valid HTML:
+          - https://www3.nhk.or.jp/news/easy/k10012759201000/k10012759201000.html
+          - No previous articles/URLs ran into this problem (would have raised an error), so it should only be a problem with this specific, new article.
+
+### Changed
+- Formatted/Linted all code using RuboCop.
+- Updated Gems.
+
+
+## [v0.3.7] - 2020-11-07
+
+### Changed
+- Updated Gem `attr_bool` to v0.2
+- Changed upper-case *'-V'* flag for *version* to be a lower-case *'-v'*
+    - Seems like a lot of apps/people expect this
+- Refactored/Formatted some code
+    - *nhkore.gemspec* especially
+- Added *samples/*, *Gemfile.lock*, and *.yardopts* to the files in *nhkore.gemspec*
+
+### Fixed
+- ArticleScraper
+    - Fixed to accept text nodes that have Kanji, due to bad article:
+        - https://www3.nhk.or.jp/news/easy/k10012639271000/k10012639271000.html
+            - `第３のビール` should have HTML ruby tags around *第*
+
+
+## [v0.3.6] - 2020-08-18
+
+### Added
+- `update_showcase` Rake task for development & personal site (GitHub Page)
+    - `$ bundle exec rake update_showcase`
+
+### Changed
+- Updated Gems
+
+### Fixed
+- ArticleScraper for title for specific site
+    - https://www3.nhk.or.jp/news/easy/article/disaster_earthquake_illust.html
+- Ignored `/cgi2.*enqform/` URLs from SearchScraper (Bing)
+- Added more detail to dictionary error in ArticleScraper
+
+
+## [v0.3.5] - 2020-05-04
+
+### Added
+- Added check for environment var `NO_COLOR`
+    - [https://no-color.org/](https://no-color.org/)
+
+### Fixed
+- Fixed URLs stored in YAML data to always be of type String (not URI)
+    - This initially caused a problem in DictScraper.parse_url() from ArticleScraper, but fixed it for all data
+
+
+## [v0.3.4] - 2020-04-25
+
+### Added
+- DatetimeParser
+    - Extracted from SiftCmd into its own class
+    - Fixed some minor logic bugs from the old code
+    - Added new feature where 1 range can be empty:
+        - `sift ez -d '...2019'` (from = 1924)
+        - `sift ez -d '2019...'` (to = current year)
+        - `sift ez -d '...'` (still an error)
+- Added `update_core` rake task for dev
+    - Makes pushing a new release much easier
+    - See *Hacking.Releasing* section in *README*
+
+### Fixed
+- SiftCmd `parse_sift_datetime()` for `-d/--datetime` option
+    - Didn't work exactly right (as written in *README*) for some special inputs:
+        - `-d '2019...3'`
+        - `-d '3-3'`
+        - `-d '3'`
 
-## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.3...master)
 
 ## [v0.3.3] - 2020-04-23
 
@@ -10,6 +98,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Added JSON support to Sifter & SiftCmd.
 - Added use of `attr_bool` Gem for `attr_accessor?` & `attr_reader?`.
 
+
 ## [v0.3.2] - 2020-04-22
 
 ### Added
@@ -33,6 +122,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - ArticleScraper
     - Renamed `mode` param to `strict`. `mode` was overshadowing File.open()'s in Scraper.
 
+
 ## [v0.3.1] - 2020-04-20
 
 ### Changed
@@ -50,6 +140,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - BingScraper
     - Fixed possible RSS infinite loop.
 
+
 ## [v0.3.0] - 2020-04-12
 
 ### Added
@@ -84,7 +175,9 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
         - ignore empty filenames in the Zip for safety.
         - ask to overwrite files instead of erroring.
 
+
 ## [v0.2.0] - 2020-04-01
+
 First working version.
 
 ### Added
@@ -120,7 +213,9 @@ First working version.
 - test/nhkore_tester.rb
     - Renamed to `test/nhkore/test_helper.rb`
 
+
 ## [v0.1.0] - 2020-02-24
+
 ### Added
 - .gitignore
 - CHANGELOG.md

data/Gemfile

@@ -1,24 +1,6 @@
 # encoding: UTF-8
 # frozen_string_literal: true
 
-#--
-# This file is part of NHKore.
-# Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)
-# 
-# NHKore is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Lesser General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# NHKore is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-# 
-# You should have received a copy of the GNU Lesser General Public License
-# along with NHKore.  If not, see <https://www.gnu.org/licenses/>.
-#++
-
 
 source 'https://rubygems.org'
 

data/Gemfile.lock

@@ -0,0 +1,89 @@
+PATH
+  remote: .
+  specs:
+    nhkore (0.3.8)
+      attr_bool (~> 0.2)
+      bimyou_segmenter (~> 1.2)
+      cri (~> 2.15)
+      down (~> 5.2)
+      highline (~> 2.0)
+      http-cookie (~> 1.0)
+      japanese_deinflector (~> 0.0)
+      nokogiri (~> 1.11)
+      psychgus (~> 1.3)
+      public_suffix (~> 4.0)
+      rainbow (~> 3.0)
+      rubyzip (~> 2.3)
+      tiny_segmenter (~> 0.0)
+      tty-progressbar (~> 0.18)
+      tty-spinner (~> 0.9)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    addressable (2.7.0)
+      public_suffix (>= 2.0.2, < 5.0)
+    attr_bool (0.2.2)
+    bimyou_segmenter (1.2.0)
+    cri (2.15.11)
+    domain_name (0.5.20190701)
+      unf (>= 0.0.5, < 1.0.0)
+    down (5.2.2)
+      addressable (~> 2.5)
+    highline (2.0.3)
+    http-cookie (1.0.4)
+      domain_name (~> 0.5)
+    japanese_deinflector (0.0.2)
+    mini_portile2 (2.5.3)
+    minitest (5.14.4)
+    nokogiri (1.11.7)
+      mini_portile2 (~> 2.5.0)
+      racc (~> 1.4)
+    psych (4.0.1)
+    psychgus (1.3.4)
+      psych (>= 3.0)
+    public_suffix (4.0.6)
+    racc (1.5.2)
+    rainbow (3.0.0)
+    rake (13.0.3)
+    raketeer (0.2.13)
+      rake
+    rdoc (6.3.1)
+    redcarpet (3.5.1)
+    rubyzip (2.3.0)
+    strings-ansi (0.2.0)
+    tiny_segmenter (0.0.6)
+    tty-cursor (0.7.1)
+    tty-progressbar (0.18.2)
+      strings-ansi (~> 0.2)
+      tty-cursor (~> 0.7)
+      tty-screen (~> 0.8)
+      unicode-display_width (>= 1.6, < 3.0)
+    tty-screen (0.8.1)
+    tty-spinner (0.9.3)
+      tty-cursor (~> 0.7)
+    unf (0.1.4)
+      unf_ext
+    unf_ext (0.0.7.7)
+    unicode-display_width (2.0.0)
+    yard (0.9.26)
+    yard_ghurt (1.2.1)
+      rake
+      yard
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 2.2)
+  minitest (~> 5.14)
+  nhkore!
+  rake (~> 13.0)
+  raketeer (~> 0.2)
+  rdoc (~> 6.3)
+  redcarpet (~> 3.5)
+  yard (~> 0.9)
+  yard_ghurt (~> 1.2)
+
+BUNDLED WITH
+   2.2.20

data/README.md

@@ -26,6 +26,8 @@ This is similar to a [core word/vocabulary list](https://www.fluentin3months.com
         - [News Command](#news-command-)
 - [Using the Library](#using-the-library-)
 - [Hacking](#hacking-)
+    - [Updating](#updating-)
+    - [Releasing](#releasing-)
 - [License](#license-)
 
 ## For Non-Power Users [^](#contents)
@@ -433,18 +435,18 @@ require 'nhkore/scraper'
 s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   open_timeout: 300, # Open timeout in seconds (default: nil)
   read_timeout: 300, # Read timeout in seconds (default: nil)
-  
+
   # Maximum number of times to retry the URL
   # - default: 3
   # - Open/connect will fail a couple of times on a bad/slow internet connection.
   max_retries: 10,
-  
+
   # Maximum number of redirects allowed.
   # - default: 3
   # - You can set this to nil or -1, but I recommend using a number
   #   for safety (infinite-loop attack).
   max_redirects: 1,
-  
+
   # How to check redirect URLs for safety.
   # - default: :strict
   # - nil      => do not check
@@ -453,7 +455,7 @@ s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   # - :strict  => check the scheme and domain
   #               (i.e., if https://bing.com, redirect URL must be https://bing.com)
   redirect_rule: :lenient,
-  
+
   # Set the HTTP header field 'cookie' from the 'set-cookie' response.
   # - default: false
   # - Currently uses the 'http-cookie' Gem.
@@ -461,7 +463,7 @@ s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   # - Necessary for Search Engines or other sites that require cookies
   #   in order to block bots.
   eat_cookie: true,
-  
+
   # Set HTTP header fields.
   # - default: nil
   # - Necessary for Search Engines or other sites that try to block bots.
@@ -524,9 +526,9 @@ doc = ss.html_doc()
 
 doc.css('a').each() do |anchor|
   link = anchor['href']
-  
-  next if ss.ignore_link?(link)
-  
+
+  next if ss.ignore_link?(link,cleaned: false)
+
   if link.include?('https://www3.nhk')
     puts link
   end
@@ -547,9 +549,9 @@ page_num  = 1
 
 while !next_page.empty?()
   puts "Page #{page_num += 1}: #{next_page.count}"
-  
+
   bs = NHKore::BingScraper.new(:yasashii,url: next_page.url)
-  
+
   next_page = bs.scrape(slinks,next_page)
 end
 
@@ -564,27 +566,28 @@ end
 
 ```Ruby
 require 'nhkore/article_scraper'
+require 'time'
 
 as = NHKore::ArticleScraper.new(
   'https://www3.nhk.or.jp/news/easy/k10011862381000/k10011862381000.html',
-  
+
   # If false, scrape the article leniently (for older articles which
   # may not have certain tags, etc.).
   # - default: true
   strict: false,
-  
+
   # {Dict} to use as the dictionary for words (Easy articles).
   # - default: :scrape
   # - nil     => don't scrape/use it (necessary for Regular articles)
   # - :scrape => auto-scrape it using {DictScraper}
   # - {Dict}  => your own {Dict}
   dict: nil,
-  
+
   # Date time to use as a fallback if the article doesn't have one
   # (for older articles).
   # - default: nil
   datetime: Time.new(2020,2,2),
-  
+
   # Year to use as a fallback if the article doesn't have one
   # (for older articles).
   # - default: nil
@@ -621,7 +624,7 @@ require 'nhkore/dict_scraper'
 url = 'https://www3.nhk.or.jp/news/easy/k10011862381000/k10011862381000.html'
 ds  = NHKore::DictScraper.new(
   url,
-  
+
   # Change the URL appropriately to the dictionary URL.
   # - default: true
   parse_url: true,
@@ -634,13 +637,13 @@ dict = ds.scrape()
 
 dict.entries.each() do |key,entry|
   entry.id
-  
+
   entry.defns.each() do |defn|
     defn.hyoukis.each() {|hyouki| }
     defn.text
     defn.words.each() {|word| }
   end
-  
+
   puts entry.build_hyouki()
   puts entry.build_defn()
   puts '---'
@@ -687,6 +690,7 @@ end
 `Sifter` will sift & sort the `News` data into a single file. The data is sorted by frequency in descending order (i.e., most frequent words first).
 
 ```Ruby
+require 'nhkore/datetime_parser'
 require 'nhkore/news'
 require 'nhkore/sifter'
 require 'time'
@@ -698,7 +702,8 @@ sifter = NHKore::Sifter.new(news)
 sifter.caption = 'Sakura Fields Forever!'
 
 # Filter the data.
-#sifter.filter_by_datetime(Time.new(2019,12,5))
+sifter.filter_by_datetime(NHKore::DatetimeParser.parse_range('2019-12-4...7'))
+sifter.filter_by_datetime([Time.new(2019,12,4),Time.new(2019,12,7)])
 sifter.filter_by_datetime(
   from: Time.new(2019,12,4),to: Time.new(2019,12,7)
 )
@@ -727,13 +732,14 @@ if !File.exist?(file)
 end
 ```
 
-### Util & UserAgents
+### Util, UserAgents, & DatetimeParser
 
 These provide a variety of useful methods/constants.
 
 Here are some of the most useful ones:
 
 ```Ruby
+require 'nhkore/datetime_parser'
 require 'nhkore/user_agents'
 require 'nhkore/util'
 
@@ -759,14 +765,16 @@ puts
 puts '========'
 puts '[ Time ]'
 puts '========'
-puts "JST now:   #{Util.jst_now}"
+puts "JST now:   #{Util.jst_now()}"
 # Drops in JST_OFFSET, does not change hour/min.
 puts "JST time:  #{Util.jst_time(Time.now)}"
 puts "JST year:  #{Util::JST_YEAR}"
 puts "1999 sane? #{Util.sane_year?(1999)}" # true
 puts "1776 sane? #{Util.sane_year?(1776)}" # false
-puts "Guess 5:   #{Util.guess_year(5)}"    # 2005
-puts "Guess 99:  #{Util.guess_year(99)}"   # 1999
+puts "Guess 5:   #{DatetimeParser.guess_year(5)}"  # 2005
+puts "Guess 99:  #{DatetimeParser.guess_year(99)}" # 1999
+# => [2020-12-01 00:00:00 +0900, 2020-12-31 23:59:59 +0900]
+puts "Parse:     #{DatetimeParser.parse_range('2020-12')}"
 puts
 puts "JST timezone offset:        #{Util::JST_OFFSET}"
 puts "JST timezone offset hour:   #{Util::JST_OFFSET_HOUR}"
@@ -781,20 +789,20 @@ JPN = ['桜','ぶ','ブ']
 
 def fmt_jpn()
   fmt = []
-  
+
   JPN.each() do |x|
     x = yield(x)
     x = x ? "\u2B55" : Util::JPN_SPACE unless x.is_a?(String)
     fmt << x
   end
-  
+
   return "[ #{fmt.join(' | ')} ]"
 end
 
 puts "          #{fmt_jpn{|x| x}}"
-puts "Hiragana? #{fmt_jpn{|x| !!Util.hiragana?(x)}}"
-puts "Kana?     #{fmt_jpn{|x| !!Util.kana?(x)}}"
-puts "Kanji?    #{fmt_jpn{|x| !!Util.kanji?(x)}}"
+puts "Hiragana? #{fmt_jpn{|x| Util.hiragana?(x)}}"
+puts "Kana?     #{fmt_jpn{|x| Util.kana?(x)}}"
+puts "Kanji?    #{fmt_jpn{|x| Util.kanji?(x)}}"
 puts "Reduce:   #{Util.reduce_jpn_space("'     '")}"
 puts
 
@@ -842,16 +850,36 @@ You can make some changes/fixes to the code and then install your local version:
 
 `$ bundle exec rake install:local`
 
-### Releasing/Publishing
+### Updating [^](#contents)
+
+This will update *core/* for you:
+
+`$ bundle exec rake update_core`
+
+### Releasing [^](#contents)
+
+1. Update *CHANGELOG.md*, *version.rb*, & *Gemfile.lock*
+    - *Raketary*: `$ raketary bump -v`
+    - Run: `$ bundle update`
+2. Run: `$ bundle exec rake update_core`
+3. Run: `$ bundle exec rake clobber pkg_core`
+4. Create a new release & tag
+    - Add `pkg/nhkore-core.zip`
+5. Run: `$ git pull`
+6. Upload GitHub package
+    - *Raketary*: `$ raketary github_pkg`
+7. Run: `$ bundle exec rake release`
+
+Releasing new HTML file for website:
 
-`$ bundle exec rake release`
+1. `$ bundle exec rake update_showcase`
 
 ## License [^](#contents)
 
 [GNU LGPL v3+](LICENSE.txt)
 
 > NHKore (<https://github.com/esotericpig/nhkore>)  
-> Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)  
+> Copyright (c) 2020-2021 Jonathan Bradley Whited  
 > 
 > NHKore is free software: you can redistribute it and/or modify  
 > it under the terms of the GNU Lesser General Public License as published by