nhkore 0.3.6 → 0.3.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 445adf6e8abd4da9fd6dd25e9632d5f477b467f6ce8c3dcecae87e3f61305d98
-  data.tar.gz: ca812639ff1edd8da835f5bbb2cde403c9cb63e17568fb3ec367eec00605ec17
+  metadata.gz: 8fd13f6cd3ad8e69daa44f5318515d5efe104c13a90a755c617cfd53302c4b50
+  data.tar.gz: 9ec8ea4b9a9f4f9862fb81611cf99b3bafdb20a6a35038fcc3d14cc934cbcdfc
 SHA512:
-  metadata.gz: 392607205c53aa2a5dfcde244e5fa6137483d216dc27becf06c76798209d2dcf328f17abee2026d795207d4e783a23fd108e615525445f52ca6442560600cd42
-  data.tar.gz: 7a1219623b6645bbc633ba9c94e767dcf86be8852a7228c1d5ddd3936f61b884897f680369d4c9d9db5aba8ab4561048d59aed15cecf7ba05695c1957f31b0ea
+  metadata.gz: 72b94a39a10e9e6dcfd74e032f45c679fee961f1f6c6f09a229720db07896566400983e83dd5a9a496f13ea3554c8be3ee2a80da6883ee44109c4bca917e14d4
+  data.tar.gz: 7a6a23790df9c138de0a1944d2f6146cba697303dc30a57eccba2f6b71342254afa67296cdfa70dae88e2aa78ccba5a401d2b8996b7b64b3a28460811b0283f7

data/.yardopts

@@ -0,0 +1,3 @@
+--files 'CHANGELOG.md,LICENSE.txt'
+--protected
+--readme 'README.md'

data/CHANGELOG.md

@@ -1,8 +1,59 @@
 # Changelog | NHKore
 
-Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+All notable changes to this project will be documented in this file.
+
+Format is based on [Keep a Changelog v1.0.0](https://keepachangelog.com/en/1.0.0),
+and this project adheres to [Semantic Versioning v2.0.0](https://semver.org/spec/v2.0.0.html).
+
+## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.10...HEAD)
+-
+
+
+## [v0.3.10] - 2021-07-13
+
+### Fixed
+- Updated `public_suffix` gem, which has a dependency on `addressable` gem vulnerability found by *Dependabot*.
+
+
+## [v0.3.9] - 2021-06-26
+
+### Fixed
+- Reverted `App#refresh_cmd()` back to not copying over the `default_proc` (from v0.3.8). Because the old code didn't know about this, it created some unintended issues with command options. Nothing major, but for example, specifying `output.html` with the `sift` command would not produce HTML output (however, using the `-e html` option still worked). This is the only instance that I know of, but reverting the code back in case of more instances. In the future, I'll need to thoroughly test all CLI options after changing `App#refresh_cmd()` to copy over the `default_proc`, but for now, not worrying about it (as it's not necessary).
+
+
+## [v0.3.8] - 2021-06-26
+
+### Fixed
+- Fixed `App#refresh_cmd()` to also copy Cri's `default_proc` to the new Hash for the command options.
+- Fixed to check for non-strings for JSON & URI.
+    - For JSON, convert `StringIO` to string in `DictScraper.scrape()`.
+    - For URL, convert URL using `URI()` because `URI.parse()` will crash with a non-string (URI object) in `Scraper.open_url()`.
+- Fixed to scrape multiple HTML Ruby tag words (instead of just 1).
+    - I thought multiple Ruby bases/texts (`<rb>`/`<rt>`) were invalid, but after running into the article below and checking the HTML with a validator, it's actually valid HTML:
+          - https://www3.nhk.or.jp/news/easy/k10012759201000/k10012759201000.html
+          - No previous articles/URLs ran into this problem (would have raised an error), so it should only be a problem with this specific, new article.
+
+### Changed
+- Formatted/Linted all code using RuboCop.
+- Updated Gems.
+
+
+## [v0.3.7] - 2020-11-07
+
+### Changed
+- Updated Gem `attr_bool` to v0.2
+- Changed upper-case *'-V'* flag for *version* to be a lower-case *'-v'*
+    - Seems like a lot of apps/people expect this
+- Refactored/Formatted some code
+    - *nhkore.gemspec* especially
+- Added *samples/*, *Gemfile.lock*, and *.yardopts* to the files in *nhkore.gemspec*
+
+### Fixed
+- ArticleScraper
+    - Fixed to accept text nodes that have Kanji, due to bad article:
+        - https://www3.nhk.or.jp/news/easy/k10012639271000/k10012639271000.html
+            - `第３のビール` should have HTML ruby tags around *第*
 
-## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.6...master)
 
 ## [v0.3.6] - 2020-08-18
 
@@ -19,6 +70,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Ignored `/cgi2.*enqform/` URLs from SearchScraper (Bing)
 - Added more detail to dictionary error in ArticleScraper
 
+
 ## [v0.3.5] - 2020-05-04
 
 ### Added
@@ -29,6 +81,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Fixed URLs stored in YAML data to always be of type String (not URI)
     - This initially caused a problem in DictScraper.parse_url() from ArticleScraper, but fixed it for all data
 
+
 ## [v0.3.4] - 2020-04-25
 
 ### Added
@@ -50,12 +103,14 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
         - `-d '3-3'`
         - `-d '3'`
 
+
 ## [v0.3.3] - 2020-04-23
 
 ### Added
 - Added JSON support to Sifter & SiftCmd.
 - Added use of `attr_bool` Gem for `attr_accessor?` & `attr_reader?`.
 
+
 ## [v0.3.2] - 2020-04-22
 
 ### Added
@@ -79,6 +134,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - ArticleScraper
     - Renamed `mode` param to `strict`. `mode` was overshadowing File.open()'s in Scraper.
 
+
 ## [v0.3.1] - 2020-04-20
 
 ### Changed
@@ -96,6 +152,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - BingScraper
     - Fixed possible RSS infinite loop.
 
+
 ## [v0.3.0] - 2020-04-12
 
 ### Added
@@ -130,7 +187,9 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
         - ignore empty filenames in the Zip for safety.
         - ask to overwrite files instead of erroring.
 
+
 ## [v0.2.0] - 2020-04-01
+
 First working version.
 
 ### Added
@@ -166,7 +225,9 @@ First working version.
 - test/nhkore_tester.rb
     - Renamed to `test/nhkore/test_helper.rb`
 
+
 ## [v0.1.0] - 2020-02-24
+
 ### Added
 - .gitignore
 - CHANGELOG.md

data/Gemfile

@@ -1,24 +1,6 @@
 # encoding: UTF-8
 # frozen_string_literal: true
 
-#--
-# This file is part of NHKore.
-# Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)
-# 
-# NHKore is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Lesser General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# NHKore is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-# 
-# You should have received a copy of the GNU Lesser General Public License
-# along with NHKore.  If not, see <https://www.gnu.org/licenses/>.
-#++
-
 
 source 'https://rubygems.org'
 

data/Gemfile.lock

@@ -0,0 +1,89 @@
+PATH
+  remote: .
+  specs:
+    nhkore (0.3.10)
+      attr_bool (~> 0.2)
+      bimyou_segmenter (~> 1.2)
+      cri (~> 2.15)
+      down (~> 5.2)
+      highline (~> 2.0)
+      http-cookie (~> 1.0)
+      japanese_deinflector (~> 0.0)
+      nokogiri (~> 1.11)
+      psychgus (~> 1.3)
+      public_suffix (~> 4.0)
+      rainbow (~> 3.0)
+      rubyzip (~> 2.3)
+      tiny_segmenter (~> 0.0)
+      tty-progressbar (~> 0.18)
+      tty-spinner (~> 0.9)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    addressable (2.8.0)
+      public_suffix (>= 2.0.2, < 5.0)
+    attr_bool (0.2.2)
+    bimyou_segmenter (1.2.0)
+    cri (2.15.11)
+    domain_name (0.5.20190701)
+      unf (>= 0.0.5, < 1.0.0)
+    down (5.2.2)
+      addressable (~> 2.5)
+    highline (2.0.3)
+    http-cookie (1.0.4)
+      domain_name (~> 0.5)
+    japanese_deinflector (0.0.2)
+    mini_portile2 (2.5.3)
+    minitest (5.14.4)
+    nokogiri (1.11.7)
+      mini_portile2 (~> 2.5.0)
+      racc (~> 1.4)
+    psych (4.0.1)
+    psychgus (1.3.4)
+      psych (>= 3.0)
+    public_suffix (4.0.6)
+    racc (1.5.2)
+    rainbow (3.0.0)
+    rake (13.0.6)
+    raketeer (0.2.13)
+      rake
+    rdoc (6.3.2)
+    redcarpet (3.5.1)
+    rubyzip (2.3.2)
+    strings-ansi (0.2.0)
+    tiny_segmenter (0.0.6)
+    tty-cursor (0.7.1)
+    tty-progressbar (0.18.2)
+      strings-ansi (~> 0.2)
+      tty-cursor (~> 0.7)
+      tty-screen (~> 0.8)
+      unicode-display_width (>= 1.6, < 3.0)
+    tty-screen (0.8.1)
+    tty-spinner (0.9.3)
+      tty-cursor (~> 0.7)
+    unf (0.1.4)
+      unf_ext
+    unf_ext (0.0.7.7)
+    unicode-display_width (2.0.0)
+    yard (0.9.26)
+    yard_ghurt (1.2.1)
+      rake
+      yard
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 2.2)
+  minitest (~> 5.14)
+  nhkore!
+  rake (~> 13.0)
+  raketeer (~> 0.2)
+  rdoc (~> 6.3)
+  redcarpet (~> 3.5)
+  yard (~> 0.9)
+  yard_ghurt (~> 1.2)
+
+BUNDLED WITH
+   2.2.20

data/README.md

@@ -435,18 +435,18 @@ require 'nhkore/scraper'
 s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   open_timeout: 300, # Open timeout in seconds (default: nil)
   read_timeout: 300, # Read timeout in seconds (default: nil)
-  
+
   # Maximum number of times to retry the URL
   # - default: 3
   # - Open/connect will fail a couple of times on a bad/slow internet connection.
   max_retries: 10,
-  
+
   # Maximum number of redirects allowed.
   # - default: 3
   # - You can set this to nil or -1, but I recommend using a number
   #   for safety (infinite-loop attack).
   max_redirects: 1,
-  
+
   # How to check redirect URLs for safety.
   # - default: :strict
   # - nil      => do not check
@@ -455,7 +455,7 @@ s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   # - :strict  => check the scheme and domain
   #               (i.e., if https://bing.com, redirect URL must be https://bing.com)
   redirect_rule: :lenient,
-  
+
   # Set the HTTP header field 'cookie' from the 'set-cookie' response.
   # - default: false
   # - Currently uses the 'http-cookie' Gem.
@@ -463,7 +463,7 @@ s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   # - Necessary for Search Engines or other sites that require cookies
   #   in order to block bots.
   eat_cookie: true,
-  
+
   # Set HTTP header fields.
   # - default: nil
   # - Necessary for Search Engines or other sites that try to block bots.
@@ -526,9 +526,9 @@ doc = ss.html_doc()
 
 doc.css('a').each() do |anchor|
   link = anchor['href']
-  
+
   next if ss.ignore_link?(link,cleaned: false)
-  
+
   if link.include?('https://www3.nhk')
     puts link
   end
@@ -549,9 +549,9 @@ page_num  = 1
 
 while !next_page.empty?()
   puts "Page #{page_num += 1}: #{next_page.count}"
-  
+
   bs = NHKore::BingScraper.new(:yasashii,url: next_page.url)
-  
+
   next_page = bs.scrape(slinks,next_page)
 end
 
@@ -570,24 +570,24 @@ require 'time'
 
 as = NHKore::ArticleScraper.new(
   'https://www3.nhk.or.jp/news/easy/k10011862381000/k10011862381000.html',
-  
+
   # If false, scrape the article leniently (for older articles which
   # may not have certain tags, etc.).
   # - default: true
   strict: false,
-  
+
   # {Dict} to use as the dictionary for words (Easy articles).
   # - default: :scrape
   # - nil     => don't scrape/use it (necessary for Regular articles)
   # - :scrape => auto-scrape it using {DictScraper}
   # - {Dict}  => your own {Dict}
   dict: nil,
-  
+
   # Date time to use as a fallback if the article doesn't have one
   # (for older articles).
   # - default: nil
   datetime: Time.new(2020,2,2),
-  
+
   # Year to use as a fallback if the article doesn't have one
   # (for older articles).
   # - default: nil
@@ -624,7 +624,7 @@ require 'nhkore/dict_scraper'
 url = 'https://www3.nhk.or.jp/news/easy/k10011862381000/k10011862381000.html'
 ds  = NHKore::DictScraper.new(
   url,
-  
+
   # Change the URL appropriately to the dictionary URL.
   # - default: true
   parse_url: true,
@@ -637,13 +637,13 @@ dict = ds.scrape()
 
 dict.entries.each() do |key,entry|
   entry.id
-  
+
   entry.defns.each() do |defn|
     defn.hyoukis.each() {|hyouki| }
     defn.text
     defn.words.each() {|word| }
   end
-  
+
   puts entry.build_hyouki()
   puts entry.build_defn()
   puts '---'
@@ -789,13 +789,13 @@ JPN = ['桜','ぶ','ブ']
 
 def fmt_jpn()
   fmt = []
-  
+
   JPN.each() do |x|
     x = yield(x)
     x = x ? "\u2B55" : Util::JPN_SPACE unless x.is_a?(String)
     fmt << x
   end
-  
+
   return "[ #{fmt.join(' | ')} ]"
 end
 
@@ -858,17 +858,23 @@ This will update *core/* for you:
 
 ### Releasing [^](#contents)
 
-1. Update *CHANGELOG.md*, *version.rb*, & *Gemfile.lock*
-    - *Raketary*: `$ raketary bump -v`
-    - Run: `$ bundle update`
-2. Run: `$ bundle exec rake update_core`
-3. Run: `$ bundle exec rake clobber pkg_core`
-4. Create a new release & tag
-    - Add `pkg/nhkore-core.zip`
-5. Run: `$ git pull`
-6. Upload GitHub package
-    - *Raketary*: `$ raketary github_pkg`
-7. Run: `$ bundle exec rake release`
+1. Update *CHANGELOG.md*, *version.rb*, & *Gemfile.lock*:
+    - With *Raketary*:
+        - `$ raketary bump -v`
+        - `$ raketary bump -p`
+    - `$ bundle update`
+    - `$ bundle outdated`
+2. Update *core* package:
+    - `$ bundle exec rake update_core`
+    - `$ bundle exec rake clobber pkg_core`
+3. Create a new tag & release:
+    - Note: make sure to add *pkg/nhkore-core.zip*
+    - `$ gh release create v0 pkg/*.gem pkg/*.zip`
+    - `$ git pull`
+4. Release to *GitHub Packages*:
+    - With *Raketary*: `$ raketary github_pkg`
+5. Release to *RubyGems*:
+    - `$ bundle exec rake release`
 
 Releasing new HTML file for website:
 
@@ -879,7 +885,7 @@ Releasing new HTML file for website:
 [GNU LGPL v3+](LICENSE.txt)
 
 > NHKore (<https://github.com/esotericpig/nhkore>)  
-> Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)  
+> Copyright (c) 2020-2021 Jonathan Bradley Whited  
 > 
 > NHKore is free software: you can redistribute it and/or modify  
 > it under the terms of the GNU Lesser General Public License as published by  

data/Rakefile

@@ -1,24 +1,6 @@
 # encoding: UTF-8
 # frozen_string_literal: true
 
-#--
-# This file is part of NHKore.
-# Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)
-# 
-# NHKore is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Lesser General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# NHKore is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-# 
-# You should have received a copy of the GNU Lesser General Public License
-# along with NHKore.  If not, see <https://www.gnu.org/licenses/>.
-#++
-
 
 require 'bundler/gem_tasks'
 
@@ -42,20 +24,20 @@ CLOBBER.include('doc/',File.join(PKG_DIR,''))
 task default: [:test]
 
 desc 'Generate documentation (YARDoc)'
-task :doc => [:yard,:yard_gfm_fix] do |task|
+task doc: %i[yard yard_gfm_fix] do |task|
 end
 
 desc "Package '#{File.join(NHKore::Util::CORE_DIR,'')}' data as a Zip file into '#{File.join(PKG_DIR,'')}'"
 task :pkg_core do |task|
   mkdir_p PKG_DIR
-  
+
   pattern = File.join(NHKore::Util::CORE_DIR,'*.{csv,html,json,yml}')
   zip_file = File.join(PKG_DIR,'nhkore-core.zip')
-  
-  sh 'zip','-9rv',zip_file,*Dir.glob(pattern).sort()
+
+  sh 'zip','-9rv',zip_file,*Dir.glob(pattern).sort
 end
 
-Rake::TestTask.new() do |task|
+Rake::TestTask.new do |task|
   task.libs = ['lib','test']
   task.pattern = File.join('test','**','*_test.rb')
   task.description += ": '#{task.pattern}'"
@@ -68,73 +50,77 @@ end
 desc "Update '#{File.join(NHKore::Util::CORE_DIR,'')}' files for release"
 task :update_core do |task|
   require 'highline'
-  
-  CONTINUE_MSG = "\nContinue (y/n)? "
-  
+
+  continue_msg = "\nContinue (y/n)? "
+
   cmd = ['ruby','-w','./lib/nhkore.rb','-t','300','-m','10']
-  hl = HighLine.new()
-  
+  hl = HighLine.new
+
   next unless sh(*cmd,'se','ez','bing')
-  next unless hl.agree(CONTINUE_MSG)
+  next unless hl.agree(continue_msg)
   puts
-  
-  next unless sh(*cmd,'news','-s','100','ez')
-  next unless hl.agree(CONTINUE_MSG)
+
+  next unless sh(*cmd,'news','-s','500','ez')
+  next unless hl.agree(continue_msg)
   puts
-  
+
   next unless sh(*cmd,'sift','-e','csv' ,'ez')
+  puts
   next unless sh(*cmd,'sift','-e','html','ez')
+  puts
   next unless sh(*cmd,'sift','-e','json','ez')
+  puts
   next unless sh(*cmd,'sift','-e','yml' ,'ez')
+  puts
 end
 
 # @since 0.3.6
 desc 'Update showcase file for release'
 task :update_showcase do |task|
   require 'highline'
-  
-  SHOWCASE_FILE = File.join('.','nhkore-ez.html')
-  
-  hl = HighLine.new()
-  
+
+  showcase_file = File.join('.','nhkore-ez.html')
+
+  hl = HighLine.new
+
   next unless sh('ruby','-w','./lib/nhkore.rb',
     'sift','ez','--no-eng',
-    '--out',SHOWCASE_FILE,
+    '--out',showcase_file,
   )
-  
+
   next unless hl.agree("\nMove the file (y/n)? ")
   puts
-  next unless sh('mv','-iv',SHOWCASE_FILE,
+  next unless sh('mv','-iv',showcase_file,
     File.join('..','esotericpig.github.io','showcase',''),
   )
 end
 
-YARD::Rake::YardocTask.new() do |task|
+YARD::Rake::YardocTask.new do |task|
   task.options += ['--template-path',File.join('yard','templates')]
   task.options += ['--title',"NHKore v#{NHKore::VERSION} Doc"]
 end
 
 # Execute "rake yard_gfm_fix" for production.
 # Execute "rake yard_gfm_fix[true]" for testing locally.
-YardGhurt::GFMFixTask.new() do |task|
+YardGhurt::GFMFixTask.new do |task|
   task.arg_names = [:dev]
   task.dry_run = false
   task.fix_code_langs = true
   task.md_files = ['index.html']
-  
-  task.before = Proc.new() do |task,args|
+
+  task.before = proc do |t,args|
     # Delete this file as it's never used (index.html is an exact copy).
-    YardGhurt::Util.rm_exist(File.join(task.doc_dir,'file.README.html'))
-    
+    YardGhurt::Util.rm_exist(File.join(t.doc_dir,'file.README.html'))
+
     # Root dir of my GitHub Page for CSS/JS.
-    GHP_ROOT = YardGhurt::Util.to_bool(args.dev) ? '../../esotericpig.github.io' : '../../..'
-    
-    task.css_styles << %Q(<link rel="stylesheet" type="text/css" href="#{GHP_ROOT}/css/prism.css" />)
-    task.js_scripts << %Q(<script src="#{GHP_ROOT}/js/prism.js"></script>)
+    ghp_root = YardGhurt::Util.to_bool(args.dev) ? '../../esotericpig.github.io' : '../../..'
+
+    t.css_styles << %Q(<link rel="stylesheet" type="text/css" href="#{ghp_root}/css/prism.css" />)
+    t.js_scripts << %Q(<script src="#{ghp_root}/js/prism.js"></script>)
   end
 end
 
 # Probably not useful for others.
-YardGhurt::GHPSyncTask.new() do |task|
+YardGhurt::GHPSyncTask.new do |task|
   task.ghp_dir = '../esotericpig.github.io/docs/nhkore/yardoc'
 end