nhkore 0.3.4 → 0.3.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1db415fe5fa2d6f112fae3ef163b79ddaa88d639de565c4cb35b40fa14fb5a66
-  data.tar.gz: c535ba1f719db2c64ba6713c21c6365f51e31361efd6046064cdf700219f30ab
+  metadata.gz: 4a7084fd097785a525cbe62fe59eaaa5a659c84a1673cb8663b3413b70e4e3a9
+  data.tar.gz: 7fd6c701cacc6bcbc2b55d8c33a1f9cefcbff36e39b2ebbb42022ce438188920
 SHA512:
-  metadata.gz: af517bada681dc850e60f1c0e275ba35d2177dc413b076612a091217fd653bba82be725c0247c52bb1e342489fe99c731ac7f5efc0b681d54cdcbb02cdb5360b
-  data.tar.gz: 579569f8c5201f3fe8b460281d585ac8b10c50c881d18ca6d6ff342fab0c178f0ad6e532df42200db117b0f8e85dbbace87846c85fcdd5c0c4eac1501f4b24e7
+  metadata.gz: e14fa1030416d99ad30c55ecb97b58135641e8008a485fe081104ab3cd177495889c3132015a8310b7aa9eee22e502e83de8378faee48c64785ed502eada4727
+  data.tar.gz: 16a856938c0a598122671cdf39ebaf569a9ec9d260a32f5830b9a21b1a3cf45b0940289c4136c273de1358211653ab3e250dd22bede936d312d4989edbb37cdd

data/.yardopts

@@ -0,0 +1,3 @@
+--files 'CHANGELOG.md,LICENSE.txt'
+--protected
+--readme 'README.md'

data/CHANGELOG.md

@@ -1,8 +1,80 @@
 # Changelog | NHKore
 
-Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+All notable changes to this project will be documented in this file.
+
+Format is based on [Keep a Changelog v1.0.0](https://keepachangelog.com/en/1.0.0),
+and this project adheres to [Semantic Versioning v2.0.0](https://semver.org/spec/v2.0.0.html).
+
+## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.9...HEAD)
+-
+
+
+## [v0.3.9] - 2021-06-26
+
+### Fixed
+- Reverted `App#refresh_cmd()` back to not copying over the `default_proc` (from v0.3.8). Because the old code didn't know about this, it created some unintended issues with command options. Nothing major, but for example, specifying `output.html` with the `sift` command would not produce HTML output (however, using the `-e html` option still worked). This is the only instance that I know of, but reverting the code back in case of more instances. In the future, I'll need to thoroughly test all CLI options after changing `App#refresh_cmd()` to copy over the `default_proc`, but for now, not worrying about it (as it's not necessary).
+
+
+## [v0.3.8] - 2021-06-26
+
+### Fixed
+- Fixed `App#refresh_cmd()` to also copy Cri's `default_proc` to the new Hash for the command options.
+- Fixed to check for non-strings for JSON & URI.
+    - For JSON, convert `StringIO` to string in `DictScraper.scrape()`.
+    - For URL, convert URL using `URI()` because `URI.parse()` will crash with a non-string (URI object) in `Scraper.open_url()`.
+- Fixed to scrape multiple HTML Ruby tag words (instead of just 1).
+    - I thought multiple Ruby bases/texts (`<rb>`/`<rt>`) were invalid, but after running into the article below and checking the HTML with a validator, it's actually valid HTML:
+          - https://www3.nhk.or.jp/news/easy/k10012759201000/k10012759201000.html
+          - No previous articles/URLs ran into this problem (would have raised an error), so it should only be a problem with this specific, new article.
+
+### Changed
+- Formatted/Linted all code using RuboCop.
+- Updated Gems.
+
+
+## [v0.3.7] - 2020-11-07
+
+### Changed
+- Updated Gem `attr_bool` to v0.2
+- Changed upper-case *'-V'* flag for *version* to be a lower-case *'-v'*
+    - Seems like a lot of apps/people expect this
+- Refactored/Formatted some code
+    - *nhkore.gemspec* especially
+- Added *samples/*, *Gemfile.lock*, and *.yardopts* to the files in *nhkore.gemspec*
+
+### Fixed
+- ArticleScraper
+    - Fixed to accept text nodes that have Kanji, due to bad article:
+        - https://www3.nhk.or.jp/news/easy/k10012639271000/k10012639271000.html
+            - `第３のビール` should have HTML ruby tags around *第*
+
+
+## [v0.3.6] - 2020-08-18
+
+### Added
+- `update_showcase` Rake task for development & personal site (GitHub Page)
+    - `$ bundle exec rake update_showcase`
+
+### Changed
+- Updated Gems
+
+### Fixed
+- ArticleScraper for title for specific site
+    - https://www3.nhk.or.jp/news/easy/article/disaster_earthquake_illust.html
+- Ignored `/cgi2.*enqform/` URLs from SearchScraper (Bing)
+- Added more detail to dictionary error in ArticleScraper
+
+
+## [v0.3.5] - 2020-05-04
+
+### Added
+- Added check for environment var `NO_COLOR`
+    - [https://no-color.org/](https://no-color.org/)
+
+### Fixed
+- Fixed URLs stored in YAML data to always be of type String (not URI)
+    - This initially caused a problem in DictScraper.parse_url() from ArticleScraper, but fixed it for all data
 
-## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.4...master)
 
 ## [v0.3.4] - 2020-04-25
 
@@ -25,12 +97,14 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
         - `-d '3-3'`
         - `-d '3'`
 
+
 ## [v0.3.3] - 2020-04-23
 
 ### Added
 - Added JSON support to Sifter & SiftCmd.
 - Added use of `attr_bool` Gem for `attr_accessor?` & `attr_reader?`.
 
+
 ## [v0.3.2] - 2020-04-22
 
 ### Added
@@ -54,6 +128,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - ArticleScraper
     - Renamed `mode` param to `strict`. `mode` was overshadowing File.open()'s in Scraper.
 
+
 ## [v0.3.1] - 2020-04-20
 
 ### Changed
@@ -71,6 +146,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - BingScraper
     - Fixed possible RSS infinite loop.
 
+
 ## [v0.3.0] - 2020-04-12
 
 ### Added
@@ -105,7 +181,9 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
         - ignore empty filenames in the Zip for safety.
         - ask to overwrite files instead of erroring.
 
+
 ## [v0.2.0] - 2020-04-01
+
 First working version.
 
 ### Added
@@ -141,7 +219,9 @@ First working version.
 - test/nhkore_tester.rb
     - Renamed to `test/nhkore/test_helper.rb`
 
+
 ## [v0.1.0] - 2020-02-24
+
 ### Added
 - .gitignore
 - CHANGELOG.md

data/Gemfile

@@ -1,24 +1,6 @@
 # encoding: UTF-8
 # frozen_string_literal: true
 
-#--
-# This file is part of NHKore.
-# Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)
-# 
-# NHKore is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Lesser General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# NHKore is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-# 
-# You should have received a copy of the GNU Lesser General Public License
-# along with NHKore.  If not, see <https://www.gnu.org/licenses/>.
-#++
-
 
 source 'https://rubygems.org'
 

data/Gemfile.lock

@@ -0,0 +1,89 @@
+PATH
+  remote: .
+  specs:
+    nhkore (0.3.9)
+      attr_bool (~> 0.2)
+      bimyou_segmenter (~> 1.2)
+      cri (~> 2.15)
+      down (~> 5.2)
+      highline (~> 2.0)
+      http-cookie (~> 1.0)
+      japanese_deinflector (~> 0.0)
+      nokogiri (~> 1.11)
+      psychgus (~> 1.3)
+      public_suffix (~> 4.0)
+      rainbow (~> 3.0)
+      rubyzip (~> 2.3)
+      tiny_segmenter (~> 0.0)
+      tty-progressbar (~> 0.18)
+      tty-spinner (~> 0.9)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    addressable (2.7.0)
+      public_suffix (>= 2.0.2, < 5.0)
+    attr_bool (0.2.2)
+    bimyou_segmenter (1.2.0)
+    cri (2.15.11)
+    domain_name (0.5.20190701)
+      unf (>= 0.0.5, < 1.0.0)
+    down (5.2.2)
+      addressable (~> 2.5)
+    highline (2.0.3)
+    http-cookie (1.0.4)
+      domain_name (~> 0.5)
+    japanese_deinflector (0.0.2)
+    mini_portile2 (2.5.3)
+    minitest (5.14.4)
+    nokogiri (1.11.7)
+      mini_portile2 (~> 2.5.0)
+      racc (~> 1.4)
+    psych (4.0.1)
+    psychgus (1.3.4)
+      psych (>= 3.0)
+    public_suffix (4.0.6)
+    racc (1.5.2)
+    rainbow (3.0.0)
+    rake (13.0.3)
+    raketeer (0.2.13)
+      rake
+    rdoc (6.3.1)
+    redcarpet (3.5.1)
+    rubyzip (2.3.0)
+    strings-ansi (0.2.0)
+    tiny_segmenter (0.0.6)
+    tty-cursor (0.7.1)
+    tty-progressbar (0.18.2)
+      strings-ansi (~> 0.2)
+      tty-cursor (~> 0.7)
+      tty-screen (~> 0.8)
+      unicode-display_width (>= 1.6, < 3.0)
+    tty-screen (0.8.1)
+    tty-spinner (0.9.3)
+      tty-cursor (~> 0.7)
+    unf (0.1.4)
+      unf_ext
+    unf_ext (0.0.7.7)
+    unicode-display_width (2.0.0)
+    yard (0.9.26)
+    yard_ghurt (1.2.1)
+      rake
+      yard
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler (~> 2.2)
+  minitest (~> 5.14)
+  nhkore!
+  rake (~> 13.0)
+  raketeer (~> 0.2)
+  rdoc (~> 6.3)
+  redcarpet (~> 3.5)
+  yard (~> 0.9)
+  yard_ghurt (~> 1.2)
+
+BUNDLED WITH
+   2.2.20

data/README.md

@@ -435,18 +435,18 @@ require 'nhkore/scraper'
 s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   open_timeout: 300, # Open timeout in seconds (default: nil)
   read_timeout: 300, # Read timeout in seconds (default: nil)
-  
+
   # Maximum number of times to retry the URL
   # - default: 3
   # - Open/connect will fail a couple of times on a bad/slow internet connection.
   max_retries: 10,
-  
+
   # Maximum number of redirects allowed.
   # - default: 3
   # - You can set this to nil or -1, but I recommend using a number
   #   for safety (infinite-loop attack).
   max_redirects: 1,
-  
+
   # How to check redirect URLs for safety.
   # - default: :strict
   # - nil      => do not check
@@ -455,7 +455,7 @@ s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   # - :strict  => check the scheme and domain
   #               (i.e., if https://bing.com, redirect URL must be https://bing.com)
   redirect_rule: :lenient,
-  
+
   # Set the HTTP header field 'cookie' from the 'set-cookie' response.
   # - default: false
   # - Currently uses the 'http-cookie' Gem.
@@ -463,7 +463,7 @@ s = NHKore::Scraper.new('https://www3.nhk.or.jp/news/easy/',
   # - Necessary for Search Engines or other sites that require cookies
   #   in order to block bots.
   eat_cookie: true,
-  
+
   # Set HTTP header fields.
   # - default: nil
   # - Necessary for Search Engines or other sites that try to block bots.
@@ -526,9 +526,9 @@ doc = ss.html_doc()
 
 doc.css('a').each() do |anchor|
   link = anchor['href']
-  
+
   next if ss.ignore_link?(link,cleaned: false)
-  
+
   if link.include?('https://www3.nhk')
     puts link
   end
@@ -549,9 +549,9 @@ page_num  = 1
 
 while !next_page.empty?()
   puts "Page #{page_num += 1}: #{next_page.count}"
-  
+
   bs = NHKore::BingScraper.new(:yasashii,url: next_page.url)
-  
+
   next_page = bs.scrape(slinks,next_page)
 end
 
@@ -570,24 +570,24 @@ require 'time'
 
 as = NHKore::ArticleScraper.new(
   'https://www3.nhk.or.jp/news/easy/k10011862381000/k10011862381000.html',
-  
+
   # If false, scrape the article leniently (for older articles which
   # may not have certain tags, etc.).
   # - default: true
   strict: false,
-  
+
   # {Dict} to use as the dictionary for words (Easy articles).
   # - default: :scrape
   # - nil     => don't scrape/use it (necessary for Regular articles)
   # - :scrape => auto-scrape it using {DictScraper}
   # - {Dict}  => your own {Dict}
   dict: nil,
-  
+
   # Date time to use as a fallback if the article doesn't have one
   # (for older articles).
   # - default: nil
   datetime: Time.new(2020,2,2),
-  
+
   # Year to use as a fallback if the article doesn't have one
   # (for older articles).
   # - default: nil
@@ -624,7 +624,7 @@ require 'nhkore/dict_scraper'
 url = 'https://www3.nhk.or.jp/news/easy/k10011862381000/k10011862381000.html'
 ds  = NHKore::DictScraper.new(
   url,
-  
+
   # Change the URL appropriately to the dictionary URL.
   # - default: true
   parse_url: true,
@@ -637,13 +637,13 @@ dict = ds.scrape()
 
 dict.entries.each() do |key,entry|
   entry.id
-  
+
   entry.defns.each() do |defn|
     defn.hyoukis.each() {|hyouki| }
     defn.text
     defn.words.each() {|word| }
   end
-  
+
   puts entry.build_hyouki()
   puts entry.build_defn()
   puts '---'
@@ -789,13 +789,13 @@ JPN = ['桜','ぶ','ブ']
 
 def fmt_jpn()
   fmt = []
-  
+
   JPN.each() do |x|
     x = yield(x)
     x = x ? "\u2B55" : Util::JPN_SPACE unless x.is_a?(String)
     fmt << x
   end
-  
+
   return "[ #{fmt.join(' | ')} ]"
 end
 
@@ -870,12 +870,16 @@ This will update *core/* for you:
     - *Raketary*: `$ raketary github_pkg`
 7. Run: `$ bundle exec rake release`
 
+Releasing new HTML file for website:
+
+1. `$ bundle exec rake update_showcase`
+
 ## License [^](#contents)
 
 [GNU LGPL v3+](LICENSE.txt)
 
 > NHKore (<https://github.com/esotericpig/nhkore>)  
-> Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)  
+> Copyright (c) 2020-2021 Jonathan Bradley Whited  
 > 
 > NHKore is free software: you can redistribute it and/or modify  
 > it under the terms of the GNU Lesser General Public License as published by  

data/Rakefile

@@ -1,24 +1,6 @@
 # encoding: UTF-8
 # frozen_string_literal: true
 
-#--
-# This file is part of NHKore.
-# Copyright (c) 2020 Jonathan Bradley Whited (@esotericpig)
-# 
-# NHKore is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Lesser General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# NHKore is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Lesser General Public License for more details.
-# 
-# You should have received a copy of the GNU Lesser General Public License
-# along with NHKore.  If not, see <https://www.gnu.org/licenses/>.
-#++
-
 
 require 'bundler/gem_tasks'
 
@@ -35,31 +17,31 @@ require 'nhkore/version'
 
 PKG_DIR = 'pkg'
 
-CLEAN.exclude('.git/','stock/')
+CLEAN.exclude('{.git,core,stock}/**/*')
 CLOBBER.include('doc/',File.join(PKG_DIR,''))
 
 
 task default: [:test]
 
 desc 'Generate documentation (YARDoc)'
-task :doc => [:yard,:yard_gfm_fix] do |task|
+task doc: %i[yard yard_gfm_fix] do |task|
 end
 
 desc "Package '#{File.join(NHKore::Util::CORE_DIR,'')}' data as a Zip file into '#{File.join(PKG_DIR,'')}'"
 task :pkg_core do |task|
   mkdir_p PKG_DIR
-  
+
   pattern = File.join(NHKore::Util::CORE_DIR,'*.{csv,html,json,yml}')
   zip_file = File.join(PKG_DIR,'nhkore-core.zip')
-  
-  sh 'zip','-9rv',zip_file,*Dir.glob(pattern).sort()
+
+  sh 'zip','-9rv',zip_file,*Dir.glob(pattern).sort
 end
 
-Rake::TestTask.new() do |task|
+Rake::TestTask.new do |task|
   task.libs = ['lib','test']
   task.pattern = File.join('test','**','*_test.rb')
   task.description += ": '#{task.pattern}'"
-  task.verbose = true
+  task.verbose = false
   task.warning = true
 end
 
@@ -68,58 +50,77 @@ end
 desc "Update '#{File.join(NHKore::Util::CORE_DIR,'')}' files for release"
 task :update_core do |task|
   require 'highline'
-  
-  CONTINUE_MSG = "\nContinue (y/n)? "
-  
+
+  continue_msg = "\nContinue (y/n)? "
+
   cmd = ['ruby','-w','./lib/nhkore.rb','-t','300','-m','10']
-  hl = HighLine.new()
-  
+  hl = HighLine.new
+
   next unless sh(*cmd,'se','ez','bing')
-  next unless hl.agree(CONTINUE_MSG)
+  next unless hl.agree(continue_msg)
   puts
-  
-  next unless sh(*cmd,'news','-s','100','ez')
-  next unless hl.agree(CONTINUE_MSG)
+
+  next unless sh(*cmd,'news','-s','500','ez')
+  next unless hl.agree(continue_msg)
   puts
-  
+
   next unless sh(*cmd,'sift','-e','csv' ,'ez')
+  puts
   next unless sh(*cmd,'sift','-e','html','ez')
+  puts
   next unless sh(*cmd,'sift','-e','json','ez')
+  puts
   next unless sh(*cmd,'sift','-e','yml' ,'ez')
+  puts
 end
 
-YARD::Rake::YardocTask.new() do |task|
-  task.files = [File.join('lib','**','*.rb')]
-  
-  task.options += ['--files','CHANGELOG.md,LICENSE.txt']
-  task.options += ['--readme','README.md']
-  
-  task.options << '--protected' # Show protected methods
+# @since 0.3.6
+desc 'Update showcase file for release'
+task :update_showcase do |task|
+  require 'highline'
+
+  showcase_file = File.join('.','nhkore-ez.html')
+
+  hl = HighLine.new
+
+  next unless sh('ruby','-w','./lib/nhkore.rb',
+    'sift','ez','--no-eng',
+    '--out',showcase_file,
+  )
+
+  next unless hl.agree("\nMove the file (y/n)? ")
+  puts
+  next unless sh('mv','-iv',showcase_file,
+    File.join('..','esotericpig.github.io','showcase',''),
+  )
+end
+
+YARD::Rake::YardocTask.new do |task|
   task.options += ['--template-path',File.join('yard','templates')]
   task.options += ['--title',"NHKore v#{NHKore::VERSION} Doc"]
 end
 
 # Execute "rake yard_gfm_fix" for production.
 # Execute "rake yard_gfm_fix[true]" for testing locally.
-YardGhurt::GFMFixTask.new() do |task|
+YardGhurt::GFMFixTask.new do |task|
   task.arg_names = [:dev]
   task.dry_run = false
   task.fix_code_langs = true
   task.md_files = ['index.html']
-  
-  task.before = Proc.new() do |task,args|
+
+  task.before = proc do |t,args|
     # Delete this file as it's never used (index.html is an exact copy).
-    YardGhurt::Util.rm_exist(File.join(task.doc_dir,'file.README.html'))
-    
+    YardGhurt::Util.rm_exist(File.join(t.doc_dir,'file.README.html'))
+
     # Root dir of my GitHub Page for CSS/JS.
-    GHP_ROOT = YardGhurt::Util.to_bool(args.dev) ? '../../esotericpig.github.io' : '../../..'
-    
-    task.css_styles << %Q(<link rel="stylesheet" type="text/css" href="#{GHP_ROOT}/css/prism.css" />)
-    task.js_scripts << %Q(<script src="#{GHP_ROOT}/js/prism.js"></script>)
+    ghp_root = YardGhurt::Util.to_bool(args.dev) ? '../../esotericpig.github.io' : '../../..'
+
+    t.css_styles << %Q(<link rel="stylesheet" type="text/css" href="#{ghp_root}/css/prism.css" />)
+    t.js_scripts << %Q(<script src="#{ghp_root}/js/prism.js"></script>)
   end
 end
 
 # Probably not useful for others.
-YardGhurt::GHPSyncTask.new() do |task|
+YardGhurt::GHPSyncTask.new do |task|
   task.ghp_dir = '../esotericpig.github.io/docs/nhkore/yardoc'
 end