nhkore 0.3.18 → 0.3.22

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +25 -9
  3. data/Gemfile +1 -0
  4. data/Gemfile.lock +2 -2
  5. data/README.md +1 -1
  6. data/bin/nhkore +1 -1
  7. data/lib/nhkore/app.rb +1 -4
  8. data/lib/nhkore/article.rb +1 -1
  9. data/lib/nhkore/article_scraper.rb +1 -1
  10. data/lib/nhkore/cleaner.rb +1 -1
  11. data/lib/nhkore/cli/fx_cmd.rb +1 -1
  12. data/lib/nhkore/cli/get_cmd.rb +6 -6
  13. data/lib/nhkore/cli/news_cmd.rb +2 -1
  14. data/lib/nhkore/cli/search_cmd.rb +3 -3
  15. data/lib/nhkore/cli/sift_cmd.rb +1 -1
  16. data/lib/nhkore/datetime_parser.rb +1 -1
  17. data/lib/nhkore/defn.rb +1 -1
  18. data/lib/nhkore/dict.rb +1 -1
  19. data/lib/nhkore/dict_scraper.rb +1 -1
  20. data/lib/nhkore/entry.rb +1 -1
  21. data/lib/nhkore/error.rb +1 -1
  22. data/lib/nhkore/fileable.rb +1 -1
  23. data/lib/nhkore/lib.rb +6 -10
  24. data/lib/nhkore/missingno.rb +1 -1
  25. data/lib/nhkore/news.rb +1 -1
  26. data/lib/nhkore/polisher.rb +1 -1
  27. data/lib/nhkore/scraper.rb +11 -3
  28. data/lib/nhkore/search_link.rb +1 -1
  29. data/lib/nhkore/search_scraper.rb +12 -5
  30. data/lib/nhkore/sifter.rb +4 -4
  31. data/lib/nhkore/splitter.rb +1 -1
  32. data/lib/nhkore/user_agents.rb +1 -1
  33. data/lib/nhkore/util.rb +1 -1
  34. data/lib/nhkore/variator.rb +1 -1
  35. data/lib/nhkore/version.rb +2 -2
  36. data/lib/nhkore/word.rb +1 -1
  37. data/lib/nhkore.rb +2 -8
  38. data/nhkore.gemspec +24 -22
  39. data/samples/looper.rb +1 -1
  40. data/test/nhkore/test_helper.rb +1 -1
  41. data/test/nhkore_test.rb +1 -1
  42. metadata +55 -55
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6c5e1fbcf5bb3f6dc23b6595e4138efb68597ac79cd01dac27cd242b914153ea
4
- data.tar.gz: 7608824e45a76a69175cf272e1301cfd062845a9de916b9034c0c4d13df73bb3
3
+ metadata.gz: 6d6942febe79f05d1cbd53e40f9048671ee0462d16e22945a6b7bdb3bb5bb2ae
4
+ data.tar.gz: 226acf4e93e6b95b475a1d42bc7a0d2dff82d5647613164fc2feba882e24b6c7
5
5
  SHA512:
6
- metadata.gz: 119c88fdd9377f96fe6bf3816d2c0b7eacbd5002fa2f0d8c852d8179f9058b8a8182ea5c828dbc61032647ac797c20289456144b30d0c5172a8be9ae527ffa14
7
- data.tar.gz: 73e2e75d1f4d444fe0fc9ade55bff57aaaa8876e5978aad62a925536bac8b0cf2a1d5db39d2ddfd5de9f5e65e95aa30adadbee70608111e1b2e310b6fd840e8b
6
+ metadata.gz: 40d4b5513b9a3d22e4969ef7d88c477d4c1dc118165660e751c8c9d742305d74470494597e5e12f61e15af9dc4c3f4e0bccdd82ac7e8488cb2a312a9749ada20
7
+ data.tar.gz: 00307ecff6363a7f0b80595fc7ec14a01f3e9e303d5d3023e084883a075bed12464a388ca16b6ff6ca143e993bdb5834493d361084452dd61846567173f0ef7f
data/CHANGELOG.md CHANGED
@@ -1,26 +1,42 @@
1
1
  # Changelog | NHKore
2
2
 
3
- All notable changes to this project will be documented in this file.
3
+ - [Keep a Changelog v1.0.0](https://keepachangelog.com/en/1.0.0)
4
+ - [Semantic Versioning v2.0.0](https://semver.org/spec/v2.0.0.html)
4
5
 
5
- Format is based on [Keep a Changelog v1.0.0](https://keepachangelog.com/en/1.0.0),
6
- and this project adheres to [Semantic Versioning v2.0.0](https://semver.org/spec/v2.0.0.html).
6
+ ## [Unreleased]
7
+ - https://github.com/esotericpig/nhkore/compare/v0.3.22...v0.3
7
8
 
8
- ## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.18...HEAD)
9
- -
9
+
10
+ ## [v0.3.22] - 2025-04-30
11
+
12
+ ### Changed
13
+ - Put v0.3 in its own branch to prepare for v0.4, which will heavily change.
14
+ - Changed v0.3 links to use v0.3 branch.
15
+ - Reverted the removing of `UserAgents` for v0.3 only.
16
+
17
+
18
+ ## [v0.3.19] - 2025-04-28
19
+
20
+ ### Fixed
21
+ - Fixed to include `fileutils` for `news` cmd when directory doesn't exist.
22
+
23
+ ### Changed
24
+ - Removed `UserAgents`. Replaced with Gem `ronin-web-user_agents`.
25
+ - Made some changes to `BingScraper`. Ultimately, it just doesn't work anymore, as Bing has become too strict. In the future, need to use a different search engine or a different way. Leaving for now as a zombie, and unnecessary for most NHKore functionality anyway.
10
26
 
11
27
 
12
28
  ## [v0.3.18] - 2025-04-24
13
29
 
30
+ ### Fixed
31
+ - Bing no longer allows `count`, so removed it. No workaround/fix for now....
32
+ - New NHK Easy pages no longer have a dictionary, so changed it to only warn instead of an exception.
33
+
14
34
  ### Changed
15
35
  - Changed Nokogiri gem version to `~> 1`, instead of `~> 1.xx`, as I got tired of the security alerts. Now, it will always be up-to-date and secure.
16
36
  - Updated gems.
17
37
  - Removed `yard_ghurt` gem.
18
38
  - Applied new RuboCop suggestions.
19
39
 
20
- ### Fixed
21
- - Bing no longer allows `count`, so removed it. No workaround/fix for now....
22
- - New NHK Easy pages no longer have a dictionary, so changed it to only warn instead of an exception.
23
-
24
40
 
25
41
  ## [v0.3.17] - 2024-09-03
26
42
 
data/Gemfile CHANGED
@@ -9,6 +9,7 @@ group :development,:test do
9
9
  gem 'bundler' ,'~> 2.6'
10
10
  gem 'rake' ,'~> 13.2'
11
11
  gem 'raketeer' ,'~> 0.2' # Extra Rake tasks.
12
+ # Doc.
12
13
  gem 'rdoc' ,'~> 6.13' # YARDoc RDoc (*.rb).
13
14
  gem 'redcarpet','~> 3.6' # YARDoc Markdown (*.md).
14
15
  gem 'yard' ,'~> 0.9' # YARDoc doc.
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- nhkore (0.3.18)
4
+ nhkore (0.3.22)
5
5
  attr_bool (~> 0.2)
6
6
  bimyou_segmenter (~> 1.2)
7
7
  cri (~> 2.15)
@@ -49,7 +49,7 @@ GEM
49
49
  stringio
50
50
  psychgus (1.3.5)
51
51
  psych (>= 3.0)
52
- public_suffix (6.0.1)
52
+ public_suffix (6.0.2)
53
53
  racc (1.8.1)
54
54
  rainbow (3.1.1)
55
55
  rake (13.2.1)
data/README.md CHANGED
@@ -884,7 +884,7 @@ Releasing new HTML file for website:
884
884
  [GNU LGPL v3+](LICENSE.txt)
885
885
 
886
886
  > NHKore (<https://github.com/esotericpig/nhkore>)
887
- > Copyright (c) 2020-2022 Jonathan Bradley Whited
887
+ > Copyright (c) 2020-2025 Bradley Whited
888
888
  >
889
889
  > NHKore is free software: you can redistribute it and/or modify
890
890
  > it under the terms of the GNU Lesser General Public License as published by
data/bin/nhkore CHANGED
@@ -4,7 +4,7 @@
4
4
 
5
5
  #--
6
6
  # This file is part of NHKore.
7
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
7
+ # Copyright (c) 2020 Bradley Whited
8
8
  #
9
9
  # SPDX-License-Identifier: LGPL-3.0-or-later
10
10
  #++
data/lib/nhkore/app.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2022 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
@@ -25,9 +25,6 @@ require 'nhkore/cli/search_cmd'
25
25
  require 'nhkore/cli/sift_cmd'
26
26
 
27
27
  module NHKore
28
- module CLI
29
- end
30
-
31
28
  ###
32
29
  # For disabling/enabling color output.
33
30
  ###
data/lib/nhkore/article.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/article_scraper.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/cleaner.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/cli/fx_cmd.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/cli/get_cmd.rb CHANGED
@@ -3,12 +3,13 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
10
10
 
11
11
  require 'nhkore/util'
12
+ require 'nhkore/version'
12
13
 
13
14
  module NHKore
14
15
  module CLI
@@ -16,7 +17,8 @@ module CLI
16
17
  DEFAULT_GET_CHUNK_SIZE = 4 * 1024
17
18
  DEFAULT_GET_URL_LENGTH = 11_000_000 # Just a generous estimation used as a fallback; may be outdated.
18
19
  GET_URL_FILENAME = 'nhkore-core.zip'
19
- GET_URL = "https://github.com/esotericpig/nhkore/releases/latest/download/#{GET_URL_FILENAME}".freeze
20
+ GET_URL = "https://github.com/esotericpig/nhkore/releases/download/v#{NHKore::VERSION}" \
21
+ "/#{GET_URL_FILENAME}".freeze
20
22
 
21
23
  def build_get_cmd
22
24
  app = self
@@ -36,9 +38,7 @@ module CLI
36
38
  DESC
37
39
 
38
40
  option :o,:out,'directory to save downloaded files to',argument: :required,default: Util::CORE_DIR,
39
- transform: lambda { |value|
40
- app.check_empty_opt(:out,value)
41
- }
41
+ transform: ->(value) { app.check_empty_opt(:out,value) }
42
42
  flag nil,:'show-url','show download URL and exit (for downloading manually)' do |_value,_cmd|
43
43
  puts GET_URL
44
44
  exit
@@ -69,7 +69,7 @@ module CLI
69
69
  out_dir = @cmd_opts[:out]
70
70
 
71
71
  begin
72
- start_spin('Opening URL')
72
+ start_spin("Opening URL: #{GET_URL} ")
73
73
 
74
74
  begin
75
75
  down = Down::NetHttp.open(GET_URL,rewindable: false,**@scraper_kargs)
data/lib/nhkore/cli/news_cmd.rb CHANGED
@@ -3,11 +3,12 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
10
10
 
11
+ require 'fileutils'
11
12
  require 'time'
12
13
 
13
14
  require 'nhkore/datetime_parser'
data/lib/nhkore/cli/search_cmd.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2022 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
@@ -27,7 +27,7 @@ module CLI
27
27
 
28
28
  description <<-DESC
29
29
  Search for links (using a Search Engine, etc.) to NHK News Web (Easy) &
30
- save to folder: #{SearchLinks::DEFAULT_DIR}
30
+ save to folder: '#{SearchLinks::DEFAULT_DIR}'
31
31
  DESC
32
32
 
33
33
  option :i,:in,<<-DESC,argument: :required,transform: lambda { |value|
@@ -38,7 +38,7 @@ module CLI
38
38
  }
39
39
  option :l,:loop,'number of times to repeat the search to ensure results',argument: :required,
40
40
  transform: lambda { |value|
41
- value = value.to_i
41
+ value = value.to_s.strip.to_i
42
42
  value = 1 if value < 1
43
43
  value
44
44
  }
data/lib/nhkore/cli/sift_cmd.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/datetime_parser.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/defn.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/dict.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/dict_scraper.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2022 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/entry.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/error.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/fileable.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/lib.rb CHANGED
@@ -3,11 +3,16 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
10
10
 
11
+ ###
12
+ # Include this file to only require the files needed to use this
13
+ # Gem as a library (i.e., don't include CLI-related files).
14
+ ###
15
+
11
16
  require 'nhkore/article'
12
17
  require 'nhkore/article_scraper'
13
18
  require 'nhkore/cleaner'
@@ -31,12 +36,3 @@ require 'nhkore/util'
31
36
  require 'nhkore/variator'
32
37
  require 'nhkore/version'
33
38
  require 'nhkore/word'
34
-
35
- module NHKore
36
- ###
37
- # Include this file to only require the files needed to use this
38
- # Gem as a library (i.e., don't include CLI-related files).
39
- ###
40
- module Lib
41
- end
42
- end
data/lib/nhkore/missingno.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/news.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/polisher.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/scraper.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
@@ -21,10 +21,18 @@ module NHKore
21
21
  extend AttrBool::Ext
22
22
 
23
23
  DEFAULT_HEADER = {
24
+ # See for better ones:
25
+ # - https://www.useragentstring.com/pages/Chrome/
24
26
  'user-agent' => UserAgents.sample,
25
- 'accept' => 'text/html,application/xhtml+xml,application/xml,application/rss+xml,text/xml;image/webp' \
26
- ',image/apng,*/*;application/signed-exchange',
27
+
28
+ 'accept' => 'text/html,application/xhtml+xml,application/xml,application/rss+xml,text/xml;' \
29
+ 'q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
30
+ 'accept-language' => 'en;q=0.9,ja-JP;q=0.8,ja',
31
+ 'cache-control' => 'max-age=0',
27
32
  'dnt' => '1',
33
+ 'ect' => '4g',
34
+ 'priority' => 'u=0, i',
35
+ 'upgrade-insecure-requests' => '1',
28
36
  }.freeze
29
37
 
30
38
  attr_accessor? :eat_cookie
data/lib/nhkore/search_link.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/search_scraper.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
@@ -111,19 +111,26 @@ module NHKore
111
111
  super(url,**kargs)
112
112
  end
113
113
 
114
- # rubocop:disable Lint/UnusedMethodArgument
115
114
  def self.build_url(site,count: DEFAULT_RESULT_COUNT,**_kargs)
116
115
  url = ''.dup
117
116
 
118
117
  url << 'https://www.bing.com/search?'
119
118
  url << URI.encode_www_form(
120
119
  q: "site:#{site}",
121
- # count: count # FIXME: `count` is no longer allowed on Bing.
120
+ count: count,
121
+
122
+ qs: 'n',
123
+ sp: '-1',
124
+ lq: '0',
125
+ pq: "site:#{site}",
126
+ sc: '1-25',
127
+ sk: '',
128
+ first: '1',
129
+ FORM: 'PERE',
122
130
  )
123
131
 
124
132
  return url
125
133
  end
126
- # rubocop:enable Lint/UnusedMethodArgument
127
134
 
128
135
  def scrape(slinks,page = NextPage.new())
129
136
  next_page,link_count = scrape_html(slinks,page)
@@ -147,7 +154,7 @@ module NHKore
147
154
 
148
155
  next if ignore_link?(href)
149
156
 
150
- if (md = href.match(/first=(\d+)/))
157
+ if (md = href.match(/first=(\d+)/i)) && href =~ /FORM=PERE/i
151
158
  count = md[1].to_i
152
159
 
153
160
  if count > page.count && (next_page.count < 0 || count < next_page.count)
data/lib/nhkore/sifter.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
@@ -322,7 +322,7 @@ module NHKore
322
322
  end
323
323
 
324
324
  def sift
325
- master_article = Article.new
325
+ result = Article.new
326
326
 
327
327
  @articles.each do |article|
328
328
  next if filter?(article)
@@ -333,11 +333,11 @@ module NHKore
333
333
  next if word.freq <= 1
334
334
  next if word.word =~ /\p{Latin}|[[:digit:]]/
335
335
 
336
- master_article.add_word(word,use_freq: true)
336
+ result.add_word(word,use_freq: true)
337
337
  end
338
338
  end
339
339
 
340
- words = master_article.words.values
340
+ words = result.words.values
341
341
 
342
342
  words.sort! do |word1,word2|
343
343
  # Order by freq DESC (most frequent words to top).
data/lib/nhkore/splitter.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/user_agents.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/util.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/variator.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore/version.rb CHANGED
@@ -3,11 +3,11 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
10
10
 
11
11
  module NHKore
12
- VERSION = '0.3.18'
12
+ VERSION = '0.3.22'
13
13
  end
data/lib/nhkore/word.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/lib/nhkore.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
@@ -15,14 +15,8 @@ if TESTING
15
15
  require 'bundler/setup'
16
16
  end
17
17
 
18
- require 'nhkore/app'
19
18
  require 'nhkore/lib'
20
-
21
- require 'nhkore/cli/fx_cmd'
22
- require 'nhkore/cli/get_cmd'
23
- require 'nhkore/cli/news_cmd'
24
- require 'nhkore/cli/search_cmd'
25
- require 'nhkore/cli/sift_cmd'
19
+ require 'nhkore/app'
26
20
 
27
21
  module NHKore
28
22
  def self.run(args = ARGV)
data/nhkore.gemspec CHANGED
@@ -6,7 +6,7 @@ require_relative 'lib/nhkore/version'
6
6
  Gem::Specification.new do |spec|
7
7
  spec.name = 'nhkore'
8
8
  spec.version = NHKore::VERSION
9
- spec.authors = ['Jonathan Bradley Whited']
9
+ spec.authors = ['Bradley Whited']
10
10
  spec.email = ['code@esotericpig.com']
11
11
  spec.licenses = ['LGPL-3.0-or-later']
12
12
  spec.homepage = 'https://github.com/esotericpig/nhkore'
@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
21
21
  'homepage_uri' => 'https://github.com/esotericpig/nhkore',
22
22
  'source_code_uri' => 'https://github.com/esotericpig/nhkore',
23
23
  'bug_tracker_uri' => 'https://github.com/esotericpig/nhkore/issues',
24
- 'changelog_uri' => 'https://github.com/esotericpig/nhkore/blob/master/CHANGELOG.md',
24
+ 'changelog_uri' => 'https://github.com/esotericpig/nhkore/blob/v0.3/CHANGELOG.md',
25
25
  # 'documentation_uri' => '',
26
26
  # 'wiki_uri' => '',
27
27
  # 'mailing_list_uri' => '',
@@ -51,23 +51,28 @@ Gem::Specification.new do |spec|
51
51
  spec.extra_rdoc_files,
52
52
  ].flatten
53
53
 
54
- spec.add_dependency 'attr_bool' ,'~> 0.2' # attr_accessor?/attr_reader?.
55
- spec.add_dependency 'bimyou_segmenter' ,'~> 1.2' # Split Japanese sentences into words.
56
- spec.add_dependency 'cri' ,'~> 2.15' # CLI commands/options.
57
- spec.add_dependency 'csv' ,'~> 3.3' # Output CSV.
58
- spec.add_dependency 'down' ,'~> 5.4' # Download files (GetCmd).
59
- spec.add_dependency 'highline' ,'~> 3.1' # CLI input/output.
60
- spec.add_dependency 'http-cookie' ,'~> 1.0' # Parse/Set cookies [(Bing)Scraper].
61
- spec.add_dependency 'japanese_deinflector','~> 0.0' # Un-conjugate Japanese words (dictionary form).
62
- spec.add_dependency 'nokogiri' ,'~> 1' # Scrape HTML.
63
- spec.add_dependency 'psychgus' ,'~> 1.3' # Style Psych YAML.
64
- spec.add_dependency 'public_suffix' ,'~> 6.0' # Parse URL domain names.
65
- spec.add_dependency 'rainbow' ,'~> 3.1' # CLI color output.
66
- spec.add_dependency 'rss' ,'~> 0.3' # Scrape RSS feeds [(Bing)Scraper].
67
- spec.add_dependency 'rubyzip' ,'~> 2.4' # Extract Zip files (GetCmd).
68
- spec.add_dependency 'tiny_segmenter' ,'~> 0.0' # Split Japanese sentences into words.
69
- spec.add_dependency 'tty-progressbar' ,'~> 0.18' # CLI progress bars.
70
- spec.add_dependency 'tty-spinner' ,'~> 0.9' # CLI spinning progress.
54
+ # Japanese.
55
+ spec.add_dependency 'bimyou_segmenter' ,'~> 1.2' # Splits Japanese sentences into words.
56
+ spec.add_dependency 'japanese_deinflector' ,'~> 0.0' # Un-conjugates Japanese words (dictionary form).
57
+ spec.add_dependency 'tiny_segmenter' ,'~> 0.0' # Splits Japanese sentences into words.
58
+ # Network/Scraping.
59
+ spec.add_dependency 'down' ,'~> 5.4' # Downloads files (GetCmd).
60
+ spec.add_dependency 'http-cookie' ,'~> 1.0' # Parses/Sets cookies [(Bing)Scraper].
61
+ spec.add_dependency 'nokogiri' ,'~> 1' # Parses HTML.
62
+ spec.add_dependency 'public_suffix' ,'~> 6.0' # Parses URL domain names.
63
+ spec.add_dependency 'rss' ,'~> 0.3' # Parses RSS feeds [(Bing)Scraper].
64
+ # Data/Files.
65
+ spec.add_dependency 'csv' ,'~> 3.3' # Outputs CSV.
66
+ spec.add_dependency 'psychgus' ,'~> 1.3' # Styles Psych YAML.
67
+ spec.add_dependency 'rubyzip' ,'~> 2.4' # Extracts Zip files (GetCmd).
68
+ # CLI.
69
+ spec.add_dependency 'cri' ,'~> 2.15' # CLI commands/options.
70
+ spec.add_dependency 'highline' ,'~> 3.1' # CLI IO.
71
+ spec.add_dependency 'rainbow' ,'~> 3.1' # CLI color output.
72
+ spec.add_dependency 'tty-progressbar' ,'~> 0.18' # CLI progress bars.
73
+ spec.add_dependency 'tty-spinner' ,'~> 0.9' # CLI spinning progress.
74
+ # Utils.
75
+ spec.add_dependency 'attr_bool' ,'~> 0.2' # attr_accessor?/attr_reader?.
71
76
 
72
77
  spec.post_install_message = <<~MSG
73
78
  +=============================================================================+
@@ -81,7 +86,4 @@ Gem::Specification.new do |spec|
81
86
  | Changelog: #{spec.metadata['changelog_uri']}
82
87
  +=============================================================================+
83
88
  MSG
84
-
85
- # Uncomment to see max line length:
86
- # puts spec.post_install_message.split("\n").map(&:length).max
87
89
  end
data/samples/looper.rb CHANGED
@@ -4,7 +4,7 @@
4
4
 
5
5
  #--
6
6
  # This file is part of NHKore.
7
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
7
+ # Copyright (c) 2020 Bradley Whited
8
8
  #
9
9
  # SPDX-License-Identifier: LGPL-3.0-or-later
10
10
  #++
data/test/nhkore/test_helper.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
data/test/nhkore_test.rb CHANGED
@@ -3,7 +3,7 @@
3
3
 
4
4
  #--
5
5
  # This file is part of NHKore.
6
- # Copyright (c) 2020-2021 Jonathan Bradley Whited
6
+ # Copyright (c) 2020 Bradley Whited
7
7
  #
8
8
  # SPDX-License-Identifier: LGPL-3.0-or-later
9
9
  #++
metadata CHANGED
@@ -1,141 +1,141 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: nhkore
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.18
4
+ version: 0.3.22
5
5
  platform: ruby
6
6
  authors:
7
- - Jonathan Bradley Whited
7
+ - Bradley Whited
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2025-04-24 00:00:00.000000000 Z
11
+ date: 2025-05-01 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
- name: attr_bool
14
+ name: bimyou_segmenter
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: '0.2'
19
+ version: '1.2'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: '0.2'
26
+ version: '1.2'
27
27
  - !ruby/object:Gem::Dependency
28
- name: bimyou_segmenter
28
+ name: japanese_deinflector
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
31
  - - "~>"
32
32
  - !ruby/object:Gem::Version
33
- version: '1.2'
33
+ version: '0.0'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
38
  - - "~>"
39
39
  - !ruby/object:Gem::Version
40
- version: '1.2'
40
+ version: '0.0'
41
41
  - !ruby/object:Gem::Dependency
42
- name: cri
42
+ name: tiny_segmenter
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
45
  - - "~>"
46
46
  - !ruby/object:Gem::Version
47
- version: '2.15'
47
+ version: '0.0'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
52
  - - "~>"
53
53
  - !ruby/object:Gem::Version
54
- version: '2.15'
54
+ version: '0.0'
55
55
  - !ruby/object:Gem::Dependency
56
- name: csv
56
+ name: down
57
57
  requirement: !ruby/object:Gem::Requirement
58
58
  requirements:
59
59
  - - "~>"
60
60
  - !ruby/object:Gem::Version
61
- version: '3.3'
61
+ version: '5.4'
62
62
  type: :runtime
63
63
  prerelease: false
64
64
  version_requirements: !ruby/object:Gem::Requirement
65
65
  requirements:
66
66
  - - "~>"
67
67
  - !ruby/object:Gem::Version
68
- version: '3.3'
68
+ version: '5.4'
69
69
  - !ruby/object:Gem::Dependency
70
- name: down
70
+ name: http-cookie
71
71
  requirement: !ruby/object:Gem::Requirement
72
72
  requirements:
73
73
  - - "~>"
74
74
  - !ruby/object:Gem::Version
75
- version: '5.4'
75
+ version: '1.0'
76
76
  type: :runtime
77
77
  prerelease: false
78
78
  version_requirements: !ruby/object:Gem::Requirement
79
79
  requirements:
80
80
  - - "~>"
81
81
  - !ruby/object:Gem::Version
82
- version: '5.4'
82
+ version: '1.0'
83
83
  - !ruby/object:Gem::Dependency
84
- name: highline
84
+ name: nokogiri
85
85
  requirement: !ruby/object:Gem::Requirement
86
86
  requirements:
87
87
  - - "~>"
88
88
  - !ruby/object:Gem::Version
89
- version: '3.1'
89
+ version: '1'
90
90
  type: :runtime
91
91
  prerelease: false
92
92
  version_requirements: !ruby/object:Gem::Requirement
93
93
  requirements:
94
94
  - - "~>"
95
95
  - !ruby/object:Gem::Version
96
- version: '3.1'
96
+ version: '1'
97
97
  - !ruby/object:Gem::Dependency
98
- name: http-cookie
98
+ name: public_suffix
99
99
  requirement: !ruby/object:Gem::Requirement
100
100
  requirements:
101
101
  - - "~>"
102
102
  - !ruby/object:Gem::Version
103
- version: '1.0'
103
+ version: '6.0'
104
104
  type: :runtime
105
105
  prerelease: false
106
106
  version_requirements: !ruby/object:Gem::Requirement
107
107
  requirements:
108
108
  - - "~>"
109
109
  - !ruby/object:Gem::Version
110
- version: '1.0'
110
+ version: '6.0'
111
111
  - !ruby/object:Gem::Dependency
112
- name: japanese_deinflector
112
+ name: rss
113
113
  requirement: !ruby/object:Gem::Requirement
114
114
  requirements:
115
115
  - - "~>"
116
116
  - !ruby/object:Gem::Version
117
- version: '0.0'
117
+ version: '0.3'
118
118
  type: :runtime
119
119
  prerelease: false
120
120
  version_requirements: !ruby/object:Gem::Requirement
121
121
  requirements:
122
122
  - - "~>"
123
123
  - !ruby/object:Gem::Version
124
- version: '0.0'
124
+ version: '0.3'
125
125
  - !ruby/object:Gem::Dependency
126
- name: nokogiri
126
+ name: csv
127
127
  requirement: !ruby/object:Gem::Requirement
128
128
  requirements:
129
129
  - - "~>"
130
130
  - !ruby/object:Gem::Version
131
- version: '1'
131
+ version: '3.3'
132
132
  type: :runtime
133
133
  prerelease: false
134
134
  version_requirements: !ruby/object:Gem::Requirement
135
135
  requirements:
136
136
  - - "~>"
137
137
  - !ruby/object:Gem::Version
138
- version: '1'
138
+ version: '3.3'
139
139
  - !ruby/object:Gem::Dependency
140
140
  name: psychgus
141
141
  requirement: !ruby/object:Gem::Requirement
@@ -151,103 +151,103 @@ dependencies:
151
151
  - !ruby/object:Gem::Version
152
152
  version: '1.3'
153
153
  - !ruby/object:Gem::Dependency
154
- name: public_suffix
154
+ name: rubyzip
155
155
  requirement: !ruby/object:Gem::Requirement
156
156
  requirements:
157
157
  - - "~>"
158
158
  - !ruby/object:Gem::Version
159
- version: '6.0'
159
+ version: '2.4'
160
160
  type: :runtime
161
161
  prerelease: false
162
162
  version_requirements: !ruby/object:Gem::Requirement
163
163
  requirements:
164
164
  - - "~>"
165
165
  - !ruby/object:Gem::Version
166
- version: '6.0'
166
+ version: '2.4'
167
167
  - !ruby/object:Gem::Dependency
168
- name: rainbow
168
+ name: cri
169
169
  requirement: !ruby/object:Gem::Requirement
170
170
  requirements:
171
171
  - - "~>"
172
172
  - !ruby/object:Gem::Version
173
- version: '3.1'
173
+ version: '2.15'
174
174
  type: :runtime
175
175
  prerelease: false
176
176
  version_requirements: !ruby/object:Gem::Requirement
177
177
  requirements:
178
178
  - - "~>"
179
179
  - !ruby/object:Gem::Version
180
- version: '3.1'
180
+ version: '2.15'
181
181
  - !ruby/object:Gem::Dependency
182
- name: rss
182
+ name: highline
183
183
  requirement: !ruby/object:Gem::Requirement
184
184
  requirements:
185
185
  - - "~>"
186
186
  - !ruby/object:Gem::Version
187
- version: '0.3'
187
+ version: '3.1'
188
188
  type: :runtime
189
189
  prerelease: false
190
190
  version_requirements: !ruby/object:Gem::Requirement
191
191
  requirements:
192
192
  - - "~>"
193
193
  - !ruby/object:Gem::Version
194
- version: '0.3'
194
+ version: '3.1'
195
195
  - !ruby/object:Gem::Dependency
196
- name: rubyzip
196
+ name: rainbow
197
197
  requirement: !ruby/object:Gem::Requirement
198
198
  requirements:
199
199
  - - "~>"
200
200
  - !ruby/object:Gem::Version
201
- version: '2.4'
201
+ version: '3.1'
202
202
  type: :runtime
203
203
  prerelease: false
204
204
  version_requirements: !ruby/object:Gem::Requirement
205
205
  requirements:
206
206
  - - "~>"
207
207
  - !ruby/object:Gem::Version
208
- version: '2.4'
208
+ version: '3.1'
209
209
  - !ruby/object:Gem::Dependency
210
- name: tiny_segmenter
210
+ name: tty-progressbar
211
211
  requirement: !ruby/object:Gem::Requirement
212
212
  requirements:
213
213
  - - "~>"
214
214
  - !ruby/object:Gem::Version
215
- version: '0.0'
215
+ version: '0.18'
216
216
  type: :runtime
217
217
  prerelease: false
218
218
  version_requirements: !ruby/object:Gem::Requirement
219
219
  requirements:
220
220
  - - "~>"
221
221
  - !ruby/object:Gem::Version
222
- version: '0.0'
222
+ version: '0.18'
223
223
  - !ruby/object:Gem::Dependency
224
- name: tty-progressbar
224
+ name: tty-spinner
225
225
  requirement: !ruby/object:Gem::Requirement
226
226
  requirements:
227
227
  - - "~>"
228
228
  - !ruby/object:Gem::Version
229
- version: '0.18'
229
+ version: '0.9'
230
230
  type: :runtime
231
231
  prerelease: false
232
232
  version_requirements: !ruby/object:Gem::Requirement
233
233
  requirements:
234
234
  - - "~>"
235
235
  - !ruby/object:Gem::Version
236
- version: '0.18'
236
+ version: '0.9'
237
237
  - !ruby/object:Gem::Dependency
238
- name: tty-spinner
238
+ name: attr_bool
239
239
  requirement: !ruby/object:Gem::Requirement
240
240
  requirements:
241
241
  - - "~>"
242
242
  - !ruby/object:Gem::Version
243
- version: '0.9'
243
+ version: '0.2'
244
244
  type: :runtime
245
245
  prerelease: false
246
246
  version_requirements: !ruby/object:Gem::Requirement
247
247
  requirements:
248
248
  - - "~>"
249
249
  - !ruby/object:Gem::Version
250
- version: '0.9'
250
+ version: '0.2'
251
251
  description: |
252
252
  Scrapes NHK News Web (Easy) for the word frequency (core list) for Japanese language learners.
253
253
  Includes a CLI app and a scraper library.
@@ -312,23 +312,23 @@ metadata:
312
312
  homepage_uri: https://github.com/esotericpig/nhkore
313
313
  source_code_uri: https://github.com/esotericpig/nhkore
314
314
  bug_tracker_uri: https://github.com/esotericpig/nhkore/issues
315
- changelog_uri: https://github.com/esotericpig/nhkore/blob/master/CHANGELOG.md
315
+ changelog_uri: https://github.com/esotericpig/nhkore/blob/v0.3/CHANGELOG.md
316
316
  post_install_message: |
317
317
  +=============================================================================+
318
- | NHKore v0.3.18
318
+ | NHKore v0.3.22
319
319
  |
320
320
  | You can now use [nhkore] on the command line.
321
321
  |
322
322
  | Homepage: https://github.com/esotericpig/nhkore
323
323
  | Code: https://github.com/esotericpig/nhkore
324
324
  | Bugs: https://github.com/esotericpig/nhkore/issues
325
- | Changelog: https://github.com/esotericpig/nhkore/blob/master/CHANGELOG.md
325
+ | Changelog: https://github.com/esotericpig/nhkore/blob/v0.3/CHANGELOG.md
326
326
  +=============================================================================+
327
327
  rdoc_options:
328
328
  - "--hyperlink-all"
329
329
  - "--show-hash"
330
330
  - "--title"
331
- - NHKore v0.3.18 Doc
331
+ - NHKore v0.3.22 Doc
332
332
  - "--main"
333
333
  - README.md
334
334
  require_paths: