nhkore 0.3.2 → 0.3.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cf151c3859812632f09b1a464164f31bb0ce050f37ed7e7377f76265571ebd41
-  data.tar.gz: 1f3ee801e7557731cae4aeacd3f18fea4d7f33ac65b6ec77511a7d3d8f17856a
+  metadata.gz: 0ca67a215cda7c49a82aa824c1322b49285abe332f627c9ad4fae774043cbfc9
+  data.tar.gz: b62a7e518787e89a3a54bcc66c191b4d3f005a911ab76861e3b118258f31b85f
 SHA512:
-  metadata.gz: 7e7d0d5b805ad6fa4312e8be26f3115dff18665b3762073c56db3a7a6a343a3ee6a05e47889e0abf7b62df3bb84cf5c977fce3efdfeb8a65c7bcff8167839d35
-  data.tar.gz: 957bc3da8492310d287a8947b9080f8be417f0874c3226db4f0bb63d020bee06c51a3da81c1fa3f779de22d354a32ab4cf41fc6f3018840774c31fd7060fbec3
+  metadata.gz: b4e84a07685c71400bd50b270c4ae662e6885f7149fc7ec3dec9476bf9b6b80f402d7f874ddcbef920c2b5034a1d39b44fbcb7e9ece06f3a2d517ca89e37de3d
+  data.tar.gz: 2527b477b7b7088f2612e4a05e0369b60cacb34bedb6ac59a3296643b6f59fcfce0c054ede67c68e0f4299864795bd79f04a85020d8f4c87b67f56c5a5dbeb77

data/CHANGELOG.md

@@ -2,7 +2,13 @@
 
 Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.2...master)
+## [[Unreleased]](https://github.com/esotericpig/nhkore/compare/v0.3.3...master)
+
+## [v0.3.3] - 2020-04-23
+
+### Added
+- Added JSON support to Sifter & SiftCmd.
+- Added use of `attr_bool` Gem for `attr_accessor?` & `attr_reader?`.
 
 ## [v0.3.2] - 2020-04-22
 
@@ -23,10 +29,7 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
     - In filter_by_datetime(), renamed keyword args `from_filter,to_filter` to simply `from,to`.
 
 ### Fixed
-- Reduced load time of app from ~1s to 0.3~0.5s.
-    - Moved many `require '...'` statements into methods.
-    - It looks ugly & is not a good coding practice, but a necessary evil.
-    - Load time is still pretty slow (but a lot better!).
+- Reduced load time of app a tiny bit more (see v0.3.1 for details).
 - ArticleScraper
     - Renamed `mode` param to `strict`. `mode` was overshadowing File.open()'s in Scraper.
 
@@ -40,7 +43,10 @@ Format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
     - Changed `dir_str?()` and `filename_str?()` to check any slash. Previously, it only checked the slash for your system. But now on both Windows & Linux, it will check for both `/` & `\`.
 
 ### Fixed
-- Reduced load time of app from ~1s to ~0.3-5s by moving some requires into methods.
+- Reduced load time of app from about 1s to about 0.3-0.5s.
+    - Moved many `require '...'` statements into methods.
+    - It looks ugly & is not good coding practice, but a necessary evil.
+    - Load time is still pretty slow (but a lot better!).
 - BingScraper
     - Fixed possible RSS infinite loop.
 

data/README.md

@@ -110,11 +110,12 @@ Example usage:
 
 `$ nhkore -t 300 -m 10 news -D -L -M -d '2011-03-07 06:30' easy -u 'https://www3.nhk.or.jp/news/easy/tsunamikeihou/index.html'`
 
-Now that the data from the article has been scraped, you can generate a CSV/HTML/YAML file of the words ordered by frequency:
+Now that the data from the article has been scraped, you can generate a CSV/HTML/JSON/YAML file of the words ordered by frequency:
 
 ```
 $ nhkore sift easy -e csv
 $ nhkore sift easy -e html
+$ nhkore sift easy -e json
 $ nhkore sift easy -e yml
 ```
 
@@ -154,11 +155,11 @@ After obtaining the scraped data, you can `sift` all of the data (or select data
 | --- | --- |
 | CSV | For uploading to a flashcard website (e.g., Memrise, Anki, Buffl) after changing the data appropriately. |
 | HTML | For comfortable viewing in a web browser or for sharing. |
-| YAML | For developers to automatically add translations or to manipulate the data in some other way programmatically. |
+| YAML/JSON | For developers to automatically add translations or to manipulate the data in some other way programmatically. |
 
 The data is sorted by frequency in descending order (i.e., most frequent words first).
 
-If you wish to sort/arrange the data in some other way, CSV editors (e.g., LibreOffice, WPS Office, Microsoft Office) can do this easily and efficiently, or if you are code-savvy, you can programmatically manipulate the CSV/YAML/HTML file.
+If you wish to sort/arrange the data in some other way, CSV editors (e.g., LibreOffice, WPS Office, Microsoft Office) can do this easily and efficiently, or if you are code-savvy, you can programmatically manipulate the CSV/YAML/JSON/HTML file.
 
 The defaults will sift all of the data into a CSV file, which may not be what you want:
 
@@ -203,7 +204,7 @@ You can save the data to a different format using one of these options:
 
 ```
 -e --ext=<value>    type of file (extension) to save;
-                    valid options: [csv, htm, html, yaml, yml];
+                    valid options: [csv, htm, html, json, yaml, yml];
                     not needed if you specify a file extension with
                     the '--out' option: '--out sift.html'
                     (default: csv)
@@ -704,16 +705,18 @@ sifter.filter_by_datetime(
 sifter.filter_by_title('桜')
 sifter.filter_by_url('k100')
 
-# Ignore (or blank out) certain columns from the output.
+# Ignore certain columns from the output.
 sifter.ignore(:defn)
 sifter.ignore(:eng)
 
-# An array of the filtered & sorted words.
-words = sifter.sift()
+# An array of the sifted words.
+words = sifter.sift() # Filtered & Sorted array of Word
+rows  = sifter.build_rows(words) # Ignored array of array
 
 # Choose the file format.
 #sifter.put_csv!()
 #sifter.put_html!()
+#sifter.put_json!()
 sifter.put_yaml!()
 
 # Save to a file.

data/Rakefile

@@ -49,7 +49,7 @@ desc "Package '#{File.join(NHKore::Util::CORE_DIR,'')}' data as a Zip file into
 task :pkg_core do |task|
   mkdir_p PKG_DIR
   
-  pattern = File.join(NHKore::Util::CORE_DIR,'*.{csv,html,yml}')
+  pattern = File.join(NHKore::Util::CORE_DIR,'*.{csv,html,json,yml}')
   zip_file = File.join(PKG_DIR,'nhkore-core.zip')
   
   sh 'zip','-9rv',zip_file,*Dir.glob(pattern).sort()

data/lib/nhkore.rb

@@ -29,28 +29,7 @@ if TESTING
 end
 
 require 'nhkore/app'
-require 'nhkore/article'
-require 'nhkore/article_scraper'
-require 'nhkore/cleaner'
-require 'nhkore/defn'
-require 'nhkore/dict'
-require 'nhkore/dict_scraper'
-require 'nhkore/entry'
-require 'nhkore/error'
-require 'nhkore/fileable'
-require 'nhkore/missingno'
-require 'nhkore/news'
-require 'nhkore/polisher'
-require 'nhkore/scraper'
-require 'nhkore/search_link'
-require 'nhkore/search_scraper'
-require 'nhkore/sifter'
-require 'nhkore/splitter'
-require 'nhkore/user_agents'
-require 'nhkore/util'
-require 'nhkore/variator'
-require 'nhkore/version'
-require 'nhkore/word'
+require 'nhkore/lib'
 
 require 'nhkore/cli/fx_cmd'
 require 'nhkore/cli/get_cmd'

data/lib/nhkore/article_scraper.rb

@@ -21,6 +21,7 @@
 #++
 
 
+require 'attr_bool'
 require 'digest'
 
 require 'nhkore/article'
@@ -49,12 +50,10 @@ module NHKore
     attr_accessor :missingno
     attr_reader :polishers
     attr_accessor :splitter
-    attr_accessor :strict
+    attr_accessor? :strict
     attr_reader :variators
     attr_accessor :year
     
-    alias_method :strict?,:strict
-    
     # @param dict [Dict,:scrape,nil] the {Dict} (dictionary) to use for {Word#defn} (definitions)
     #             [+:scrape+] auto-scrape it using {DictScraper}
     #             [+nil+]     don't scrape/use it

data/lib/nhkore/cli/sift_cmd.rb

@@ -39,7 +39,7 @@ module CLI
     DEFAULT_SIFT_EXT = :csv
     DEFAULT_SIFT_FUTSUU_FILE = "#{Sifter::DEFAULT_FUTSUU_FILE}{search.criteria}{file.ext}"
     DEFAULT_SIFT_YASASHII_FILE = "#{Sifter::DEFAULT_YASASHII_FILE}{search.criteria}{file.ext}"
-    SIFT_EXTS = [:csv,:htm,:html,:yaml,:yml]
+    SIFT_EXTS = [:csv,:htm,:html,:json,:yaml,:yml]
     
     # Order matters.
     SIFT_DATETIME_FMTS = [
@@ -364,6 +364,8 @@ module CLI
         sifter.put_csv!()
       when :htm,:html
         sifter.put_html!()
+      when :json
+        sifter.put_json!()
       when :yaml,:yml
         sifter.put_yaml!()
       else

data/lib/nhkore/scraper.rb

@@ -21,6 +21,7 @@
 #++
 
 
+require 'attr_bool'
 require 'nokogiri'
 require 'open-uri'
 
@@ -40,8 +41,8 @@ module NHKore
       'dnt' => '1',
     }
     
-    attr_accessor :eat_cookie
-    attr_accessor :is_file
+    attr_accessor? :eat_cookie
+    attr_accessor? :is_file
     attr_reader :kargs
     attr_accessor :max_redirects
     attr_accessor :max_retries
@@ -49,9 +50,6 @@ module NHKore
     attr_accessor :str_or_io
     attr_accessor :url
     
-    alias_method :eat_cookie?,:eat_cookie
-    alias_method :is_file?,:is_file
-    
     # +max_redirects+ defaults to 3 for safety (infinite-loop attack).
     # 
     # All URL options: https://ruby-doc.org/stdlib-2.7.0/libdoc/open-uri/rdoc/OpenURI/OpenRead.html

data/lib/nhkore/search_link.rb

@@ -21,6 +21,7 @@
 #++
 
 
+require 'attr_bool'
 require 'time'
 
 require 'nhkore/fileable'
@@ -35,13 +36,11 @@ module NHKore
   class SearchLink
     attr_accessor :datetime
     attr_accessor :futsuurl
-    attr_accessor :scraped
+    attr_accessor? :scraped
     attr_accessor :sha256
     attr_accessor :title
     attr_accessor :url
     
-    alias_method :scraped?,:scraped
-    
     def initialize(url,scraped: false)
       super()
       

data/lib/nhkore/sifter.rb

@@ -60,6 +60,40 @@ module NHKore
       @output = nil
     end
     
+    def build_header()
+      header = []
+      
+      header << 'Frequency' unless @ignores[:freq]
+      header << 'Word' unless @ignores[:word]
+      header << 'Kana' unless @ignores[:kana]
+      header << 'English' unless @ignores[:eng]
+      header << 'Definition' unless @ignores[:defn]
+      
+      return header
+    end
+    
+    def build_rows(words)
+      rows = []
+      
+      words.each() do |word|
+        rows << build_word_row(word)
+      end
+      
+      return rows
+    end
+    
+    def build_word_row(word)
+      row = []
+      
+      row << word.freq unless @ignores[:freq]
+      row << word.word unless @ignores[:word]
+      row << word.kana unless @ignores[:kana]
+      row << word.eng unless @ignores[:eng]
+      row << word.defn unless @ignores[:defn]
+      
+      return row
+    end
+    
     def filter?(article)
       return false if @filters.empty?()
       
@@ -151,26 +185,10 @@ module NHKore
       words = sift()
       
       @output = CSV.generate(headers: :first_row,write_headers: true) do |csv|
-        row = []
-        
-        row << 'Frequency' unless @ignores[:freq]
-        row << 'Word' unless @ignores[:word]
-        row << 'Kana' unless @ignores[:kana]
-        row << 'English' unless @ignores[:eng]
-        row << 'Definition' unless @ignores[:defn]
-        
-        csv << row
+        csv << build_header()
         
         words.each() do |word|
-          row = []
-          
-          row << word.freq unless @ignores[:freq]
-          row << word.word unless @ignores[:word]
-          row << word.kana unless @ignores[:kana]
-          row << word.eng unless @ignores[:eng]
-          row << word.defn unless @ignores[:defn]
-          
-          csv << row
+          csv << build_word_row(word)
         end
       end
       
@@ -232,7 +250,7 @@ module NHKore
         <h2>#{@caption}</h2>
         <table>
       EOH
-      #" # Fix for editor
+      #"
       
       # If have too few or too many '<col>', invalid HTML.
       @output << %Q{<col style="width:6em;">\n} unless @ignores[:freq]
@@ -242,20 +260,20 @@ module NHKore
       @output << "<col>\n" unless @ignores[:defn] # No width for defn, fills rest of page
       
       @output << '<tr>'
-      @output << '<th>Frequency</th>' unless @ignores[:freq]
-      @output << '<th>Word</th>' unless @ignores[:word]
-      @output << '<th>Kana</th>' unless @ignores[:kana]
-      @output << '<th>English</th>' unless @ignores[:eng]
-      @output << '<th>Definition</th>' unless @ignores[:defn]
+      
+      build_header().each() do |h|
+        @output << "<th>#{h}</th>"
+      end
+      
       @output << "</tr>\n"
       
       words.each() do |word|
         @output << '<tr>'
-        @output << "<td>#{Util.escape_html(word.freq.to_s())}</td>" unless @ignores[:freq]
-        @output << "<td>#{Util.escape_html(word.word.to_s())}</td>" unless @ignores[:word]
-        @output << "<td>#{Util.escape_html(word.kana.to_s())}</td>" unless @ignores[:kana]
-        @output << "<td>#{Util.escape_html(word.eng.to_s())}</td>" unless @ignores[:eng]
-        @output << "<td>#{Util.escape_html(word.defn.to_s())}</td>" unless @ignores[:defn]
+        
+        build_word_row(word).each() do |w|
+          @output << "<td>#{Util.escape_html(w.to_s())}</td>"
+        end
+        
         @output << "</tr>\n"
       end
       
@@ -264,31 +282,63 @@ module NHKore
         </body>
         </html>
       EOH
-      #/ # Fix for editor
+      #/
       
       return @output
     end
     
-    def put_yaml!()
+    def put_json!()
+      require 'json'
+      
       words = sift()
       
-      # Just blank out ignores.
-      if !@ignores.empty?()
-        words.each() do |word|
-          # word/kanji/kana do not have setters/mutators.
-          word.defn = nil if @ignores[:defn]
-          word.eng = nil if @ignores[:eng]
-          word.freq = nil if @ignores[:freq]
+      @output = ''.dup()
+      
+      @output << <<~EOJ
+        {
+        "caption": #{JSON.generate(@caption)},
+        "header": #{JSON.generate(build_header())},
+        "words": [
+      EOJ
+      
+      if !words.empty?()
+        0.upto(words.length - 2) do |i|
+          @output << "  #{JSON.generate(build_word_row(words[i]))},\n"
         end
+        
+        @output << "  #{JSON.generate(build_word_row(words[-1]))}\n"
       end
       
+      @output << "]\n}\n"
+      
+      return @output
+    end
+    
+    def put_yaml!()
+      require 'psychgus'
+      
+      words = sift()
+      
       yaml = {
         caption: @caption,
-        words: words
+        header: build_header(),
+        words: build_rows(words),
       }
       
+      header_styler = Class.new() do
+        include Psychgus::Styler
+        
+        def style_sequence(sniffer,node)
+          parent = sniffer.parent
+          
+          if !parent.nil?() && parent.node.respond_to?(:value) && parent.value == 'header'
+            node.style = Psychgus::SEQUENCE_FLOW
+          end
+        end
+      end
+      
       # Put each Word on one line (flow/inline style).
-      @output = Util.dump_yaml(yaml,flow_level: 4)
+      @output = Util.dump_yaml(yaml,flow_level: 4,stylers: header_styler.new())
       
       return @output
     end
@@ -306,7 +356,7 @@ module NHKore
       
       words = master_article.words.values()
       
-      words = words.sort() do |word1,word2|
+      words.sort!() do |word1,word2|
         # Order by freq DESC (most frequent words to top).
         i = (word2.freq <=> word1.freq)
         

data/lib/nhkore/util.rb

@@ -75,17 +75,20 @@ module NHKore
       return domain
     end
     
-    def self.dump_yaml(obj,flow_level: 8)
+    def self.dump_yaml(obj,flow_level: 8,stylers: nil)
       require 'psychgus'
       
+      stylers = Array(stylers)
+      
       return Psychgus.dump(obj,
         deref_aliases: true, # Dereference aliases for load_yaml()
+        header: true, # %YAML [version]
         line_width: 10000, # Try not to wrap; ichiman!
         stylers: [
           Psychgus::FlowStyler.new(flow_level), # Put extra details on one line (flow/inline style)
           Psychgus::NoSymStyler.new(cap: false), # Remove symbols, don't capitalize
           Psychgus::NoTagStyler.new(), # Remove class names (tags)
-        ],
+        ].concat(stylers),
       )
     end
     

data/lib/nhkore/version.rb

@@ -22,5 +22,5 @@
 
 
 module NHKore
-  VERSION = '0.3.2'
+  VERSION = '0.3.3'
 end

data/nhkore.gemspec

@@ -59,6 +59,7 @@ Gem::Specification.new() do |spec|
   
   spec.requirements << 'Nokogiri: https://www.nokogiri.org/tutorials/installing_nokogiri.html'
   
+  spec.add_runtime_dependency 'attr_bool'            ,'~> 0.1'  # For attr_accessor?/attr_reader?
   spec.add_runtime_dependency 'bimyou_segmenter'     ,'~> 1.2'  # For splitting Japanese sentences into words
   spec.add_runtime_dependency 'cri'                  ,'~> 2.15' # For CLI commands/options
   spec.add_runtime_dependency 'down'                 ,'~> 5.1'  # For downloading files (GetCmd)

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: nhkore
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.3.3
 platform: ruby
 authors:
 - Jonathan Bradley Whited (@esotericpig)
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-04-21 00:00:00.000000000 Z
+date: 2020-04-23 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: attr_bool
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
 - !ruby/object:Gem::Dependency
   name: bimyou_segmenter
   requirement: !ruby/object:Gem::Requirement
@@ -375,7 +389,7 @@ metadata:
   changelog_uri: https://github.com/esotericpig/nhkore/blob/master/CHANGELOG.md
   homepage_uri: https://github.com/esotericpig/nhkore
   source_code_uri: https://github.com/esotericpig/nhkore
-post_install_message: "  \n  NHKore v0.3.2\n  \n  You can now use [nhkore] on the
+post_install_message: "  \n  NHKore v0.3.3\n  \n  You can now use [nhkore] on the
   command line.\n  \n  Homepage:  https://github.com/esotericpig/nhkore\n  \n  Code:
   \     https://github.com/esotericpig/nhkore\n  Changelog: https://github.com/esotericpig/nhkore/blob/master/CHANGELOG.md\n
   \ Bugs:      https://github.com/esotericpig/nhkore/issues\n  \n"