yasuri 3.1.0 → 3.2.0

data/USAGE.md

@@ -1,27 +1,31 @@
-# Yasuri Usage
+# Yasuri
 
 ## What is Yasuri
-`Yasuri` is an easy web-scraping library for supporting "Mechanize".
+`Yasuri` (鑢) is a library for declarative web scraping and a command line tool for scraping with it. It performs scraping using "[Mechanize](https://github.com/sparklemotion/mechanize)" by simply describing the expected result in a simple declarative notation.
 
-Yasuri (鑢) is an easy web-scraping library for supporting "[Mechanize](https://github.com/sparklemotion/mechanize)".
+Yasuri makes it easy to write common scraping operations.
+For example, the following processes can be easily implemented.
 
-Yasuri can reduce frequently processes in Scraping.
++ Scrape multiple texts in a page and name them into a Hash
++ Open multiple links in a page and get the result of scraping each page as a Hash
++ Scrape each table that appears repeatedly in the page and get the result as an array
++ Scrape only the first three pages of each page provided by pagination
 
-For example,
-
-+ Open links in the page, scraping each page, and getting result as Hash.
-+ Scraping texts in the page, and named result in Hash.
-+ A table that repeatedly appears in a page each, scraping, get as an array.
-+ Of each page provided by the pagination, scraping the only top 3.
-
-You can implement easy by Yasuri.
 
 ## Quick Start
 
+
+#### Install
+```sh
+# for Ruby 2.3.2
+$ gem 'yasuri', '~> 2.0', '>= 2.0.13'
 ```
+または
+```sh
+# for Ruby 3.0.0 or upper
 $ gem install yasuri
 ```
-
+#### Use as library
 ```ruby
 require 'yasuri'
 require 'machinize'
@@ -33,88 +37,59 @@ root = Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
        end
 
 agent = Mechanize.new
-root_page = agent.get("http://some.scraping.page.net/")
+root_page = agent.get("http://some.scraping.page.tac42.net/")
 
 result = root.inject(agent, root_page)
-# => [ {"title" => "PageTitle1", "content" => "Page Contents1" },
-#      {"title" => "PageTitle2", "content" => "Page Contents2" }, ...  ]
-
+# => [
+#      {"title" => "PageTitle 01", "content" => "Page Contents  01" },
+#      {"title" => "PageTitle 02", "content" => "Page Contents  02" },
+#      ...
+#      {"title" => "PageTitle N",  "content" => "Page Contents  N" }
+#    ]
 ```
-This example, from the pages of each link that is expressed by the xpath of LinkNode(`links_root`), to scraping the two text that is expressed by the xpath of TextNode(`text_title`,`text_content`).
-
-(i.e. open each links `//*[@id="menu"]/ul/li/a` and, scrape `//*[@id="contents"]/h2` and `//*[@id="contents"]/p[1]`.)
-
-## Basics
-
-1. Construct parse tree.
-2. Start parse with Mechanize agent and first page.
-
-### Construct parse tree
-
-```ruby
-require 'mechanize'
-require 'yasuri'
 
+This example, from the pages of each link that is expressed by the xpath of LinkNode(`links_root`), to scraping the two text that is expressed by the xpath of TextNode(`text_title`,`text_content`).
 
-# 1. Construct parse tree.
-tree = Yasuri.links_title '/html/body/a' do
-         text_name '/html/body/p'
-       end
+(in other words, open each links `//*[@id="menu"]/ul/li/a` and, scrape `//*[@id="contents"]/h2` and `//*[@id="contents"]/p[1]`.)
 
-# 2. Start parse with Mechanize agent and first page.
-agent = Mechanize.new
-page = agent.get(uri)
 
+#### Use as CLI tool
+The same thing as above can be executed as a CLI command.
 
-tree.inject(agent, page)
-```
-
-Tree is definable by 3(+1) ways, json, yaml, and DSL (or basic ruby code). In above example, DSL.
+```sh
+$ yasuri scrape "http://some.scraping.page.tac42.net/" -j '
+{
+  "links_root": {
+    "path": "//*[@id=\"menu\"]/ul/li/a",
+    "text_title": "//*[@id=\"contents\"]/h2",
+    "text_content": "//*[@id=\"contents\"]/p[1]"
+    }
+}'
 
-```ruby
-# Construct by json.
-src = <<-EOJSON
-   { "node"     : "links",
-     "name"     : "title",
-     "path"     : "/html/body/a",
-     "children" : [
-                    { "node" : "text",
-                      "name" : "name",
-                      "path" : "/html/body/p"
-                    }
-                  ]
-   }
-EOJSON
-tree = Yasuri.json2tree(src)
+[
+  {"title":"PageTitle 01","content":"Page Contents  01"},
+  {"title":"PageTitle 02","content":"Page Contents  02"},
+  ...,
+  {"title":"PageTitle N","content":"Page Contents  N"}
+]
 ```
 
-```ruby
-# Construct by yaml.
-src = <<-EOYAML
-title:
-  node: links
-  path: "/html/body/a"
-  children:
-    - name:
-        node: text
-        path: "/html/body/p"
-EOYAML
-tree = Yasuri.yaml2tree(src)
-```
+The result can be obtained as a string in json format.
 
+----------------------------
+## Parse Tree
 
-### Node
-Tree is constructed by nested Nodes.
-Node has `Type`, `Name`, `Path`, `Childlen`, and `Options`.
-(But only `MapNode` does not have `Path`.)
+A parse tree is a tree structure data for declaratively defining the elements to be scraped and the output structure.
 
-Node is defined by this format.
+A parse tree consists of nested `Node`s, each of which has `Type`, `Name`, `Path`, `Childlen`, and `Options` attributes, and scrapes according to its `Type`. (Note that only `MapNode` does not have `Path`).
 
+The parse tree is defined in the following format:
 
 ```ruby
+# A simple tree consisting of one node
 Yasuri.<Type>_<Name> <Path> [,<Options>]
 
-# Nested case
+# Nested tree
 Yasuri.<Type>_<Name> <Path> [,<Options>] do
   <Type>_<Name> <Path> [,<Options>] do
     <Type>_<Name> <Path> [,<Options>]
@@ -123,12 +98,13 @@ Yasuri.<Type>_<Name> <Path> [,<Options>] do
 end
 ```
 
-Example
+**Example**
 
 ```ruby
+# A simple tree consisting of one node
 Yasuri.text_title '/html/head/title', truncate:/^[^,]+/
 
-# Nested case
+# Nested tree
 Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
   struct_table './tr' do
     text_title    './td[1]'
@@ -137,6 +113,72 @@ Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
 end
 ```
 
+Parsing trees can be defined in Ruby DSL, JSON, or YAML.
+The following is an example of the same parse tree as above, defined in each notation.
+
+
+**Case of defining as Ruby DSL**
+```ruby
+Yasuri.links_title '/html/body/a' do
+  text_name '/html/body/p'
+end
+```
+
+**Case of defining as JSON**
+```json
+{
+  links_title": {
+    "path": "/html/body/a",
+    "text_name": "/html/body/p"
+  }
+}
+```
+
+**Case of defining as YAML**
+```yaml
+links_title:
+  path: "/html/body/a"
+  text_name: "/html/body/p"
+```
+
+**Special case of purse tree**
+
+If there is only one element directly under the root, it will return that element directly instead of Hash(Object).
+```json
+{
+  "text_title": "/html/head/title",
+  "text_body": "/html/body",
+}
+# => {"title": "Welcome to yasuri!", "body": "Yasuri is ..."}
+
+{
+  "text_title": "/html/head/title"}
+}
+# => Welcome to yasuri!
+```
+
+
+In json or yaml format, a attribute can directly specify `path` as a value if it doesn't have any child Node. The following two json will have the same parse tree.
+
+```json
+{
+  "text_name": "/html/body/p"
+}
+
+{
+  "text_name": {
+    "path": "/html/body/p"
+  }
+}
+```
+
+
+--------------------------
+## Node
+
+Node is a node or leaf of the parse tree, which has `Type`, `Name`, `Path`, `Childlen`, and `Options`, and scrapes according to its `Type`. (Note that only `MapNode` does not have `Path`).
+
+
 #### Type
 Type meen behavior of Node.
 
@@ -146,6 +188,8 @@ Type meen behavior of Node.
 - *Paginate*
 - *Map*
 
+See the description of each node for details.
+
 #### Name
 Name is used keys in returned hash.
 
@@ -167,6 +211,8 @@ node.opt #=> {:truncate => /^[^,]+/, :proc => nil}
 ## Text Node
 TextNode return scraped text. This node have to be leaf.
 
+
+
 ### Example
 
 ```html
@@ -548,3 +594,104 @@ tree.inject(agent, page) #=> {
 
 ### Options
 None.
+
+
+
+
+-------------------------
+## Usage
+
+#### Use as library
+When used as a library, the tree can be defined in DSL, json, or yaml format.
+```ruby
+require 'mechanize'
+require 'yasuri'
+
+
+# 1. Create a parse tree.
+# Define by Ruby's DSL
+tree = Yasuri.links_title '/html/body/a' do
+         text_name '/html/body/p'
+       end
+
+# Define by JSON
+src = <<-EOJSON
+{
+  links_title": {
+    "path": "/html/body/a",
+    "text_name": "/html/body/p"
+  }
+}
+EOJSON
+tree = Yasuri.json2tree(src)
+
+
+# Define by YAML
+src = <<-EOYAML
+links_title:
+  path: "/html/body/a"
+  text_name: "/html/body/p"
+EOYAML
+tree = Yasuri.yaml2tree(src)
+
+
+
+# 2. Give the Mechanize agent and the target page to start parsing
+agent = Mechanize.new
+page = agent.get(uri)
+
+
+tree.inject(agent, page)
+```
+
+#### Use as CLI tool
+
+**Help**
+```sh
+$ yasuri help scrape
+Usage:
+  yasuri scrape <URI> [[--file <TREE_FILE>] or [--json <JSON>]]
+
+Options:
+  f, [--file=FILE]  # path to file that written yasuri tree as json or yaml
+  j, [--json=JSON]  # yasuri tree format json string
+
+Getting from <URI> and scrape it. with <JSON> or json/yml from <TREE_FILE>. They should be Yasuri's format json or yaml string.
+```
+
+In the CLI tool, you can specify the parse tree in either of the following ways.
++ `--file`, `-f`  option to read the parse tree in json or yaml format output to a file.
++ `--json`, `-j`  option to specify the parse tree directly as a string.
+
+
+**Example of specifying a parse tree as a file**
+```sh
+% cat sample.yml
+text_title: "/html/head/title"
+text_desc: "//*[@id=\"intro\"]/p"
+
+% yasuri scrape "https://www.ruby-lang.org/en/" --file sample.yml
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+
+% cat sample.json
+{
+  "text_title": "/html/head/title",
+  "text_desc": "//*[@id=\"intro\"]/p"
+}
+
+% yasuri scrape "https://www.ruby-lang.org/en/" --file sample.json
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+```
+
+Whether the file is written in json or yaml will be determined automatically.
+
+**Example of specifying a parse tree directly in json**
+```sh
+$ yasuri scrape "https://www.ruby-lang.org/en/" -j '
+{
+  "text_title": "/html/head/title",
+  "text_desc": "//*[@id=\"intro\"]/p"
+}'
+
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+```

data/exe/yasuri

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "yasuri"
+
+Yasuri::CLI.start

data/lib/yasuri.rb

@@ -1,5 +1,6 @@
 require "yasuri/version"
 require "yasuri/yasuri"
+require "yasuri/yasuri_cli"
 
 module Yasuri
   # Your code goes here...

data/lib/yasuri/version.rb

@@ -1,3 +1,3 @@
 module Yasuri
-  VERSION = "3.1.0"
+  VERSION = "3.2.0"
 end

data/lib/yasuri/yasuri.rb

@@ -16,45 +16,29 @@ require_relative 'yasuri_node_generator'
 
 module Yasuri
 
+  DefaultRetryCount = 5
+
   def self.json2tree(json_string)
-    json = JSON.parse(json_string, {symbolize_names: true})
-    Yasuri.hash2node(json)
+    raise RuntimeError if json_string.nil? or json_string.empty?
+
+    node_hash = JSON.parse(json_string, {symbolize_names: true})
+    Yasuri.hash2node(node_hash)
   end
 
   def self.tree2json(node)
+    raise RuntimeError if node.nil?
+
     Yasuri.node2hash(node).to_json
   end
 
   def self.yaml2tree(yaml_string)
     raise RuntimeError if yaml_string.nil? or yaml_string.empty?
 
-    yaml = YAML.load(yaml_string)
-    raise RuntimeError if yaml.keys.size < 1
-
-    root_key, root = yaml.keys.first, yaml.values.first
-    hash = Yasuri.yaml2tree_sub(root_key, root)
-
-    Yasuri.hash2node(hash)
+    node_hash = YAML.load(yaml_string)
+    Yasuri.hash2node(node_hash.deep_symbolize_keys)
   end
 
   private
-  def self.yaml2tree_sub(name, body)
-    return nil if name.nil? or body.nil?
-
-    new_body = Hash[:name, name]
-    body.each{|k,v| new_body[k.to_sym] = v}
-    body = new_body
-
-    return body if body[:children].nil?
-
-    body[:children] = body[:children].map do |c|
-      k, b = c.keys.first, c.values.first
-      Yasuri.yaml2tree_sub(k, b)
-    end
-
-    body
-  end
-
   def self.method_missing(method_name, pattern=nil, **opt, &block)
     generated = Yasuri::NodeGenerator.gen(method_name, pattern, **opt, &block)
     generated || super(method_name, **opt)
@@ -66,24 +50,64 @@ module Yasuri
     struct: Yasuri::StructNode,
     links:  Yasuri::LinksNode,
     pages:  Yasuri::PaginateNode,
-    map:   Yasuri::MapNode
+    map:    Yasuri::MapNode
   }
-  Node2Text = Text2Node.invert
 
-  ReservedKeys = %i|node name path children|
-  def self.hash2node(node_h)
-    node = node_h[:node]
+  def self.hash2node(node_hash, node_name = nil, node_type_class = nil)
+    raise RuntimeError.new("") if node_name.nil? and node_hash.empty?
+
+    node_prefixes = Text2Node.keys.freeze
+    child_nodes = []
+    opt = {}
+    path = nil
+
+    if node_hash.is_a?(String)
+      path = node_hash
+    else
+      node_hash.each do |key, value|
+        # is node?
+        node_regexps = Text2Node.keys.map do |node_type_sym|
+          /^(#{node_type_sym.to_s})_(.+)$/
+        end
+        node_regexp = node_regexps.find do |node_regexp|
+          key =~ node_regexp
+        end
+
+        case key
+        when node_regexp
+          node_type_sym = $1.to_sym
+          child_node_name = $2
+          child_node_type = Text2Node[node_type_sym]
+          child_nodes << self.hash2node(value, child_node_name, child_node_type)
+        when :path
+          path = value
+        else
+          opt[key] = value
+        end
+      end
+    end
+
+    # If only single node under root, return only the node.
+    return child_nodes.first if node_name.nil? and child_nodes.size == 1
+
+    node = if node_type_class.nil?
+      Yasuri::MapNode.new(node_name, child_nodes, **opt)
+    else
+      node_type_class::new(path, node_name, child_nodes, **opt)
+    end
 
-    fail "Not found 'node' value in map" if node.nil?
-    klass = Text2Node[node.to_sym]
-    klass::hash2node(node_h)
+    node
   end
 
   def self.node2hash(node)
-    node.to_h
+    return node.to_h if node.instance_of?(Yasuri::MapNode)
+
+    {
+      "#{node.node_type_str}_#{node.name}" => node.to_h
+    }
   end
 
-  def self.NodeName(name, opt)
+  def self.node_name(name, opt)
     symbolize_names = opt[:symbolize_names]
     symbolize_names ? name.to_sym : name
   end
@@ -101,3 +125,14 @@ module Yasuri
     end
   end
 end
+
+class Hash
+  def deep_symbolize_keys
+    Hash[
+      self.map do |k, v|
+        v = v.deep_symbolize_keys if v.kind_of?(Hash)
+        [k.to_sym, v]
+      end
+    ]
+  end
+end