yasuri 2.0.11 → 3.2.0

data/USAGE.md

@@ -1,27 +1,31 @@
-# Yasuri Usage
+# Yasuri
 
 ## What is Yasuri
-`Yasuri` is an easy web-scraping library for supporting "Mechanize".
+`Yasuri` (鑢) is a library for declarative web scraping and a command line tool for scraping with it. It performs scraping using "[Mechanize](https://github.com/sparklemotion/mechanize)" by simply describing the expected result in a simple declarative notation.
 
-Yasuri (鑢) is an easy web-scraping library for supporting "[Mechanize](https://github.com/sparklemotion/mechanize)".
+Yasuri makes it easy to write common scraping operations.
+For example, the following processes can be easily implemented.
 
-Yasuri can reduce frequently processes in Scraping.
++ Scrape multiple texts in a page and name them into a Hash
++ Open multiple links in a page and get the result of scraping each page as a Hash
++ Scrape each table that appears repeatedly in the page and get the result as an array
++ Scrape only the first three pages of each page provided by pagination
 
-For example,
-
-+ Open links in the page, scraping each page, and getting result as Hash.
-+ Scraping texts in the page, and named result in Hash.
-+ A table that repeatedly appears in a page each, scraping, get as an array.
-+ Of each page provided by the pagination, scraping the only top 3.
-
-You can implement easy by Yasuri.
 
 ## Quick Start
 
+
+#### Install
+```sh
+# for Ruby 2.3.2
+$ gem 'yasuri', '~> 2.0', '>= 2.0.13'
 ```
+または
+```sh
+# for Ruby 3.0.0 or upper
 $ gem install yasuri
 ```
-
+#### Use as library
 ```ruby
 require 'yasuri'
 require 'machinize'
@@ -33,80 +37,148 @@ root = Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
        end
 
 agent = Mechanize.new
-root_page = agent.get("http://some.scraping.page.net/")
+root_page = agent.get("http://some.scraping.page.tac42.net/")
 
 result = root.inject(agent, root_page)
-# => [ {"title" => "PageTitle1", "content" => "Page Contents1" },
-#      {"title" => "PageTitle2", "content" => "Page Contents2" }, ...  ]
-
+# => [
+#      {"title" => "PageTitle 01", "content" => "Page Contents  01" },
+#      {"title" => "PageTitle 02", "content" => "Page Contents  02" },
+#      ...
+#      {"title" => "PageTitle N",  "content" => "Page Contents  N" }
+#    ]
 ```
+
 This example, from the pages of each link that is expressed by the xpath of LinkNode(`links_root`), to scraping the two text that is expressed by the xpath of TextNode(`text_title`,`text_content`).
 
-(i.e. open each links `//*[@id="menu"]/ul/li/a` and, scrape `//*[@id="contents"]/h2` and `//*[@id="contents"]/p[1]`.)
+(in other words, open each links `//*[@id="menu"]/ul/li/a` and, scrape `//*[@id="contents"]/h2` and `//*[@id="contents"]/p[1]`.)
 
-## Basics
 
-1. Construct parse tree.
-2. Start parse with Mechanize agent and first page.
+#### Use as CLI tool
+The same thing as above can be executed as a CLI command.
 
-### Construct parse tree
+```sh
+$ yasuri scrape "http://some.scraping.page.tac42.net/" -j '
+{
+  "links_root": {
+    "path": "//*[@id=\"menu\"]/ul/li/a",
+    "text_title": "//*[@id=\"contents\"]/h2",
+    "text_content": "//*[@id=\"contents\"]/p[1]"
+    }
+}'
 
-```ruby
-require 'mechanize'
-require 'yasuri'
+[
+  {"title":"PageTitle 01","content":"Page Contents  01"},
+  {"title":"PageTitle 02","content":"Page Contents  02"},
+  ...,
+  {"title":"PageTitle N","content":"Page Contents  N"}
+]
+```
 
+The result can be obtained as a string in json format.
 
-# 1. Construct parse tree.
-tree = Yasuri.links_title '/html/body/a' do
-         text_name '/html/body/p'
-       end
+----------------------------
+## Parse Tree
 
-# 2. Start parse with Mechanize agent and first page.
-agent = Mechanize.new
-page = agent.get(uri)
+A parse tree is a tree structure data for declaratively defining the elements to be scraped and the output structure.
 
+A parse tree consists of nested `Node`s, each of which has `Type`, `Name`, `Path`, `Childlen`, and `Options` attributes, and scrapes according to its `Type`. (Note that only `MapNode` does not have `Path`).
 
-tree.inject(agent, page)
+The parse tree is defined in the following format:
+
+```ruby
+# A simple tree consisting of one node
+Yasuri.<Type>_<Name> <Path> [,<Options>]
+
+# Nested tree
+Yasuri.<Type>_<Name> <Path> [,<Options>] do
+  <Type>_<Name> <Path> [,<Options>] do
+    <Type>_<Name> <Path> [,<Options>]
+    ...
+  end
+end
 ```
 
-Tree is definable by 2(+1) ways, DSL and json (and basic ruby code). In above example, DSL.
+**Example**
 
 ```ruby
-# Construct by json.
-src = <<-EOJSON
-   { "node"     : "links",
-     "name"     : "title",
-     "path"     : "/html/body/a",
-     "children" : [
-                    { "node" : "text",
-                      "name" : "name",
-                      "path" : "/html/body/p"
-                    }
-                  ]
-   }
-EOJSON
-tree = Yasuri.json2tree(src)
-```
+# A simple tree consisting of one node
+Yasuri.text_title '/html/head/title', truncate:/^[^,]+/
 
-### Node
-Tree is constructed by nested Nodes.
-Node has `Type`, `Name`, `Path`, `Childlen`, and `Options`.
+# Nested tree
+Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
+  struct_table './tr' do
+    text_title    './td[1]'
+    text_pub_date './td[2]'
+  end
+end
+```
 
-Node is defined by this format.
+Parsing trees can be defined in Ruby DSL, JSON, or YAML.
+The following is an example of the same parse tree as above, defined in each notation.
 
 
+**Case of defining as Ruby DSL**
 ```ruby
-# Top Level
-Yasuri.<Type>_<Name> <Path> [,<Options>]
-
-# Nested
-Yasuri.<Type>_<Name> <Path> [,<Options>] do
-  <Type>_<Name> <Path> [,<Options>] do
-    <Children>
-  end
+Yasuri.links_title '/html/body/a' do
+  text_name '/html/body/p'
 end
 ```
 
+**Case of defining as JSON**
+```json
+{
+  links_title": {
+    "path": "/html/body/a",
+    "text_name": "/html/body/p"
+  }
+}
+```
+
+**Case of defining as YAML**
+```yaml
+links_title:
+  path: "/html/body/a"
+  text_name: "/html/body/p"
+```
+
+**Special case of purse tree**
+
+If there is only one element directly under the root, it will return that element directly instead of Hash(Object).
+```json
+{
+  "text_title": "/html/head/title",
+  "text_body": "/html/body",
+}
+# => {"title": "Welcome to yasuri!", "body": "Yasuri is ..."}
+
+{
+  "text_title": "/html/head/title"}
+}
+# => Welcome to yasuri!
+```
+
+
+In json or yaml format, a attribute can directly specify `path` as a value if it doesn't have any child Node. The following two json will have the same parse tree.
+
+```json
+{
+  "text_name": "/html/body/p"
+}
+
+{
+  "text_name": {
+    "path": "/html/body/p"
+  }
+}
+```
+
+
+--------------------------
+## Node
+
+Node is a node or leaf of the parse tree, which has `Type`, `Name`, `Path`, `Childlen`, and `Options`, and scrapes according to its `Type`. (Note that only `MapNode` does not have `Path`).
+
+
 #### Type
 Type meen behavior of Node.
 
@@ -114,17 +186,20 @@ Type meen behavior of Node.
 - *Struct*
 - *Links*
 - *Paginate*
+- *Map*
+
+See the description of each node for details.
 
-### Name
+#### Name
 Name is used keys in returned hash.
 
-### Path
+#### Path
 Path determine target node by xpath or css selector. It given by Machinize `search`.
 
-### Childlen
+#### Childlen
 Child nodes. TextNode has always empty set, because TextNode is leaf.
 
-### Options
+#### Options
 Parse options. It different in each types. You can get options and values by `opt` method.
 
 ```ruby
@@ -136,6 +211,8 @@ node.opt #=> {:truncate => /^[^,]+/, :proc => nil}
 ## Text Node
 TextNode return scraped text. This node have to be leaf.
 
+
+
 ### Example
 
 ```html
@@ -155,13 +232,15 @@ page = agent.get("http://yasuri.example.net")
 
 p1  = Yasuri.text_title '/html/body/p[1]'
 p1t = Yasuri.text_title '/html/body/p[1]', truncate:/^[^,]+/
-p2u = Yasuri.text_title '/html/body/p[2]', proc: :upcase
+p2u = Yasuri.text_title '/html/body/p[1]', proc: :upcase
 
-p1.inject(agent, page)   #=> { "title" => "Hello,World" }
-p1t.inject(agent, page)  #=> { "title" => "Hello" }
-node.inject(agent, page) #=> { "title" => "HELLO,YASURI" }
+p1.inject(agent, page)   #=> "Hello,World"
+p1t.inject(agent, page)  #=> "Hello"
+p2u.inject(agent, page)  #=> "HELLO,WORLD"
 ```
 
+Note that if you want to scrape multiple elements in the same page at once, use `MapNode`. See the `MapNode` example for details.
+
 ### Options
 ##### `truncate`
 Match to regexp, and truncate text. When you use group, it will return first matched group only.
@@ -429,3 +508,190 @@ node.inject(agent, page)
 #=> [ {"content" => "Pagination01"}, {"content" => "Pagination02"}]
 ```
 Paginate Node open upto 2 given by `limit`. In this situation, pagination has 4 pages, but result Array has 2 texts because given `limit:2`.
+
+##### `flatten`
+`flatten` option expands each page results.
+
+```ruby
+agent = Mechanize.new
+page = agent.get("http://yasuri.example.net/page01.html")
+
+node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" , flatten:true do
+         text_title   '/html/head/title'
+         text_content '/html/body/p'
+       end
+node.inject(agent, page)
+
+#=> [ {"title" => "Page01",
+       "content" => "Patination01"},
+      {"title"   => "Page01",
+       "content" => "Patination02"},
+      {"title"   => "Page01",
+       "content" => "Patination03"}]
+
+
+node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" , flatten:true do
+        text_title   '/html/head/title'
+        text_content '/html/body/p'
+      end
+node.inject(agent, page)
+
+#=> [ "Page01",
+      "Patination01",
+      "Page02",
+      "Patination02",
+      "Page03",
+      "Patination03"]
+```
+
+## Map Node
+*MapNode* is a node that summarizes the results of scraping. This node is always a branch node in the parse tree.
+
+### Example
+
+```html
+<!-- http://yasuri.example.net -->
+<html>
+  <head><title>Yasuri Example</title></head>
+  <body>
+    <p>Hello,World</p>
+    <p>Hello,Yasuri</p>
+  </body>
+</html>
+```
+
+```ruby
+agent = Mechanize.new
+page = agent.get("http://yasuri.example.net")
+
+
+tree = Yasuri.map_root do
+  text_title  '/html/head/title'
+  text_body_p '/html/body/p[1]'
+end
+
+tree.inject(agent, page) #=> { "title" => "Yasuri Example", "body_p" => "Hello,World" }
+
+
+tree = Yasuri.map_root do
+  map_group1 { text_child01  '/html/body/a[1]' }
+  map_group2 do
+    text_child01 '/html/body/a[1]'
+    text_child03 '/html/body/a[3]'
+  end
+end
+
+tree.inject(agent, page) #=> {
+#   "group1" => {
+#           "child01" => "child01"
+#         },
+#         "group2" => {
+#           "child01" => "child01",
+#           "child03" => "child03"
+#         }
+# }
+```
+
+### Options
+None.
+
+
+
+
+-------------------------
+## Usage
+
+#### Use as library
+When used as a library, the tree can be defined in DSL, json, or yaml format.
+```ruby
+require 'mechanize'
+require 'yasuri'
+
+
+# 1. Create a parse tree.
+# Define by Ruby's DSL
+tree = Yasuri.links_title '/html/body/a' do
+         text_name '/html/body/p'
+       end
+
+# Define by JSON
+src = <<-EOJSON
+{
+  links_title": {
+    "path": "/html/body/a",
+    "text_name": "/html/body/p"
+  }
+}
+EOJSON
+tree = Yasuri.json2tree(src)
+
+
+# Define by YAML
+src = <<-EOYAML
+links_title:
+  path: "/html/body/a"
+  text_name: "/html/body/p"
+EOYAML
+tree = Yasuri.yaml2tree(src)
+
+
+
+# 2. Give the Mechanize agent and the target page to start parsing
+agent = Mechanize.new
+page = agent.get(uri)
+
+
+tree.inject(agent, page)
+```
+
+#### Use as CLI tool
+
+**Help**
+```sh
+$ yasuri help scrape
+Usage:
+  yasuri scrape <URI> [[--file <TREE_FILE>] or [--json <JSON>]]
+
+Options:
+  f, [--file=FILE]  # path to file that written yasuri tree as json or yaml
+  j, [--json=JSON]  # yasuri tree format json string
+
+Getting from <URI> and scrape it. with <JSON> or json/yml from <TREE_FILE>. They should be Yasuri's format json or yaml string.
+```
+
+In the CLI tool, you can specify the parse tree in either of the following ways.
++ `--file`, `-f`  option to read the parse tree in json or yaml format output to a file.
++ `--json`, `-j`  option to specify the parse tree directly as a string.
+
+
+**Example of specifying a parse tree as a file**
+```sh
+% cat sample.yml
+text_title: "/html/head/title"
+text_desc: "//*[@id=\"intro\"]/p"
+
+% yasuri scrape "https://www.ruby-lang.org/en/" --file sample.yml
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+
+% cat sample.json
+{
+  "text_title": "/html/head/title",
+  "text_desc": "//*[@id=\"intro\"]/p"
+}
+
+% yasuri scrape "https://www.ruby-lang.org/en/" --file sample.json
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+```
+
+Whether the file is written in json or yaml will be determined automatically.
+
+**Example of specifying a parse tree directly in json**
+```sh
+$ yasuri scrape "https://www.ruby-lang.org/en/" -j '
+{
+  "text_title": "/html/head/title",
+  "text_desc": "//*[@id=\"intro\"]/p"
+}'
+
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+```