yasuri 2.0.12 → 3.3.0

data/USAGE.md

@@ -1,27 +1,32 @@
-# Yasuri Usage
+# Yasuri
 
 ## What is Yasuri
-`Yasuri` is an easy web-scraping library for supporting "Mechanize".
+`Yasuri` (鑢) is a library for declarative web scraping and a command line tool for scraping with it.
 
-Yasuri (鑢) is an easy web-scraping library for supporting "[Mechanize](https://github.com/sparklemotion/mechanize)".
+It performs scraping by simply describing the expected result in a simple declarative notation.
 
-Yasuri can reduce frequently processes in Scraping.
+Yasuri makes it easy to write common scraping operations.
+For example, the following processes can be easily implemented.
 
-For example,
-
-+ Open links in the page, scraping each page, and getting result as Hash.
-+ Scraping texts in the page, and named result in Hash.
-+ A table that repeatedly appears in a page each, scraping, get as an array.
-+ Of each page provided by the pagination, scraping the only top 3.
-
-You can implement easy by Yasuri.
++ Scrape multiple texts in a page and name them into a Hash
++ Open multiple links in a page and get the result of scraping each page as a Hash
++ Scrape each table that appears repeatedly in the page and get the result as an array
++ Scrape only the first three pages of each page provided by pagination
 
 ## Quick Start
 
+
+#### Install
+```sh
+# for Ruby 2.3.2
+$ gem 'yasuri', '~> 2.0', '>= 2.0.13'
 ```
+または
+```sh
+# for Ruby 3.0.0 or upper
 $ gem install yasuri
 ```
-
+#### Use as library
 ```ruby
 require 'yasuri'
 require 'machinize'
@@ -32,81 +37,190 @@ root = Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
          text_content '//*[@id="contents"]/p[1]'
        end
 
-agent = Mechanize.new
-root_page = agent.get("http://some.scraping.page.net/")
+result = root.scrape("http://some.scraping.page.tac42.net/")
+# => [
+#      {"title" => "PageTitle 01", "content" => "Page Contents  01" },
+#      {"title" => "PageTitle 02", "content" => "Page Contents  02" },
+#      ...
+#      {"title" => "PageTitle N",  "content" => "Page Contents  N" }
+#    ]
+```
+
+This example, from the pages of each link that is expressed by the xpath of LinkNode(`links_root`), to scraping the two text that is expressed by the xpath of TextNode(`text_title`,`text_content`).
+
+(in other words, open each links `//*[@id="menu"]/ul/li/a` and, scrape `//*[@id="contents"]/h2` and `//*[@id="contents"]/p[1]`.)
+
 
-result = root.inject(agent, root_page)
-# => [ {"title" => "PageTitle1", "content" => "Page Contents1" },
-#      {"title" => "PageTitle2", "content" => "Page Contents2" }, ...  ]
+#### Use as CLI tool
+The same thing as above can be executed as a CLI command.
 
+```sh
+$ yasuri scrape "http://some.scraping.page.tac42.net/" -j '
+{
+  "links_root": {
+    "path": "//*[@id=\"menu\"]/ul/li/a",
+    "text_title": "//*[@id=\"contents\"]/h2",
+    "text_content": "//*[@id=\"contents\"]/p[1]"
+    }
+}'
+
+[
+  {"title":"PageTitle 01","content":"Page Contents  01"},
+  {"title":"PageTitle 02","content":"Page Contents  02"},
+  ...,
+  {"title":"PageTitle N","content":"Page Contents  N"}
+]
 ```
-This example, from the pages of each link that is expressed by the xpath of LinkNode(`links_root`), to scraping the two text that is expressed by the xpath of TextNode(`text_title`,`text_content`).
 
-(i.e. open each links `//*[@id="menu"]/ul/li/a` and, scrape `//*[@id="contents"]/h2` and `//*[@id="contents"]/p[1]`.)
+The result can be obtained as a string in json format.
 
-## Basics
+----------------------------
+## Parse Tree
 
-1. Construct parse tree.
-2. Start parse with Mechanize agent and first page.
+A parse tree is a tree structure data for declaratively defining the elements to be scraped and the output structure.
 
-### Construct parse tree
+A parse tree consists of nested `Node`s, each of which has `Type`, `Name`, `Path`, `Childlen`, and `Options` attributes, and scrapes according to its `Type`. (Note that only `MapNode` does not have `Path`).
 
-```ruby
-require 'mechanize'
-require 'yasuri'
+The parse tree is defined in the following format:
 
+```ruby
+# A simple tree consisting of one node
+Yasuri.<Type>_<Name> <Path> [,<Options>]
 
-# 1. Construct parse tree.
-tree = Yasuri.links_title '/html/body/a' do
-         text_name '/html/body/p'
-       end
+# Nested tree
+Yasuri.<Type>_<Name> <Path> [,<Options>] do
+  <Type>_<Name> <Path> [,<Options>] do
+    <Type>_<Name> <Path> [,<Options>]
+    ...
+  end
+end
+```
 
-# 2. Start parse with Mechanize agent and first page.
-agent = Mechanize.new
-page = agent.get(uri)
+**Example**
 
+```ruby
+# A simple tree consisting of one node
+Yasuri.text_title '/html/head/title', truncate:/^[^,]+/
 
-tree.inject(agent, page)
+# Nested tree
+Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
+  struct_table './tr' do
+    text_title    './td[1]'
+    text_pub_date './td[2]'
+  end
+end
 ```
 
-Tree is definable by 2(+1) ways, DSL and json (and basic ruby code). In above example, DSL.
+Parsing trees can be defined in Ruby DSL, JSON, or YAML.
+The following is an example of the same parse tree as above, defined in each notation.
+
 
+**Case of defining as Ruby DSL**
 ```ruby
-# Construct by json.
-src = <<-EOJSON
-   { "node"     : "links",
-     "name"     : "title",
-     "path"     : "/html/body/a",
-     "children" : [
-                    { "node" : "text",
-                      "name" : "name",
-                      "path" : "/html/body/p"
-                    }
-                  ]
-   }
-EOJSON
-tree = Yasuri.json2tree(src)
+Yasuri.links_title '/html/body/a' do
+  text_name '/html/body/p'
+end
 ```
 
-### Node
-Tree is constructed by nested Nodes.
-Node has `Type`, `Name`, `Path`, `Childlen`, and `Options`.
+**Case of defining as JSON**
+```json
+{
+  links_title": {
+    "path": "/html/body/a",
+    "text_name": "/html/body/p"
+  }
+}
+```
 
-Node is defined by this format.
+**Case of defining as YAML**
+```yaml
+links_title:
+  path: "/html/body/a"
+  text_name: "/html/body/p"
+```
 
+**Special case of purse tree**
 
+If there is only one element directly under the root, it will return that element directly instead of Hash(Object).
+```json
+{
+  "text_title": "/html/head/title",
+  "text_body": "/html/body",
+}
+# => {"title": "Welcome to yasuri!", "body": "Yasuri is ..."}
+
+{
+  "text_title": "/html/head/title"}
+}
+# => Welcome to yasuri!
+```
+
+
+In json or yaml format, a attribute can directly specify `path` as a value if it doesn't have any child Node. The following two json will have the same parse tree.
+
+```json
+{
+  "text_name": "/html/body/p"
+}
+
+{
+  "text_name": {
+    "path": "/html/body/p"
+  }
+}
+```
+### Run ParseTree
+Call the `Node#scrape(uri, opt={})` method on the root node of the parse tree.
+
+**Example**
 ```ruby
-# Top Level
-Yasuri.<Type>_<Name> <Path> [,<Options>]
+root = Yasuri.links_root '//*[@id="menu"]/ul/li/a' do
+         text_title '//*[@id="contents"]/h2'
+         text_content '//*[@id="contents"]/p[1]'
+       end
 
-# Nested
-Yasuri.<Type>_<Name> <Path> [,<Options>] do
-  <Type>_<Name> <Path> [,<Options>] do
-    <Children>
-  end
-end
+result = root.scrape("http://some.scraping.page.tac42.net/", interval_ms: 1000)
 ```
 
++ `uri` is the URI of the page to be scraped.
++ `opt` is options as Hash. The following options are available.
+
+Yasuri uses `Mechanize` internally as an agent to do scraping.
+If you want to specify this instance, call `Node#scrape_with_agent(uri, agent, opt={})`.
+
+```ruby
+require 'logger'
+
+agent = Mechanize.new
+agent.log = Logger.new $stderr
+agent.request_headers = {
+  # ...
+}
+
+result = root.scrape_with_agent(
+  "http://some.scraping.page.tac42.net/",
+  agent,
+  interval_ms: 1000)
+```
+
+### `opt`
+#### `interval_ms`
+Interval [milliseconds] for requesting multiple pages.
+
+If omitted, requests will be made continuously without an interval, but if requests to many pages are expected, it is strongly recommended to specify an interval time to avoid high load on the target host.
+
+#### `retry_count`
+Number of retries when page acquisition fails. If omitted, it will retry 5 times.
+
+#### `symbolize_names`
+If true, returns the keys of the result set as symbols.
+
+--------------------------
+## Node
+
+Node is a node or leaf of the parse tree, which has `Type`, `Name`, `Path`, `Childlen`, and `Options`, and scrapes according to its `Type`. (Note that only `MapNode` does not have `Path`).
+
+
 #### Type
 Type meen behavior of Node.
 
@@ -114,17 +228,20 @@ Type meen behavior of Node.
 - *Struct*
 - *Links*
 - *Paginate*
+- *Map*
 
-### Name
+See the description of each node for details.
+
+#### Name
 Name is used keys in returned hash.
 
-### Path
+#### Path
 Path determine target node by xpath or css selector. It given by Machinize `search`.
 
-### Childlen
+#### Childlen
 Child nodes. TextNode has always empty set, because TextNode is leaf.
 
-### Options
+#### Options
 Parse options. It different in each types. You can get options and values by `opt` method.
 
 ```ruby
@@ -136,10 +253,12 @@ node.opt #=> {:truncate => /^[^,]+/, :proc => nil}
 ## Text Node
 TextNode return scraped text. This node have to be leaf.
 
+
+
 ### Example
 
 ```html
-<!-- http://yasuri.example.net -->
+<!-- http://yasuri.example.tac42.net -->
 <html>
   <head></head>
   <body>
@@ -150,25 +269,24 @@ TextNode return scraped text. This node have to be leaf.
 ```
 
 ```ruby
-agent = Mechanize.new
-page = agent.get("http://yasuri.example.net")
-
 p1  = Yasuri.text_title '/html/body/p[1]'
 p1t = Yasuri.text_title '/html/body/p[1]', truncate:/^[^,]+/
-p2u = Yasuri.text_title '/html/body/p[2]', proc: :upcase
+p2u = Yasuri.text_title '/html/body/p[1]', proc: :upcase
 
-p1.inject(agent, page)   #=> { "title" => "Hello,World" }
-p1t.inject(agent, page)  #=> { "title" => "Hello" }
-node.inject(agent, page) #=> { "title" => "HELLO,YASURI" }
+p1.scrape("http://yasuri.example.tac42.net")   #=> "Hello,World"
+p1t.scrape("http://yasuri.example.tac42.net")  #=> "Hello"
+p2u.scrape("http://yasuri.example.tac42.net")  #=> "HELLO,WORLD"
 ```
 
+Note that if you want to scrape multiple elements in the same page at once, use `MapNode`. See the `MapNode` example for details.
+
 ### Options
 ##### `truncate`
 Match to regexp, and truncate text. When you use group, it will return first matched group only.
 
 ```ruby
 node  = Yasuri.text_example '/html/body/p[1]', truncate:/H(.+)i/
-node.inject(agent, index_page)
+node.scrape(uri)
 #=> { "example" => "ello,Yasur" }
 ```
 
@@ -179,21 +297,22 @@ If it is given `truncate` option, apply method after truncated.
 
 ```ruby
 node = Yasuri.text_example '/html/body/p[1]', proc: :upcase, truncate:/H(.+)i/
-node.inject(agent, index_page)
+node.scrape(uri)
 #=> { "example" => "ELLO,YASUR" }
 ```
 
 ## Struct Node
 Struct Node return structured text.
 
-At first, Struct Node narrow down sub-tags by `Path`. Child nodes parse narrowed tags, and struct node returns hash contains parsed result.
+At first, Struct Node narrow down sub-tags by `Path`.
+Child nodes parse narrowed tags, and struct node returns hash contains parsed result.
 
 If Struct Node `Path` matches multi sub-tags, child nodes parse each sub-tags and struct node returns array.
 
 ### Example
 
 ```html
-<!-- http://yasuri.example.net -->
+<!-- http://yasuri.example.tac42.net -->
 <html>
   <head>
     <title>Books</title>
@@ -234,15 +353,12 @@ If Struct Node `Path` matches multi sub-tags, child nodes parse each sub-tags an
 ```
 
 ```ruby
-agent = Mechanize.new
-page = agent.get("http://yasuri.example.net")
-
 node = Yasuri.struct_table '/html/body/table[1]/tr' do
   text_title    './td[1]'
   text_pub_date './td[2]'
-])
+end
 
-node.inject(agent, page)
+node.scrape("http://yasuri.example.tac42.net")
 #=> [ { "title"    => "The Perfect Insider",
 #       "pub_date" => "1996/4/5" },
 #     { "title"    => "Doctors in Isolated Room",
@@ -261,17 +377,14 @@ Struct node can contain not only Text node.
 ### Example
 
 ```ruby
-agent = Mechanize.new
-page = agent.get("http://yasuri.example.net")
-
 node = Yasuri.strucre_tables '/html/body/table' do
   struct_table './tr' do
     text_title    './td[1]'
     text_pub_date './td[2]'
   end
-])
+end
 
-node.inject(agent, page)
+node.scrape("http://yasuri.example.tac42.net")
 
 #=>      [ { "table" => [ { "title"    => "The Perfect Insider",
 #                           "pub_date" => "1996/4/5" },
@@ -304,7 +417,7 @@ Links Node returns parsed text in each linked pages.
 
 ### Example
 ```html
-<!-- http://yasuri.example.net -->
+<!-- http://yasuri.example.tac42.net -->
 <html>
   <head><title>Yasuri Test</title></head>
   <body>
@@ -317,7 +430,7 @@ Links Node returns parsed text in each linked pages.
 ```
 
 ```html
-<!-- http://yasuri.example.net/child01.html -->
+<!-- http://yasuri.example.tac42.net/child01.html -->
 <html>
   <head><title>Child 01 Test</title></head>
   <body>
@@ -331,7 +444,7 @@ Links Node returns parsed text in each linked pages.
 ```
 
 ```html
-<!-- http://yasuri.example.net/child02.html -->
+<!-- http://yasuri.example.tac42.net/child02.html -->
 <html>
   <head><title>Child 02 Test</title></head>
   <body>
@@ -341,7 +454,7 @@ Links Node returns parsed text in each linked pages.
 ```
 
 ```html
-<!-- http://yasuri.example.net/child03.html -->
+<!-- http://yasuri.example.tac42.net/child03.html -->
 <html>
   <head><title>Child 03 Test</title></head>
   <body>
@@ -354,20 +467,17 @@ Links Node returns parsed text in each linked pages.
 ```
 
 ```ruby
-agent = Mechanize.new
-page = agent.get("http://yasuri.example.net")
-
 node = Yasuri.links_title '/html/body/a' do
   text_content '/html/body/p'
 end
 
-node.inject(agent, page)
+node.scrape("http://yasuri.example.tac42.net")
 #=> [ {"content" => "Child 01 page."},
       {"content" => "Child 02 page."},
       {"content" => "Child 03 page."}]
 ```
 
-At first, Links Node find all links in the page by path. In this case, LinksNode find `/html/body/a` tags in `http://yasuri.example.net`. Then, open href attributes (`./child01.html`, `./child02.html` and `./child03.html`).
+At first, Links Node find all links in the page by path. In this case, LinksNode find `/html/body/a` tags in `http://yasuri.example.tac42.net`. Then, open href attributes (`./child01.html`, `./child02.html` and `./child03.html`).
 
 Then, Links Node and apply child nodes. Links Node will return applied result of each page as array.
 
@@ -381,7 +491,7 @@ Paginate Node parses and returns each pages that provid by paginate.
 Target page `page01.html` is like this. `page02.html` to `page04.html` are similarly.
 
 ```html
-<!-- http://yasuri.example.net/page01.html -->
+<!-- http://yasuri.example.tac42.net/page01.html -->
 <html>
   <head><title>Page01</title></head>
   <body>
@@ -401,21 +511,17 @@ Target page `page01.html` is like this. `page02.html` to `page04.html` are simil
 ```
 
 ```ruby
-agent = Mechanize.new
-page = agent.get("http://yasuri.example.net/page01.html")
-
-node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" do
+node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" , limit:3 do
          text_content '/html/body/p'
        end
 
-node.inject(agent, page)
-#=> [ {"content" => "Pagination01"},
-      {"content" => "Pagination02"},
-      {"content" => "Pagination03"},
-      {"content" => "Pagination04"}]
+node.scrape("http://yasuri.example.tac42.net/page01.html")
+#=> [ {"content" => "Patination01"},
+#     {"content" => "Patination02"},
+#     {"content" => "Patination03"}]
 ```
-
-Paginate Node require link for next page. In this case, it is `NextPage` `/html/body/nav/span/a[@class='next']`.
+Paginate Node require link for next page.
+In this case, it is `NextPage` `/html/body/nav/span/a[@class='next']`.
 
 ### Options
 ##### `limit`
@@ -425,7 +531,7 @@ Upper limit of open pages in pagination.
 node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" , limit:2 do
          text_content '/html/body/p'
        end
-node.inject(agent, page)
+node.scrape(uri)
 #=> [ {"content" => "Pagination01"}, {"content" => "Pagination02"}]
 ```
 Paginate Node open upto 2 given by `limit`. In this situation, pagination has 4 pages, but result Array has 2 texts because given `limit:2`.
@@ -434,33 +540,177 @@ Paginate Node open upto 2 given by `limit`. In this situation, pagination has 4
 `flatten` option expands each page results.
 
 ```ruby
-agent = Mechanize.new
-page = agent.get("http://yasuri.example.net/page01.html")
-
 node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" , flatten:true do
          text_title   '/html/head/title'
          text_content '/html/body/p'
        end
-node.inject(agent, page)
+node.scrape("http://yasuri.example.tac42.net/page01.html")
 
 #=> [ {"title" => "Page01",
-       "content" => "Patination01"},
-      {"title"   => "Page01",
-       "content" => "Patination02"},
-      {"title"   => "Page01",
-       "content" => "Patination03"}]
+#      "content" => "Patination01"},
+#     {"title"   => "Page01",
+#      "content" => "Patination02"},
+#     {"title"   => "Page01",
+#      "content" => "Patination03"}]
 
 
 node = Yasuri.pages_root "/html/body/nav/span/a[@class='next']" , flatten:true do
         text_title   '/html/head/title'
         text_content '/html/body/p'
       end
-node.inject(agent, page)
+node.scrape("http://yasuri.example.tac42.net/page01.html")
 
 #=> [ "Page01",
-      "Patination01",
-      "Page02",
-      "Patination02",
-      "Page03",
-      "Patination03"]
+#     "Patination01",
+#     "Page02",
+#     "Patination02",
+#     "Page03",
+#     "Patination03"]
+```
+
+## Map Node
+*MapNode* is a node that summarizes the results of scraping. This node is always a branch node in the parse tree.
+
+### Example
+
+```html
+<!-- http://yasuri.example.tac42.net -->
+<html>
+  <head><title>Yasuri Example</title></head>
+  <body>
+    <p>Hello,World</p>
+    <p>Hello,Yasuri</p>
+  </body>
+</html>
+```
+
+```ruby
+tree = Yasuri.map_root do
+  text_title  '/html/head/title'
+  text_body_p '/html/body/p[1]'
+end
+
+tree.scrape("http://yasuri.example.tac42.net") #=> { "title" => "Yasuri Example", "body_p" => "Hello,World" }
+
+
+tree = Yasuri.map_root do
+  map_group1 { text_child01  '/html/body/a[1]' }
+  map_group2 do
+    text_child01 '/html/body/a[1]'
+    text_child03 '/html/body/a[3]'
+  end
+end
+
+tree.scrape("http://yasuri.example.tac42.net") #=> {
+#   "group1" => {
+#           "child01" => "child01"
+#         },
+#         "group2" => {
+#           "child01" => "child01",
+#           "child03" => "child03"
+#         }
+# }
+```
+
+### Options
+None.
+
+
+-------------------------
+## Usage
+
+### Use as library
+When used as a library, the tree can be defined in DSL, json, or yaml format.
+
+```ruby
+require 'yasuri'
+
+# 1. Create a parse tree.
+# Define by Ruby's DSL
+tree = Yasuri.links_title '/html/body/a' do
+         text_name '/html/body/p'
+       end
+
+# Define by JSON
+src = <<-EOJSON
+{
+  links_title": {
+    "path": "/html/body/a",
+    "text_name": "/html/body/p"
+  }
+}
+EOJSON
+tree = Yasuri.json2tree(src)
+
+
+# Define by YAML
+src = <<-EOYAML
+links_title:
+  path: "/html/body/a"
+  text_name: "/html/body/p"
+EOYAML
+tree = Yasuri.yaml2tree(src)
+
+# 2. Give the URL to start parsing
+tree.inject(uri)
+```
+
+### Use as CLI tool
+
+**Help**
+```sh
+$ yasuri help scrape
+Usage:
+  yasuri scrape <URI> [[--file <TREE_FILE>] or [--json <JSON>]]
+
+Options:
+  f, [--file=FILE]  # path to file that written yasuri tree as json or yaml
+  j, [--json=JSON]  # yasuri tree format json string
+  i, [--interval=N]  # interval each request [ms]
+
+Getting from <URI> and scrape it. with <JSON> or json/yml from <TREE_FILE>. They should be Yasuri's format json or yaml string.
+```
+
+In the CLI tool, you can specify the parse tree in either of the following ways.
++ `--file`, `-f` : option to read the parse tree in json or yaml format output to a file.
++ `--json`, `-j` : option to specify the parse tree directly as a string.
+
+
+**Example of specifying a parse tree as a file**
+```sh
+% cat sample.yml
+text_title: "/html/head/title"
+text_desc: "//*[@id=\"intro\"]/p"
+
+% yasuri scrape "https://www.ruby-lang.org/en/" --file sample.yml
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+
+% cat sample.json
+{
+  "text_title": "/html/head/title",
+  "text_desc": "//*[@id=\"intro\"]/p"
+}
+
+% yasuri scrape "https://www.ruby-lang.org/en/" --file sample.json
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
 ```
+
+Whether the file is written in json or yaml will be determined automatically.
+
+**Example of specifying a parse tree directly in json**
+```sh
+$ yasuri scrape "https://www.ruby-lang.org/en/" -j '
+{
+  "text_title": "/html/head/title",
+  "text_desc": "//*[@id=\"intro\"]/p"
+}'
+
+{"title":"Ruby Programming Language","desc":"\n    A dynamic, open source programming language with a focus on\n    simplicity and productivity. It has an elegant syntax that is\n    natural to read and easy to write.\n    "}
+```
+
+#### Other options
++ `--interval`, `-i` : The interval [milliseconds] for requesting multiple pages.
+   **Example: Request at 1 second intervals**
+   ```sh
+   $ yasuri scrape "https://www.ruby-lang.org/en/" --file sample.yml --interval 1000
+   ```