gammo 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '009d6d5682151d83fe688e67ba57541bcccf5542b2865d8b85be77fae8156178'
-  data.tar.gz: 31a2f1d37e01a3c9e47db2b034965c75b0bc7ddd4d2f86826ae08fd37d199788
+  metadata.gz: f88ade2267388af8d29137f6ff0db934d324d77f5bface53cd6ae9fea3f1e466
+  data.tar.gz: 92642a6208ede13520e29c20529f29fd21bb040eac39ec4af97935990b94b0eb
 SHA512:
-  metadata.gz: c77bcb2f3cc9b25ac7400eff41819289980b1ff9a53481cca51b1444bca24dbdd114ead1c90f6cec219b0c844b0e63775338a7a7f74910b24c0ab6b0a00e2d54
-  data.tar.gz: a907e000dd8d4c01bcdb3f834ec17fbc20bdddcc91c38b77add44d3b56116eef8b7a64acdbd967fff65b1076df65871899816493ca012b98947266cc1ba51d8c
+  metadata.gz: f15b94b27a1738234662a2f196c440b97e4cd2b83a9c9bb468d7c4cfd4058ad6482b0ac06559dc4cd50a0dc8d41ba6114db1a09d51177bca511d1ce4196ef9e6
+  data.tar.gz: 5377c2d17574abcc5e0069e7506ad5e47da211c970a79718218c88d9099723c04a7edba0b58da61eb9898e24fc300c04b3f8e2daba49ce41994ec990c27ff5fe

data/Gemfile

@@ -7,3 +7,6 @@ gem 'yard'
 gem 'rake', '~> 12.0'
 gem 'test-unit', '~> 3.3.5'
 gem 'erubi'
+
+gem 'racc'
+gem 'simplecov'

data/Gemfile.lock

@@ -6,9 +6,15 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
+    docile (1.3.2)
     erubi (1.9.0)
     power_assert (1.1.5)
+    racc (1.5.0)
     rake (12.3.3)
+    simplecov (0.18.5)
+      docile (~> 1.1)
+      simplecov-html (~> 0.11)
+    simplecov-html (0.12.2)
     test-unit (3.3.5)
       power_assert
     yard (0.9.20)
@@ -19,9 +25,11 @@ PLATFORMS
 DEPENDENCIES
   erubi
   gammo!
+  racc
   rake (~> 12.0)
+  simplecov
   test-unit (~> 3.3.5)
   yard
 
 BUNDLED WITH
-   2.0.2
+   2.1.4

data/README.md

@@ -1,6 +1,11 @@
 # Gammo - A pure-Ruby HTML5 parser
 
 [![Build Status](https://travis-ci.org/namusyaka/gammo.svg?branch=master)](https://travis-ci.org/namusyaka/gammo)
+[![GitHub issues](https://img.shields.io/github/issues/namusyaka/gammo)](https://github.com/namusyaka/gammo/issues)
+[![GitHub forks](https://img.shields.io/github/forks/namusyaka/gammo?color=brightgreen)](https://github.com/namusyaka/gammo/network)
+[![GitHub stars](https://img.shields.io/github/stars/namusyaka/gammo?color=brightgreen)](https://github.com/namusyaka/gammo/stargazers)
+[![GitHub license](https://img.shields.io/github/license/namusyaka/gammo?color=brightgreen)](https://github.com/namusyaka/gammo/blob/master/LICENSE.txt)
+[![Documentation](http://img.shields.io/:yard-docs-38c800.svg)](http://www.rubydoc.info/gems/gammo/frames)
 
 Gammo is an implementation of the HTML5 parsing algorithm which conforms [the WHATWG specification](https://html.spec.whatwg.org/multipage/parsing.html), without any dependencies. Given an HTML string, Gammo parses it and builds DOM tree based on the tokenization and tree-construction algorithm defined in WHATWG parsing algorithm.
 
@@ -10,7 +15,7 @@ Gammo, its naming is inspired by [Gumbo](https://github.com/google/gumbo-parser)
 require 'gammo'
 require 'open-uri'
 
-parser = Gammo.new(open('https://google.com'))
+parser = open('https://google.com') { |f| Gammo.new(f.read) }
 parser.parse #=> #<Gammo::Node::Document>
 ```
 
@@ -21,6 +26,7 @@ parser.parse #=> #<Gammo::Node::Document>
 - [Tokenization](#tokenization): Gammo has a tokenizer for implementing [the tokenization algorithm](https://html.spec.whatwg.org/multipage/parsing.html#tokenization).
 - [Parsing](#parsing): Gammo provides a parser which implements the parsing algorithm by the above tokenization and [the tree-construction algorithm](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction).
 - [Node](#node): Gammo provides the nodes which implement [WHATWG DOM specification](https://dom.spec.whatwg.org/) partially.
+- [DOM Tree Traversal](#dom-tree-traversal): Gammo provides a way of DOM tree traversal.
 - [Performance](#performance): Gammo does not prioritize performance, and there are a few potential performance notes.
 
 ## Tokenizaton
@@ -155,7 +161,394 @@ The nodes generated by the parser will be categorized into one of the following
   </tbody>
 </table>
 
-For some nodes such as `Gammo::Node::Element` and `Gammo::Node::Document`, they contains pointers to nodes that can be referenced by itself, such as `Gammo::Node#next_sibling` or `Gammo::Node#first_child`. In addition, APIs such as `Gammo::Node#append_child` and `Gammo::Node#remove_child` that perform operations defined in DOM living standard are also provided.
+For some nodes such as `Gammo::Node::Element` and `Gammo::Node::Document`, they contain pointers to nodes that can be referenced by itself, such as `Gammo::Node#next_sibling` or `Gammo::Node#first_child`. In addition, APIs such as `Gammo::Node#append_child` and `Gammo::Node#remove_child` that perform operations defined in DOM living standard are also provided.
+
+## DOM Tree Traversal
+
+Currently, XPath 1.0 is the only way for traversing DOM tree built by Gammo.
+CSS selector support is also planned but not having any ETA.
+
+### XPath 1.0 (experimental)
+
+Gammo has an original lexer/parser for XPath 1.0, it's provided as a helper in the DOM tree built by Gammo. 
+Here is a simple example:
+
+```ruby
+document = Gammo.new('<!doctype html><input type="button">').parse
+node_set = document.xpath('//input[@type="button"]') #=> "<Gammo::XPath::NodeSet>"
+
+node_set.length #=> 1
+node_set.first #=> "<Gammo::Node::Element>"
+```
+
+**Since this is implemented by full scratch, Gammo is providing this support as a very experimental feature.**
+Please [file an issue](/issues/new) if you find bugs.
+
+#### Example
+
+Before proceeding at the details of XPath support, let's have a look at a few simple examples.
+Given a sample HTML text and its DOM tree:
+
+```ruby
+document = Gammo.new(<<-EOS).parse
+<!DOCTYPE html>
+<html>
+<head>
+</head>
+<body>
+  <h1>namusyaka.com</h1>
+  <p class="description">Here is a sample web site.</p>
+  <ul>
+    <li>hello</li>
+    <li>world</li>
+  </ul>
+  <ul id="links">
+    <li>Google <a href="https://google.com/">google.com</a></li>
+    <li>GitHub <a href="https://github.com/namusyaka">github.com/namusyaka</a></li>
+  </ul>
+</body>
+</html>
+EOS
+```
+
+The following XPath expression gets all `li` elements and prints those text contents:
+
+```ruby
+document.xpath('//li').each do |elm|
+  puts elm.inner_text
+end
+```
+
+The following XPath expression gets all `li` elements under the `ul` element having the `id=links` attribute:
+
+```ruby
+document.xpath('//ul[@id="links"]/li').each do |elm|
+  puts elm.inner_text
+end
+```
+
+The following XPath expression gets each text node for each `li` element under the `ul` element having the `id=links` attribute:
+
+```ruby
+document.xpath('//ul[@id="links"]/li/text()').each do |elm|
+  puts elm.data
+end
+```
+
+#### Axis Specifiers
+
+In the combination with Gammo, the axis specifier indicates navigation direction within the DOM tree built by Gammo. Here is list of axes. As you can see, Gammo fully supports the all of axes.
+
+<table>
+  <thead>
+    <tr>
+      <th>Full Syntax</th>
+      <th>Abbreviated Syntax</th>
+      <th>Supported</th>
+      <th>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>ancestor</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>ancestor-or-self</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>attribute</code></td>
+      <td><code>@</code></td>
+      <td>yes</td>
+      <td><code>@abc</code> is the alias for <code>attribute::abc</code></td>
+    </tr>
+    <tr>
+      <td><code>child</code></td>
+      <td><code></code></td>
+      <td>yes</td>
+      <td><code>abc</code> is the short for <code>child::abc</code></td>
+    </tr>
+    <tr>
+      <td><code>descendant</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>descendant-or-self</code></td>
+      <td><code>//</code></td>
+      <td>yes</td>
+      <td><code>//</code> is the alias for <code>/descendant-or-self::node()/</code></td>
+    </tr>
+    <tr>
+      <td><code>following</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>following-sibling</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>namespace</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>parent</code></td>
+      <td><code>..</code></td>
+      <td>yes</td>
+      <td><code>..</code> is the alias for <code>parent::node()</code></td>
+    </tr>
+    <tr>
+      <td><code>preceding</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>preceding-sibling</code></td>
+      <td></td>
+      <td>yes</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>self</code></td>
+      <td><code>.</code></td>
+      <td>yes</td>
+      <td><code>.</code> is the alias for <code>self::node()</code></td>
+    </tr>
+  </tbody>
+</table>
+
+#### Node Test
+
+Node tests consist of specific node names or more general expressions. Although particular syntax like `:` should work for specifying namespace prefix in XPath, Gammo does not support it yet as it's [not a core feature in HTML5](https://html.spec.whatwg.org/multipage/introduction.html#html-vs-xhtml).
+
+<table>
+  <thead>
+    <tr>
+      <th>Full Syntax</th>
+      <th>Supported</th>
+      <th>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>text()</code></td>
+      <td>yes</td>
+      <td>Finds a node of type text, e.g. <code>hello</code> in <code><p>hello <a href="https://hello">world</a></p></code></td>
+    </tr>
+    <tr>
+      <td><code>comment()</code></td>
+      <td>yes</td>
+      <td>Finds a node of type comment, e.g. <code><!-- comment --></code></td>
+    </tr>
+    <tr>
+      <td><code>node()</code></td>
+      <td>yes</td>
+      <td>Finds any node at all.</td>
+    </tr>
+  </tbody>
+</table>
+
+Also note that the `processing-instruction` is not supported. There is no plan to support it.
+
+#### Operators
+
+- The `/`, `//` and `[]` are used in the path expression.
+- The union operator `|` forms the union of two node sets.
+- The boolean operators: `and`, `or`
+- The arithmetic operators: `+`, `-`, `*`, `div` and `mod`
+- Comparison operators: `=`, `!=`, `<`, `>`, `<=`, `>=`
+
+#### Functions
+
+XPath 1.0 defines four data types (nodeset, string, number, boolean) and there are various functions based on the types. Gammo supports those functions partially, please check it to be supported before using functions.
+
+##### Node set functions
+
+<table>
+  <thead>
+    <tr>
+      <th>Function Name</th>
+      <th>Supported</th>
+      <th>Specification</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>last()</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-last</td>
+    </tr>
+    <tr>
+      <td><code>position()</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-position</td>
+    </tr>
+    <tr>
+      <td><code>count(node-set)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-count</td>
+    </tr>
+  </tbody>
+</table>
+
+##### String Functions
+
+<table>
+  <thead>
+    <tr>
+      <th>Function Name</th>
+      <th>Supported</th>
+      <th>Specification</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>string(object?)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string</td>
+    </tr>
+    <tr>
+      <td><code>concat(string, string, string*)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-concat</td>
+    </tr>
+    <tr>
+      <td><code>starts-with(string, string)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-starts-with</td>
+    </tr>
+    <tr>
+      <td><code>contains(string, string)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-contains</td>
+    </tr>
+    <tr>
+      <td><code>substring-before(string, string)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-substring-before</td>
+    </tr>
+    <tr>
+      <td><code>substring-after(string, string)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-substring-after</td>
+    </tr>
+    <tr>
+      <td><code>substring(string, number, number?)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-substring</td>
+    </tr>
+    <tr>
+      <td><code>string-length(string?)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string-length</td>
+    </tr>
+    <tr>
+      <td><code>normalize-space(string?)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string-normalize-space</td>
+    </tr>
+    <tr>
+      <td><code>translate(string, string, string)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-string-translate</td>
+    </tr>
+  </tbody>
+</table>
+
+##### Boolean Functions
+
+<table>
+  <thead>
+    <tr>
+      <th>Function Name</th>
+      <th>Supported</th>
+      <th>Specification</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>boolean(object)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-boolean</td>
+    </tr>
+    <tr>
+      <td><code>not(object)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-not</td>
+    </tr>
+    <tr>
+      <td><code>true()</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-true</td>
+    </tr>
+    <tr>
+      <td><code>false()</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-false</td>
+    </tr>
+    <tr>
+      <td><code>lang()</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-lang</td>
+    </tr>
+  </tbody>
+</table>
+
+##### Number Functions
+
+<table>
+  <thead>
+    <tr>
+      <th>Function Name</th>
+      <th>Supported</th>
+      <th>Specification</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>number(object?)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-number</td>
+    </tr>
+    <tr>
+      <td><code>sum(node-set)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-sum</td>
+    </tr>
+    <tr>
+      <td><code>floor(number)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-floor</td>
+    </tr>
+    <tr>
+      <td><code>ceiling(number)</code></td>
+      <td>yes</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-ceiling</td>
+    </tr>
+    <tr>
+      <td><code>round(number)</code></td>
+      <td>no</td>
+      <td>https://www.w3.org/TR/1999/REC-xpath-19991116/#function-round</td>
+    </tr>
+  </tbody>
+</table>
+
+### CSS Selector
+
+TBD.
 
 ## Performance
 
@@ -175,3 +568,10 @@ This was developed with reference to the following softwares.
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Release History
+
+- v0.2.0
+    - XPath 1.0 support [#4](https://github.com/namusyaka/gammo/pull/4)
+- v0.1.0
+    - Initial Release