gammo 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '009d6d5682151d83fe688e67ba57541bcccf5542b2865d8b85be77fae8156178'
+  data.tar.gz: 31a2f1d37e01a3c9e47db2b034965c75b0bc7ddd4d2f86826ae08fd37d199788
+SHA512:
+  metadata.gz: c77bcb2f3cc9b25ac7400eff41819289980b1ff9a53481cca51b1444bca24dbdd114ead1c90f6cec219b0c844b0e63775338a7a7f74910b24c0ab6b0a00e2d54
+  data.tar.gz: a907e000dd8d4c01bcdb3f834ec17fbc20bdddcc91c38b77add44d3b56116eef8b7a64acdbd967fff65b1076df65871899816493ca012b98947266cc1ba51d8c

data/.gitignore

@@ -0,0 +1,8 @@
+/.bundle/
+/.yardoc
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,6 @@
+---
+language: ruby
+cache: bundler
+rvm:
+  - 2.7.0
+before_install: gem install bundler -v 2.1.2

data/Gemfile

@@ -0,0 +1,9 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in gammo.gemspec
+gemspec
+
+gem 'yard'
+gem 'rake', '~> 12.0'
+gem 'test-unit', '~> 3.3.5'
+gem 'erubi'

data/Gemfile.lock

@@ -0,0 +1,27 @@
+PATH
+  remote: .
+  specs:
+    gammo (0.1.0)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    erubi (1.9.0)
+    power_assert (1.1.5)
+    rake (12.3.3)
+    test-unit (3.3.5)
+      power_assert
+    yard (0.9.20)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  erubi
+  gammo!
+  rake (~> 12.0)
+  test-unit (~> 3.3.5)
+  yard
+
+BUNDLED WITH
+   2.0.2

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2020 namusyaka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,177 @@
+# Gammo - A pure-Ruby HTML5 parser
+
+[![Build Status](https://travis-ci.org/namusyaka/gammo.svg?branch=master)](https://travis-ci.org/namusyaka/gammo)
+
+Gammo is an implementation of the HTML5 parsing algorithm which conforms [the WHATWG specification](https://html.spec.whatwg.org/multipage/parsing.html), without any dependencies. Given an HTML string, Gammo parses it and builds DOM tree based on the tokenization and tree-construction algorithm defined in WHATWG parsing algorithm.
+
+Gammo, its naming is inspired by [Gumbo](https://github.com/google/gumbo-parser). But Gammo is a fried tofu fritter made with vegetables.
+
+```ruby
+require 'gammo'
+require 'open-uri'
+
+parser = Gammo.new(open('https://google.com'))
+parser.parse #=> #<Gammo::Node::Document>
+```
+
+## Overview
+
+### Features
+
+- [Tokenization](#tokenization): Gammo has a tokenizer for implementing [the tokenization algorithm](https://html.spec.whatwg.org/multipage/parsing.html#tokenization).
+- [Parsing](#parsing): Gammo provides a parser which implements the parsing algorithm by the above tokenization and [the tree-construction algorithm](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction).
+- [Node](#node): Gammo provides the nodes which implement [WHATWG DOM specification](https://dom.spec.whatwg.org/) partially.
+- [Performance](#performance): Gammo does not prioritize performance, and there are a few potential performance notes.
+
+## Tokenizaton
+
+`Gammo::Tokenizer` implements the tokenization algorithm in WHATWG. You can get tokens in order by calling `Gammo::Tokenizer#next_token`.
+
+Here is a simple example for performing only the tokenizer.
+
+```ruby
+def dump_for(token)
+  puts "data: #{token.data}, class: #{token.class}"
+end
+
+tokenizer = Gammo::Tokenizer.new('<!doctype html><input type="button"><frameset>')
+dump_for tokenizer.next_token #=> data: html, class: Gammo::Tokenizer::DoctypeToken
+dump_for tokenizer.next_token #=> data: input, class: Gammo::Tokenizer::StartTagToken
+dump_for tokenizer.next_token #=> data: frameset, class: Gammo::Tokenizer::StartTagToken
+dump_for tokenizer.next_token #=> data: end of string, class: Gammo::Tokenizer::ErrorToken
+```
+
+The parser described below depends on this tokenizer, it applies the WHATWG parsing algorithm to the tokens extracted by this tokenization in order.
+
+### Token types
+
+The tokens generated by the tokenizer will be categorized into one of the following types:
+
+<table>
+  <thead>
+    <tr>
+      <th>Token type</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>Gammo::Tokenizer::ErrorToken</code></td>
+      <td>Represents an error token, it usually means end-of-string.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Tokenizer::TextToken</code></td>
+      <td>Represents a text token like "foo" which is inner text of elements.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Tokenizer::StartTagToken</code></td>
+      <td>Represents a start tag token like <code>&lt;a&gt;</code>.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Tokenizer::EndTagToken</code></td>
+      <td>Represents an end tag token like <code>&lt;/a&gt;</code>.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Tokenizer::SelfClosingTagToken</code></td>
+      <td>Represents a self closing tag token like <code>&lt;img /&gt;</code></td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Tokenizer::CommentToken</code></td>
+      <td>Represents a comment token like <code>&lt;!-- comment --&gt;</code>.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Tokenizer::DoctypeToken</code></td>
+      <td>Represents a doctype token like <code>&lt;!doctype html&gt;</code>.</td>
+    </tr>
+  </tbody>
+</table>
+
+## Parsing
+
+`Gammo::Parser` implements processing in [the tree-construction stage](https://html.spec.whatwg.org/multipage/parsing.html#tree-construction) based on the tokenization described above.
+
+A successfully parsed parser has the `document` accessor as the root document (this is the same as the return value of the `Gammo::Parser#parse`). From the `document` accessor, you can traverse the DOM tree constructed by the parser.
+
+```ruby
+require 'gammo'
+require 'pp'
+
+document = Gammo.new('<!doctype html><input type="button">').parse
+
+def dump_for(node, strm)
+  strm << node.to_h
+  return unless node && (child = node.first_child)
+  while child
+    dump_for(child, (strm.last[:children] ||= []))
+    child = child.next_sibling
+  end
+  strm
+end
+
+pp dump_for(document, [])
+```
+
+### Notes
+
+Currently, it's not possible to traverse the DOM tree with css selector or xpath like [Nokogiri](https://nokogiri.org/).
+However, Gammo plans to implement these features in the future.
+
+## Node
+
+The nodes generated by the parser will be categorized into one of the following types:
+
+<table>
+  <thead>
+    <tr>
+      <th>Node type</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>Gammo::Node::Error</code></td>
+      <td>Represents error node, it usually means end-of-string.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Node::Text</code></td>
+      <td>Represents the text node like "foo" which is inner text of elements.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Node::Document</code></td>
+      <td>Represents the root document type. It's always returned by <code>Gammo::Parser#document</code>.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Node::Element</code></td>
+      <td>Represents any elements of HTML like <code>&lt;p&gt;</code>.</td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Node::Comment</code></td>
+      <td>Represents comments like <code>&lt;!-- foo --&gt;</code></td>
+    </tr>
+    <tr>
+      <td><code>Gammo::Node::Doctype</code></td>
+      <td>Represents doctype like <code>&lt;!doctype html&gt;</code></td>
+    </tr>
+  </tbody>
+</table>
+
+For some nodes such as `Gammo::Node::Element` and `Gammo::Node::Document`, they contains pointers to nodes that can be referenced by itself, such as `Gammo::Node#next_sibling` or `Gammo::Node#first_child`. In addition, APIs such as `Gammo::Node#append_child` and `Gammo::Node#remove_child` that perform operations defined in DOM living standard are also provided.
+
+## Performance
+
+As mentioned in the features at the beginning, Gammo doesn't prioritize its performance.
+Thus, for example, Gammo is not suitable for very performance-sensitive applications (e.g. performing Gammo parsing synchronously from an incoming request from an end user).
+Instead, the goal is to work well with batch processing such as crawlers.
+Gammo places the highest priority on making it easy to parse HTML by peforming it without depending on native-extensions and external gems.
+
+## References
+
+This was developed with reference to the following softwares.
+
+- [x/net/html](https://godoc.org/golang.org/x/net/html): I've been working on this package, it gave me strong reason to make this happen.
+- [Blink](https://www.chromium.org/blink): Blink gave me great impression about tree construction.
+- [html5lib-tests](https://github.com/html5lib/html5lib-tests): Gammo relies on this test.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,25 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+require 'yaml'
+require 'erubi'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: :test
+
+def camelize(str)
+  str.sub(/^[a-z\d]*/) { $&.capitalize }.sub(/\-[a-z]*/) { $&.slice(1..-1).capitalize }
+end
+
+task default: :test
+
+task :generate do
+  data = YAML.load(File.read('misc/html.yaml'), symbolize_names: true)
+  @tags = data.each_value.inject(:+).uniq
+  table = eval(Erubi::Engine.new(File.read('misc/table.erubi')).src, binding)
+  File.write('lib/gammo/tags/table.rb', table)
+end

data/gammo.gemspec

@@ -0,0 +1,23 @@
+require_relative 'lib/gammo/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "gammo"
+  spec.version       = Gammo::VERSION
+  spec.authors       = ["namusyaka"]
+  spec.email         = ["namusyaka@gmail.com"]
+
+  spec.summary       = %q{An HTML parser which implements WHATWG parsing algorithm.}
+  spec.description   = %q{Gammo is an implementation of the HTML5 parsing algorithm which conforms the WHATWG specification with pure Ruby.}
+  spec.homepage      = "https://github.com/namusyaka/gammo"
+  spec.license       = "MIT"
+  spec.required_ruby_version = Gem::Requirement.new(">= 2.3.0")
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/namusyaka/gammo"
+  spec.files         = Dir.chdir(File.expand_path('..', __FILE__)) do
+    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+end

data/lib/gammo.rb

@@ -0,0 +1,15 @@
+require "gammo/version"
+require "gammo/parser"
+require "gammo/fragment_parser"
+
+module Gammo
+  # Constructs a parser based on the input.
+  #
+  # @param [String] input
+  # @param [TrueClass, FalseClass] fragment
+  # @param [Hash] options
+  # @return [Gammo::Parser]
+  def self.new(input, fragment: false, **options)
+    (fragment ? FragmentParser : Parser).new(input, **options)
+  end
+end

data/lib/gammo/attribute.rb

@@ -0,0 +1,17 @@
+module Gammo
+  # Class for representing an attribute.
+  class Attribute
+    attr_accessor :key, :value, :namespace
+
+    # Constructs an attribute with the key-value pair.
+    # @param [String] key
+    # @param [String] value
+    # @param [String] namespace
+    # @return [Attribute]
+    def initialize(key:, value:, namespace: nil)
+      @key       = key
+      @value     = value
+      @namespace = namespace
+    end
+  end
+end

data/lib/gammo/fragment_parser.rb

@@ -0,0 +1,65 @@
+require 'gammo/parser'
+
+module Gammo
+  # Class for parsing a fragment of an HTML input and building an HTML tree.
+  class FragmentParser < ::Gammo::Parser
+    # Constructs a parser instance for fragment parsing algorithm. 
+    # @see https://html.spec.whatwg.org/multipage/parsing.html#html-fragment-parsing-algorithm
+    # @param [String] input
+    # @param [Gammo::Node] context
+    # @raise [Gammo::ParseError] raises if context is not valid.
+    # @return Gammo::FragmentParser
+    def initialize(input, context:, **options)
+      validate_context(context)
+      super(input, context: context, **options)
+      @root          = Node::Element.new(tag: Tags::Html, data: Tags::Html.to_s)
+      @tokenizer     = Tokenizer.new(input, context: !context.namespace && context.tag.to_s)
+      @open_elements = NodeStack.new([@root])
+      document.append_child(@root)
+      template_stack << InTemplate if context.tag == Tags::Template
+      reset_insertion_mode
+      while context
+        if context.instance_of?(Node::Element) && context.tag == Tags::Form
+          @form = context
+          break
+        end
+        context = context.parent
+      end
+    end
+
+    # Parses a fragment of the current input and builds HTML tree from it.
+    # @raise [Gammo::ParseError] Raised if the parser gets error while parsing.
+    # @return [Array<Gammo::Node>]
+    def parse
+      super
+      parent     = context ? @root : document
+      child      = parent.first_child
+      collection = []
+      while child
+        node = child.next_sibling
+        parent.remove_child(child)
+        collection << child
+        child = node
+      end
+      collection
+    end
+
+    # Always returns true.
+    # @return [TrueClass]
+    # @!visibility private
+    def fragment?
+      true
+    end
+
+    # Validates given context. Raises {Gammo::ParseError} if context is not
+    # {Gammo::Node}.
+    # @param [Gammo::Node] context
+    # @raise [Gammo::ParseError]
+    def validate_context(context)
+      fail ParseError, 'given non-element node in "context"' unless context.instance_of?(Node::Element)
+      unless context.tag == Tags.lookup(context.data)
+        fail ParseError, "inconsistent context node, tag = #{context.tag}, data = #{Tags.lookup(context.data)}"
+      end
+    end
+  end
+end

data/lib/gammo/node.rb

@@ -0,0 +1,157 @@
+module Gammo
+  # Class for representing Node.
+  # https://html.spec.whatwg.org/multipage/parsing.html#tokenization
+  class Node
+    # Represents the error token.
+    Error = Class.new(Node)
+
+    # Represents the text token.
+    Text = Class.new(Node)
+
+    # Represents the root document token.
+    Document = Class.new(Node)
+
+    # Represents the element token including start, end and self-closing token.
+    Element = Class.new(Node)
+
+    # Represents the comment token like "<!-- foo -->".
+    Comment = Class.new(Node) 
+
+    # Represents the document type token.
+    Doctype = Class.new(Node) 
+
+    # Represents the marker defined in 12.2.4.3.
+    # https://html.spec.whatwg.org/multipage/parsing.html#tokenization
+    ScopeMarker = Class.new(Node)
+
+    # Default scope marker is inserted when entering applet,
+    # object, marquee, template, td, th, and caption elements, and are used
+    # to prevent formatting from "leaking" into applet, object, marquee,
+    # template, td, th, and caption elements"
+    DEFAULT_SCOPE_MARKER = Node::ScopeMarker.new
+
+    # Raised if uncaught node is given for particular operations.
+    # @!visibility private
+    UncaughtTypeError = Class.new(ArgumentError)
+
+    # Raised if anything goes wrong on hierarchy while node operations.
+    # @!visibility private
+    HierarchyRequestError = Class.new(ArgumentError)
+
+    # `parent` is the pointer for the parent node.
+    attr_accessor :parent
+
+    # `first_child` and `last_child` are pointers for the first and the last nodes.
+    attr_accessor :first_child, :last_child
+
+    # `previous_sibling` and `next_sibling` are pointers for the previous and next sibling nodes.
+    attr_accessor :previous_sibling, :next_sibling
+
+    # Properties required to represent node.
+    attr_accessor :tag, :data, :namespace, :attributes
+
+    # Constructs a node which represents HTML element node.
+    # @param [String] tag
+    # @param [String] data
+    # @param [String, NilClass] namespace
+    # @param [Hash(String => String)] attributes
+    # @return [Gammo::Node]
+    def initialize(tag: nil, data: nil, namespace: nil, attributes: [])
+      @tag        = tag
+      @data       = data
+      @namespace  = namespace
+      @attributes = attributes
+    end
+
+    # Inserts a node before a reference node as a child of a specified parent node.
+    # @param [Gammo::Node] node
+    # @param [Gammo::Node] ref
+    # @raise [HierarchyRequestError] Raised if given node is already attached to the self node.
+    # @return [Gammo::Node] A node inserted before the reference node.
+    def insert_before(node, ref)
+      raise HierarchyRequestError,
+        'insert_before called for an attached child node' if attached?(node)
+      if ref
+        previous_sibling, next_sibling = ref.previous_sibling, ref
+      else
+        previous_sibling = last_child
+      end
+      if previous_sibling
+        previous_sibling.next_sibling = node
+      else
+        @first_child = node
+      end
+      if next_sibling
+        next_sibling.previous_sibling = node
+      else
+        @last_child = node
+      end
+      node.parent = self
+      node.previous_sibling = previous_sibling
+      node.next_sibling = next_sibling
+      node
+    end
+
+    # Appends given `child` into self node.
+    # @param [Gammo::Node] child
+    # @raise [HierarchyRequestError] Raised if given node is already attached to the self node.
+    # @return [Gammo::Node] A node appended into the self node.
+    def append_child(child)
+      raise HierarchyRequestError,
+        'append_child called for an attached child node' if attached?(child)
+      if last = last_child
+        last.next_sibling = child
+      else
+        @first_child = child
+      end
+      @last_child = child
+      child.parent = self
+      child.previous_sibling = last
+      child
+    end
+
+    # Removes given `child` from self node.
+    # @param [Gammo::Node] child
+    # @raise [UncaughtTypeError] Raised unless given node is not child of the self node.
+    # @return [Gammo::Node] A node removed from the self node.
+    def remove_child(child)
+      raise UncaughtTypeError,
+        'remove_child called for a non-child node' unless child?(child)
+      @first_child = child.next_sibling if first_child == child
+      child.next_sibling.previous_sibling = child.previous_sibling if child.next_sibling
+      @last_child = child.previous_sibling if last_child == child
+      child.previous_sibling.next_sibling = child.next_sibling if child.previous_sibling
+      child.parent = child.previous_sibling = child.next_sibling = nil
+      child
+    end
+
+    # Clones self into a new node.
+    # @return [Gammo::Node]
+    # @!visibility private
+    def clone
+      self.class.new(tag: self.tag, data: self.data, attributes: self.attributes.dup)
+    end
+
+    # @!visibility private
+    def to_h
+      {
+        tag: tag,
+        data: data,
+        attributes: attributes,
+        type: self.class
+      }
+    end
+
+    private
+
+    # @!visibility private
+    def attached?(node)
+      node.parent || node.previous_sibling || node.next_sibling
+    end
+
+    # @!visibility private
+    def child?(node)
+      node.parent == self
+    end
+  end
+end