nokogumbo 1.5.0 → 2.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 96fa61565f78d5491e0b6d5b505cf936524745eb848b8b6584fc15e20c7ae35b
-  data.tar.gz: e5416f71bbe90323f04b8aad4dc48b28947e43a9eb46f446f8ca1444f519a07b
+  metadata.gz: a84b367d94046358f7844781b0f92cea51a75e052d54e35b53ab03602743f1b8
+  data.tar.gz: 8d96a5adfa701f658f7ba193ee96bb8a7e6901c1ff4d3fb2dad6f3e372ce66d2
 SHA512:
-  metadata.gz: 676bf3585d38cd4ad5c72b8b3afd4952e248c747683ae1072dd43f6ce1ccd279177e4d0c75a9821ed76d32806333128152231349d8d113ae5d81279580b13004
-  data.tar.gz: 3459078d96977399e75551c4a3ee5623091f48569984b771e540ec111125f5af91e39a8d78cbd3ce9280326b1b9395dc4a0b0d7f0a72294876682cb9fe35e3d9
+  metadata.gz: de2472c6ff89e3f0076a44ac13fa67688e82f909b265a2b70fe45225daf01aaf6059c6ca94f06e10ff94e10ac8a8f42b685e63f494849f04f3af56f337a73382
+  data.tar.gz: 3880defdaa15cb278236cf170d5727d1d73b14698f1ea41e7a7141da7a2fe8c3bafea19367196214c0dc0c1c27854602714d80abd30ecfd6be90f4277f3e33d7

data/README.md

@@ -1,23 +1,21 @@
-Nokogumbo - a Nokogiri interface to the Gumbo HTML5 parser.
-===========
+# Nokogumbo - a Nokogiri interface to the Gumbo HTML5 parser.
 
-Nokogumbo provides the ability for a Ruby program to invoke the 
-[Gumbo HTML5 parser](https://github.com/google/gumbo-parser#readme)
+Nokogumbo provides the ability for a Ruby program to invoke 
+[our version of the  Gumbo HTML5 parser](https://github.com/rubys/nokogumbo/tree/master/gumbo-parser/src)
 and to access the result as a
 [Nokogiri::HTML::Document](http://rdoc.info/github/sparklemotion/nokogiri/Nokogiri/HTML/Document).
 
-[![Build Status](https://travis-ci.org/rubys/nokogumbo.svg)](https://travis-ci.org/rubys/nokogumbo) 
+[![Travis-CI Build Status](https://travis-ci.org/rubys/nokogumbo.svg)](https://travis-ci.org/rubys/nokogumbo) 
+[![Appveyor Build Status](https://ci.appveyor.com/api/projects/status/github/rubys/nokogumbo)](https://ci.appveyor.com/project/rubys/nokogumbo/branch/master)
 
-Usage
------
+## Usage
 
 ```ruby
 require 'nokogumbo'
-doc = Nokogiri::HTML5(string)
+doc = Nokogiri.HTML5(string)
 ```
 
-An experimental _fragment_ method is also provided.  While not HTML5
-compliant, it may be useful:
+To parse an HTML fragment, a `fragment` method is provided.
 
 ```ruby
 require 'nokogumbo'
@@ -32,21 +30,207 @@ require 'nokogumbo'
 doc = Nokogiri::HTML5.get(uri)
 ```
 
-Example
------
+## Parsing options
+The document and fragment parsing methods,
+- `Nokogiri.HTML5(html, url = nil, encoding = nil, options = {})`
+- `Nokogiri::HTML5.parse(html, url = nil, encoding = nil, options = {})`
+- `Nokogiri::HTML5::Document.parse(html, url = nil, encoding = nil, options = {})`
+- `Nokogiri::HTML5.fragment(html, encoding = nil, options = {})`
+- `Nokogiri::HTML5::DocumentFragment.parse(html, encoding = nil, options = {})`
+support options that are different from Nokogiri's.
+
+The three currently supported options are `:max_errors`, `:max_tree_depth` and
+`:max_attributes`, described below.
+
+### Error reporting
+Nokogumbo contains an experimental parse error reporting facility. By default,
+no parse errors are reported but this can be configured by passing the
+`:max_errors` option to `::parse` or `::fragment`.
+
 ```ruby
 require 'nokogumbo'
-puts Nokogiri::HTML5.get('http://nokogiri.org').search('ol li')[2].text
+doc = Nokogiri::HTML5.parse('<span/>Hi there!</span foo=bar />', max_errors: 10)
+doc.errors.each do |err|
+  puts(err)
+end
 ```
 
-Use `.to_html` instead of `.to_s` when parsing and serializing multiple times
+This prints the following.
 ```
+1:1: ERROR: Expected a doctype token
+<span/>Hi there!</span foo=bar />
+^
+1:1: ERROR: Start tag of nonvoid HTML element ends with '/>', use '>'.
+<span/>Hi there!</span foo=bar />
+^
+1:17: ERROR: End tag ends with '/>', use '>'.
+<span/>Hi there!</span foo=bar />
+                ^
+1:17: ERROR: End tag contains attributes.
+<span/>Hi there!</span foo=bar />
+                ^
+```
+
+Using `max_errors: -1` results in an unlimited number of errors being
+returned.
+
+The errors returned by `#errors` are instances of
+[`Nokogiri::XML::SyntaxError`](https://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SyntaxError).
+
+The [HTML
+standard](https://html.spec.whatwg.org/multipage/parsing.html#parse-errors)
+defines a number of standard parse error codes. These error codes only cover
+the "tokenization" stage of parsing HTML. The parse errors in the
+"tree construction" stage do not have standardized error codes (yet).
+
+As a convenience to Nokogumbo users, the defined error codes are available
+via the
+[`Nokogiri::XML::SyntaxError#str1`](https://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SyntaxError#str1-instance_method)
+method.
+
+```ruby
 require 'nokogumbo'
-Nokogiri::HTML5.parse(Nokogiri::HTML5.parse('<div></div> a').to_html).to_html
+doc = Nokogiri::HTML5.parse('<span/>Hi there!</span foo=bar />', max_errors: 10)
+doc.errors.each do |err|
+  puts("#{err.line}:#{err.column}: #{err.str1}")
+end
+```
+
+This prints the following.
+```
+1:1: generic-parser
+1:1: non-void-html-element-start-tag-with-trailing-solidus
+1:17: end-tag-with-trailing-solidus
+1:17: end-tag-with-attributes
+```
+
+Note that the first error is `generic-parser` because it's an error from the
+tree construction stage and doesn't have a standardized error code.
+
+For the purposes of semantic versioning, the error messages, error locations,
+and error codes are not part of Nokogumbo's public API. That is, these are
+subject to change without Nokogumbo's major version number changing. These may
+be stabilized in the future.
+
+### Maximum tree depth
+The maximum depth of the DOM tree parsed by the various parsing methods is
+configurable by the `:max_tree_depth` option. If the depth of the tree would
+exceed this limit, then an
+[ArgumentError](https://ruby-doc.org/core-2.5.0/ArgumentError.html) is thrown.
+
+This limit (which defaults to `Nokogumbo::DEFAULT_MAX_TREE_DEPTH = 400`) can
+be removed by giving the option `max_tree_depth: -1`.
+
+``` ruby
+html = '<!DOCTYPE html>' + '<div>' * 1000
+doc = Nokogiri.HTML5(html)
+# raises ArgumentError: Document tree depth limit exceeded
+doc = Nokogiri.HTML5(html, max_tree_depth: -1)
 ```
 
-Notes
------
+### Attribute limit per element
+The maximum number of attributes per DOM element is configurable by the
+`:max_attributes` option. If a given element would exceed this limit, then an
+[ArgumentError](https://ruby-doc.org/core-2.5.0/ArgumentError.html) is thrown.
+
+This limit (which defaults to `Nokogumbo::DEFAULT_MAX_ATTRIBUTES = 400`) can
+be removed by giving the option `max_attributes: -1`.
+
+``` ruby
+html = '<!DOCTYPE html><div ' + (1..1000).map { |x| "attr-#{x}" }.join(' ') + '>'
+# "<!DOCTYPE html><div attr-1 attr-2 attr-3 ... attr-1000>"
+doc = Nokogiri.HTML5(html)
+# raises ArgumentError: Attributes per element limit exceeded
+doc = Nokogiri.HTML5(html, max_attributes: -1)
+```
+
+## HTML Serialization
+
+After parsing HTML, it may be serialized using any of the Nokogiri
+[serialization
+methods](https://www.rubydoc.info/gems/nokogiri/Nokogiri/XML/Node). In
+particular, `#serialize`, `#to_html`, and `#to_s` will serialize a given node
+and its children. (This is the equivalent of JavaScript's
+`Element.outerHTML`.) Similarly, `#inner_html` will serialize the children of
+a given node. (This is the equivalent of JavaScript's `Element.innerHTML`.)
+
+``` ruby
+doc = Nokogiri::HTML5("<!DOCTYPE html><span>Hello world!</span>")
+puts doc.serialize
+# Prints: <!DOCTYPE html><html><head></head><body><span>Hello world!</span></body></html>
+```
+
+Due to quirks in how HTML is parsed and serialized, it's possible for a DOM
+tree to be serialized and then re-parsed, resulting in a different DOM.
+Mostly, this happens with DOMs produced from invalid HTML. Unfortunately, even
+valid HTML may not survive serialization and re-parsing.
+
+In particular, a newline at the start of `pre`, `listing`, and `textarea`
+elements is ignored by the parser.
+
+``` ruby
+doc = Nokogiri::HTML5(<<-EOF)
+<!DOCTYPE html>
+<pre>
+Content</pre>
+EOF
+puts doc.at('/html/body/pre').serialize
+# Prints: <pre>Content</pre>
+```
+
+In this case, the original HTML is semantically equivalent to the serialized
+version. If the `pre`, `listing`, or `textarea` content starts with two
+newlines, the first newline will be stripped on the first parse and the second
+newline will be stripped on the second, leading to semantically different
+DOMs. Passing the parameter `preserve_newline: true` will cause two or more
+newlines to be preserved. (A single leading newline will still be removed.)
+
+``` ruby
+doc = Nokogiri::HTML5(<<-EOF)
+<!DOCTYPE html>
+<listing>
+
+Content</listing>
+EOF
+puts doc.at('/html/body/listing').serialize(preserve_newline: true)
+# Prints: <listing>
+#
+# Content</listing>
+```
+
+## Encodings
+Nokogumbo always parses HTML using
+[UTF-8](https://en.wikipedia.org/wiki/UTF-8); however, the encoding of the
+input can be explicitly selected via the optional `encoding` parameter. This
+is most useful when the input comes not from a string but from an IO object.
+
+When serializing a document or node, the encoding of the output string can be
+specified via the `:encoding` options. Characters that cannot be encoded in
+the selected encoding will be encoded as [HTML numeric
+entities](https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references).
+
+``` ruby
+frag = Nokogiri::HTML5.fragment('<span>아는 길도 물어가라</span>')
+html = frag.serialize(encoding: 'US-ASCII')
+puts html
+# Prints: <span>&#xc544;&#xb294; &#xae38;&#xb3c4; &#xbb3c;&#xc5b4;&#xac00;&#xb77c;</span>
+frag = Nokogiri::HTML5.fragment(html)
+puts frag.serialize
+# Prints: <span>아는 길도 물어가라</span>
+```
+
+(There's a [bug](https://bugs.ruby-lang.org/issues/15033) in all current
+versions of Ruby that can cause the entity encoding to fail. Of the mandated
+supported encodings for HTML, the only encoding I'm aware of that has this bug
+is `'ISO-2022-JP'`. I recommend avoiding this encoding.)
+
+## Examples
+```ruby
+require 'nokogumbo'
+puts Nokogiri::HTML5.get('http://nokogiri.org').search('ol li')[2].text
+```
+
+## Notes
 
 * The `Nokogiri::HTML5.fragment` function takes a string and parses it
   as a HTML5 document.  The `<html>`, `<head>`, and `<body>` elements are
@@ -74,20 +258,47 @@ rules defined in the HTML5 specification for doing so.
 * Instead of returning `unknown` as the element name for unknown tags, the
 original tag name is returned verbatim.
 
-* If the Gumbo HTML5 parser is not already installed, the source for the
-parser will be downloaded and compiled into the Gem itself.
+# Flavors of Nokogumbo
+Nokogumbo uses libxml2, the XML library underlying Nokogiri, to speed up
+parsing. If the libxml2 headers are not available, then Nokogumbo resorts to
+using Nokogiri's Ruby API to construct the DOM tree.
+
+Nokogiri can be configured to either use the system library version of libxml2
+or use a bundled version. By default (as of Nokogiri version 1.8.4), Nokogiri
+will use a bundled version.
+
+To prevent differences between versions of libxml2, Nokogumbo will only use
+libxml2 if the build process can find the exact same version used by Nokogiri.
+This leads to three possibilities
+
+1. Nokogiri is compiled with the bundled libxml2. In this case, Nokogumbo will
+   (by default) use the same version of libxml2.
+2. Nokogiri is compiled with the system libxml2. In this case, if the libxml2
+   headers are available, then Nokogumbo will (by default) use the system
+   version and headers.
+3. Nokogiri is compiled with the system libxml2 but its headers aren't
+   available at build time for Nokogumbo. In this case, Nokogumbo will use the
+   slower Ruby API.
+
+Using libxml2 can be required by passing `-- --with-libxml2` to `bundle exec
+rake` or to `gem install`. Using libxml2 can be prohibited by instead passing
+`-- --without-libxml2`.
+
+Functionally, the only difference between using libxml2 or not is in the
+behavior of `Nokogiri::XML::Node#line`. If it is used, then `#line` will
+return the line number of the corresponding node. Otherwise, it will return 0.
 
-Installation
-============
+# Installation
 
-    git clone --recursive https://github.com/rubys/nokogumbo.git
+    git clone https://github.com/rubys/nokogumbo.git
     cd nokogumbo
     bundle install
     rake gem
     gem install pkg/nokogumbo*.gem
 
-Related efforts
-============
+# Related efforts
 
-* [ruby-gumbo](https://github.com/nevir/ruby-gumbo#readme) - a ruby binding
-for the Gumbo HTML5 parser.
+* [ruby-gumbo](https://github.com/nevir/ruby-gumbo#readme) -- a ruby binding
+  for the Gumbo HTML5 parser.
+* [lua-gumbo](https://gitlab.com/craigbarnes/lua-gumbo) -- a lua binding for
+  the Gumbo HTML5 parser.

data/ext/nokogumbo/extconf.rb

@@ -0,0 +1,144 @@
+require 'rubygems'
+require 'fileutils'
+require 'mkmf'
+require 'nokogiri'
+
+$CFLAGS += " -std=c99"
+$LDFLAGS.gsub!('-Wl,--no-undefined', '')
+$DLDFLAGS.gsub!('-Wl,--no-undefined', '')
+$warnflags = CONFIG['warnflags'] = '-Wall'
+
+NG_SPEC = Gem::Specification.find_by_name('nokogiri', "= #{Nokogiri::VERSION}")
+
+def download_headers
+  begin
+    require 'yaml'
+
+    dependencies = YAML.load_file(File.join(NG_SPEC.gem_dir, 'dependencies.yml'))
+    version = dependencies['libxml2']['version']
+    host = RbConfig::CONFIG["host_alias"].empty? ? RbConfig::CONFIG["host"] : RbConfig::CONFIG["host_alias"]
+    path = File.join('ports', host, 'libxml2', version, 'include/libxml2')
+    return path if File.directory?(path)
+
+    # Make sure we're using the same version Nokogiri uses
+    dep_index = NG_SPEC.dependencies.index { |dep| dep.name == 'mini_portile2' and dep.type == :runtime }
+    return nil if dep_index.nil?
+    requirement = NG_SPEC.dependencies[dep_index].requirement.to_s
+
+    gem 'mini_portile2', requirement
+    require 'mini_portile2'
+    p = MiniPortile::new('libxml2', version).tap do |r|
+      r.host = RbConfig::CONFIG["host_alias"].empty? ? RbConfig::CONFIG["host"] : RbConfig::CONFIG["host_alias"]
+      r.files = [{
+        url: "http://xmlsoft.org/sources/libxml2-#{r.version}.tar.gz",
+        sha256: dependencies['libxml2']['sha256']
+      }]
+      r.configure_options += [
+        "--without-python",
+        "--without-readline",
+        "--with-c14n",
+        "--with-debug",
+        "--with-threads"
+      ]
+    end
+    p.download unless p.downloaded?
+    p.extract
+    p.configure unless p.configured?
+    system('make', '-C', "tmp/#{p.host}/ports/libxml2/#{version}/libxml2-#{version}/include/libxml", 'install-xmlincHEADERS')
+    path
+  rescue
+    puts 'failed to download/install headers'
+    nil
+  end
+end
+
+required = arg_config('--with-libxml2')
+prohibited = arg_config('--without-libxml2')
+if required and prohibited
+  abort "cannot use both --with-libxml2 and --without-libxml2"
+end
+
+have_libxml2 = false
+have_ng = false
+
+def windows?
+  ::RUBY_PLATFORM =~ /mingw|mswin/
+end
+
+def modern_nokogiri?
+  nokogiri_version = Gem::Version.new(Nokogiri::VERSION)
+  requirement = windows? ? ">= 1.11.2" : ">= 1.11.0.rc4"
+  Gem::Requirement.new(requirement).satisfied_by?(nokogiri_version)
+end
+
+if !prohibited
+  if modern_nokogiri?
+    append_cflags(Nokogiri::VERSION_INFO["nokogiri"]["cppflags"])
+    append_ldflags(Nokogiri::VERSION_INFO["nokogiri"]["ldflags"]) # may be nil for nokogiri pre-1.11.2
+    have_libxml2 = if Nokogiri::VERSION_INFO["nokogiri"]["ldflags"].empty?
+                     have_header('libxml/tree.h')
+                   else
+                     have_func("xmlNewDoc", "libxml/tree.h")
+                   end
+  end
+
+  if !have_libxml2
+    if Nokogiri::VERSION_INFO.include?('libxml') and
+       Nokogiri::VERSION_INFO['libxml']['source'] == 'packaged'
+      # Nokogiri has libxml2 built in. Find the headers.
+      libxml2_path = File.join(Nokogiri::VERSION_INFO['libxml']['libxml2_path'],
+                               'include/libxml2')
+      if find_header('libxml/tree.h', libxml2_path)
+        have_libxml2 = true
+      else
+        # Unfortunately, some versions of Nokogiri delete these files.
+        # https://github.com/sparklemotion/nokogiri/pull/1788
+        # Try to download them
+        libxml2_path = download_headers
+        unless libxml2_path.nil?
+          have_libxml2 = find_header('libxml/tree.h', libxml2_path)
+        end
+      end
+    else
+      # Nokogiri is compiled with system headers.
+      # Hack to work around broken mkmf on macOS
+      # (https://bugs.ruby-lang.org/issues/14992 fixed now)
+      if RbConfig::MAKEFILE_CONFIG['LIBPATHENV'] == 'DYLD_LIBRARY_PATH'
+        RbConfig::MAKEFILE_CONFIG['LIBPATHENV'] = 'DYLD_FALLBACK_LIBRARY_PATH'
+      end
+
+      pkg_config('libxml-2.0')
+      have_libxml2 = have_library('xml2', 'xmlNewDoc')
+    end
+  end
+
+  if required and !have_libxml2
+    abort "libxml2 required but could not be located"
+  end
+
+
+  if have_libxml2
+    have_ng = have_header('nokogiri.h') || find_header('nokogiri.h', File.join(NG_SPEC.gem_dir, 'ext/nokogiri'))
+  end
+end
+
+if have_libxml2 and have_ng
+  $CFLAGS += " -DNGLIB=1"
+end
+
+# Symlink gumbo-parser source files.
+ext_dir = File.dirname(__FILE__)
+
+Dir.chdir(ext_dir) do
+  $srcs = Dir['*.c', '../../gumbo-parser/src/*.c']
+  $hdrs = Dir['*.h', '../../gumbo-parser/src/*.h']
+end
+$INCFLAGS << ' -I$(srcdir)/../../gumbo-parser/src'
+$VPATH << '$(srcdir)/../../gumbo-parser/src'
+
+create_makefile('nokogumbo/nokogumbo') do |conf|
+  conf.map! do |chunk|
+    chunk.gsub(/^HDRS = .*$/, "HDRS = #{$hdrs.map { |h| File.join('$(srcdir)', h)}.join(' ')}")
+  end
+end
+# vim: set sw=2 sts=2 ts=8 et: