nokogiri 1.10.10-java → 1.11.0-java

data/README.md

@@ -1,105 +1,153 @@
+<div><img src="https://nokogiri.org/images/nokogiri-serif-black.png" align="right"/></div>
+
 # Nokogiri
 
-## Description
+Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby. It provides a sensible, easy-to-understand API for reading, writing, modifying, and querying documents. It is fast and standards-compliant by relying on native parsers like libxml2 (C) and xerces (Java).
+
+## Guiding Principles
+
+Some guiding principles Nokogiri tries to follow:
 
-Nokogiri (鋸) is an HTML, XML, SAX, and Reader parser.  Among
-Nokogiri's many features is the ability to search documents via XPath
-or CSS3 selectors.
+- be secure-by-default by treating all documents as **untrusted** by default
+- be a **thin-as-reasonable layer** on top of the underlying parsers, and don't attempt to fix behavioral differences between the parsers
 
 
-## Links
+## Features Overview
 
-* https://nokogiri.org
-* [Installation Help](https://nokogiri.org/tutorials/installing_nokogiri.html)
-* [Tutorials](https://nokogiri.org)
-* [Cheat Sheet](https://github.com/sparklemotion/nokogiri/wiki/Cheat-sheet)
-* [GitHub](https://github.com/sparklemotion/nokogiri)
-* [Mailing List](https://groups.google.com/group/nokogiri-talk)
-* [Chat/Gitter](https://gitter.im/sparklemotion/nokogiri)
+- DOM Parser for XML and HTML4
+- SAX Parser for XML and HTML4
+- Push Parser for XML and HTML4
+- Document search via XPath 1.0
+- Document search via CSS3 selectors, with some jquery-like extensions
+- XSD Schema validation
+- XSLT transformation
+- "Builder" DSL for XML and HTML documents
 
 
 ## Status
 
-[![Concourse CI](https://ci.nokogiri.org/api/v1/teams/nokogiri-core/pipelines/nokogiri/jobs/ruby-2.4-system/badge)](https://ci.nokogiri.org/teams/nokogiri-core/pipelines/nokogiri)
+[![Concourse CI](https://ci.nokogiri.org/api/v1/teams/nokogiri-core/pipelines/nokogiri/jobs/cruby-2.7/badge)](https://ci.nokogiri.org/teams/nokogiri-core/pipelines/nokogiri)
 [![Appveyor CI](https://ci.appveyor.com/api/projects/status/xj2pqwvlxwuwgr06/branch/master?svg=true)](https://ci.appveyor.com/project/flavorjones/nokogiri/branch/master)
 [![Code Climate](https://codeclimate.com/github/sparklemotion/nokogiri.svg)](https://codeclimate.com/github/sparklemotion/nokogiri)
 [![Test Coverage](https://api.codeclimate.com/v1/badges/59c67b0e8976027a45ad/test_coverage)](https://codeclimate.com/github/sparklemotion/nokogiri/test_coverage)
 
 [![Gem Version](https://badge.fury.io/rb/nokogiri.svg)](https://rubygems.org/gems/nokogiri)
-[![SemVer compatibility](https://api.dependabot.com/badges/compatibility_score?dependency-name=nokogiri&package-manager=bundler&version-scheme=semver)](https://dependabot.com/compatibility-score.html?dependency-name=nokogiri&package-manager=bundler&version-scheme=semver)
-[![Tidelift dependencies](https://tidelift.com/badges/github/sparklemotion/nokogiri)](https://tidelift.com/subscription/pkg/rubygems-nokogiri?utm_source=rubygems-nokogiri&utm_medium=referral&utm_campaign=readme)
+[![SemVer compatibility](https://api.dependabot.com/badges/compatibility_score?dependency-name=nokogiri&package-manager=bundler&version-scheme=semver)](https://dependabot.com/compatibility-score/?dependency-name=nokogiri&package-manager=bundler)
+[![Tidelift dependencies](https://tidelift.com/badges/package/rubygems/nokogiri)](https://tidelift.com/subscription/pkg/rubygems-nokogiri?utm_source=rubygems-nokogiri&utm_medium=referral&utm_campaign=readme)
 
 
-## Features
+## Support and Help
 
-* XML/HTML DOM parser which handles broken HTML
-* XML/HTML SAX parser
-* XML/HTML Push parser
-* XPath 1.0 support for document searching
-* CSS3 selector support for document searching
-* XML/HTML builder
-* XSLT transformer
+All official documentation is posted at https://nokogiri.org (the source for which is at https://github.com/sparklemotion/nokogiri.org/, and we welcome contributions).
 
-Nokogiri parses and searches XML/HTML using native libraries (either C
-or Java, depending on your Ruby), which means it's fast and
-standards-compliant.
+### Reading
 
+Your first stops for API documentation should be:
 
-## Installation
+- RDocs: https://nokogiri.org/rdoc/index.html
+- An excellent community-maintained [Cheat Sheet](https://github.com/sparklemotion/nokogiri/wiki/Cheat-sheet)
 
-If this doesn't work:
 
-```
-gem install nokogiri
-```
+### Questions
 
-then please start troubleshooting here:
+If you'd like to talk to a human:
 
-> https://nokogiri.org/tutorials/installing_nokogiri.html
+- The Discord chat channel is `#nokogiri-💎` at https://discord.gg/UyQnKrT
+- The Gitter chat channel is https://gitter.im/sparklemotion/nokogiri
+- The IRC chat channel is `#nokogiri` on freenode.
+- The Nokogiri mailing list is active at https://groups.google.com/group/nokogiri-talk
+- The Nokogiri bug tracker is at https://github.com/sparklemotion/nokogiri/issues
 
-There are currently 1,237 Stack Overflow questions about Nokogiri
-installation. The vast majority of them are out of date and therefore
-incorrect. __Please do not use Stack Overflow.__
+Consider subscribing to [Tidelift][tidelift] which provides license assurances and timely security notifications for your open source dependencies, including Nokogiri. [Tidelift][tidelift] subscriptions also help the Nokogiri maintainers fund our [automated testing](https://ci.nokogiri.org) which in turn allows us to ship releases, bugfixes, and security updates more often.
 
-Instead, [tell us](https://nokogiri.org/tutorials/getting_help.html)
-when the above instructions don't work for you. This allows us to both
-help you directly and improve the documentation.
+  [tidelift]: https://tidelift.com/subscription/pkg/rubygems-nokogiri?utm_source=rubygems-nokogiri&utm_medium=referral&utm_campaign=readme
 
 
-### Binary packages
+### Security and Vulnerability Reporting
 
-Binary packages are available for some distributions.
+Please report vulnerabilities at https://hackerone.com/nokogiri
 
-* Debian: https://packages.debian.org/sid/ruby-nokogiri
-* SuSE: https://download.opensuse.org/repositories/devel:/languages:/ruby:/extensions/
-* Fedora: http://s390.koji.fedoraproject.org/koji/packageinfo?packageID=6756
+Full information and description of our security policy is in [`SECURITY.md`](SECURITY.md)
 
 
-## Support
+### Semantic Versioning Policy
 
-All official documentation is posted at https://nokogiri.org (the source for which is at https://github.com/sparklemotion/nokogiri.org/, and we welcome contributions).
+Nokogiri follows [Semantic Versioning](https://semver.org/) (since 2017 or so). [![Dependabot's SemVer compatibility score for Nokogiri](https://api.dependabot.com/badges/compatibility_score?dependency-name=nokogiri&package-manager=bundler&version-scheme=semver)](https://dependabot.com/compatibility-score/?dependency-name=nokogiri&package-manager=bundler)
 
-* The Nokogiri mailing list is active: https://groups.google.com/group/nokogiri-talk
-* The Nokogiri bug tracker is here: https://github.com/sparklemotion/nokogiri/issues
-* Before filing a bug report, please read our submission guidelines: http://nokogiri.org/tutorials/getting_help.html
-* The IRC channel is `#nokogiri` on freenode.
-* The project's GitHub wiki has an excellent community-maintained [Cheat Sheet](https://github.com/sparklemotion/nokogiri/wiki/Cheat-sheet) which might be useful.
+We bump `Major.Minor.Patch` versions following this guidance:
 
-Consider subscribing to [Tidelift][tidelift] which provides license assurances and timely security notifications for your open source dependencies, including Nokogiri. [Tidelift][tidelift] subscriptions also help the Nokogiri maintainers fund our [automated testing](https://ci.nokogiri.org) which in turn allows us to ship releases, bugfixes, and security updates more often.
+`Major`: (we've never done this)
 
-  [tidelift]: https://tidelift.com/subscription/pkg/rubygems-nokogiri?utm_source=rubygems-nokogiri&utm_medium=referral&utm_campaign=readme
+- Significant backwards-incompatible changes to the public API that would require rewriting existing application code.
+- Some examples of backwards-incompatible changes we might someday consider for a Major release are at [`ROADMAP.md`](ROADMAP.md).
 
+`Minor`:
 
-## Security and Vulnerability Reporting
+- Features and bugfixes.
+- Updating packaged libraries for non-security-related reasons.
+- Dropping support for EOLed Ruby versions. [Some folks find this objectionable](https://github.com/sparklemotion/nokogiri/issues/1568), but [SemVer says this is OK if the public API hasn't changed](https://semver.org/#what-should-i-do-if-i-update-my-own-dependencies-without-changing-the-public-api).
+- Backwards-incompatible changes to internal or private methods and constants. These are detailed in the "Changes" section of each changelog entry.
 
-Please report vulnerabilities at https://hackerone.com/nokogiri
+`Patch`:
 
-Full information and description of our security policy is in [`SECURITY.md`](SECURITY.md)
+- Bugfixes.
+- Security updates.
+- Updating packaged libraries for security-related reasons.
 
 
-## Synopsis
+## Installation
+
+Requirements:
+
+- Ruby >= 2.5
+- JRuby >= 9.2.0.0
+
+
+### Native Gems: Faster, more reliable installation
+
+"Native gems" contain pre-compiled libraries for a specific machine architecture. On supported platforms, this removes the need for compiling the C extension and the packaged libraries, or for system dependencies to exist. This results in **much faster installation** and **more reliable installation**, which as you probably know are the biggest headaches for Nokogiri users.
+
+### Supported Platforms
+
+As of v1.11.0, Nokogiri ships pre-compiled, "native" gems for the following platforms:
+
+- Linux: `x86-linux` and `x86_64-linux` (req: `glibc >= 2.17`), including musl platforms like Alpine
+- Darwin/MacOS: `x86_64-darwin` and `arm64-darwin`
+- Windows: `x86-mingw32` and `x64-mingw32`
+- Java: any platform running JRuby 9.2 or higher
+
+To determine whether your system supports one of these gems, look at the output of `bundle platform` or `ruby -e 'puts Gem::Platform.local.to_s'`.
+
+If you're on a supported platform, either `gem install` or `bundle install` should install a native gem without any additional action on your part. This installation should only take a few seconds, and your output should look something like:
+
+``` sh
+$ gem install nokogiri
+Fetching nokogiri-1.11.0-x86_64-linux.gem
+Successfully installed nokogiri-1.11.0-x86_64-linux
+1 gem installed
+```
+
+
+### Other Installation Options
+
+Because Nokogiri is a C extension, it requires that you have a C compiler toolchain, Ruby development header files, and some system dependencies installed.
+
+The following may work for you if you have an appropriately-configured system:
+
+``` bash
+gem install nokogiri
+```
 
-Nokogiri is a large library, but here is example usage for parsing and examining a document:
+If you have any issues, please visit [Installing Nokogiri](https://nokogiri.org/tutorials/installing_nokogiri.html) for more complete instructions and troubleshooting.
+
+
+## How To Use Nokogiri
+
+Nokogiri is a large library, and so it's challenging to briefly summarize it. We've tried to provide long, real-world examples at [Tutorials](https://nokogiri.org/tutorials/toc.html).
+
+### Parsing and Querying
+
+Here is example usage for parsing and querying a document:
 
 ```ruby
 #! /usr/bin/env ruby
@@ -108,51 +156,26 @@ require 'nokogiri'
 require 'open-uri'
 
 # Fetch and parse HTML document
-doc = Nokogiri::HTML(open('https://nokogiri.org/tutorials/installing_nokogiri.html'))
+doc = Nokogiri::HTML(URI.open('https://nokogiri.org/tutorials/installing_nokogiri.html'))
 
-puts "### Search for nodes by css"
+# Search for nodes by css
 doc.css('nav ul.menu li a', 'article h2').each do |link|
   puts link.content
 end
 
-puts "### Search for nodes by xpath"
+# Search for nodes by xpath
 doc.xpath('//nav//ul//li/a', '//article//h2').each do |link|
   puts link.content
 end
 
-puts "### Or mix and match."
+# Or mix and match
 doc.search('nav ul.menu li a', '//article//h2').each do |link|
   puts link.content
 end
 ```
 
 
-## Requirements
-
-* Ruby 2.3.0 or higher, including any development packages necessary
-  to compile native extensions.
-
-* In Nokogiri 1.6.0 and later libxml2 and libxslt are bundled with the
-  gem, but if you want to use the system versions:
-
-  * First, check out [the long list](http://www.xmlsoft.org/news.html)
-    of fixes and changes between releases before deciding to use any
-    version older than is bundled with Nokogiri.
-
-  * At install time, set the environment variable
-    `NOKOGIRI_USE_SYSTEM_LIBRARIES` or else use the
-    `--use-system-libraries` argument. (See
-    https://nokogiri.org/tutorials/installing_nokogiri.html#install-with-system-libraries
-    for specifics.)
-
-  * libxml2 >=2.6.21 with iconv support
-    (libxml2-dev/-devel is also required)
-
-  * libxslt, built with and supported by the given libxml2
-    (libxslt-dev/-devel is also required)
-
-
-## Encoding
+### Encoding
 
 Strings are always stored as UTF-8 internally.  Methods that return
 text values will always return UTF-8 encoded strings.  Methods that
@@ -178,11 +201,43 @@ explicitly setting the encoding to EUC-JP on the parser:
 ```
 
 
-## Development
+## Technical Overview
+
+### Guiding Principles
+
+As noted above, two guiding principles of the software are:
+
+- be secure-by-default by treating all documents as **untrusted** by default
+- be a **thin-as-reasonable layer** on top of the underlying parsers, and don't attempt to fix behavioral differences between the parsers
+
+Notably, despite all parsers being standards-compliant, there are behavioral inconsistencies between the parsers used in the CRuby and JRuby implementations, and Nokogiri does not and should not attempt to remove these inconsistencies. Instead, we surface these differences in the test suite when they are important/semantic; or we intentionally write tests to depend only on the important/semantic bits (omitting whitespace from regex matchers on results, for example).
+
+
+### CRuby
+
+The Ruby (a.k.a., CRuby, MRI, YARV) implementation is a C extension that depends on libxml2 and libxslt (which in turn depend on zlib and possibly libiconv).
+
+These dependencies are met by default by Nokogiri's packaged versions of the libxml2 and libxslt source code, but a configuration option `--use-system-libraries` is provided to allow specification of alternative library locations. See [Installing Nokogiri](https://nokogiri.org/tutorials/installing_nokogiri.html) for full documentation.
+
+We provide native gems by pre-compiling libxml2 and libxslt (and potentially zlib and libiconv) and packaging them into the gem file. In this case, no compilation is necessary at installation time, which leads to faster and more reliable installation.
+
+See [`LICENSE-DEPENDENCIES.md`](LICENSE-DEPENDENCIES.md) for more information on which dependencies are provided in which native and source gems.
+
+
+### JRuby
+
+The Java (a.k.a. JRuby) implementation is a Java extension that depends primarily on Xerces and NekoHTML for parsing, though additional dependencies are on `isorelax`, `nekodtd`, `jing`, `serializer`, `xalan-j`, and `xml-apis`.
+
+These dependencies are provided by pre-compiled jar files packaged in the `java` platform gem.
+
+See [`LICENSE-DEPENDENCIES.md`](LICENSE-DEPENDENCIES.md) for more information on which dependencies are provided in which native and source gems.
+
+
+## Contributing
 
 ```bash
-  bundle install
-  bundle exec rake compile test
+bundle install
+bundle exec rake compile test
 ```
 
 
@@ -196,3 +251,22 @@ We've adopted the Contributor Covenant code of conduct, which you can read in fu
 This project is licensed under the terms of the MIT license.
 
 See this license at [`LICENSE.md`](LICENSE.md).
+
+
+### Dependencies
+
+Some additional libraries may be distributed with your version of Nokogiri. Please see [`LICENSE-DEPENDENCIES.md`](LICENSE-DEPENDENCIES.md) for a discussion of the variations as well as the licenses thereof.
+
+
+## Authors
+
+- Mike Dalessio
+- Aaron Patterson
+- Yoko Harada
+- Akinori MUSHA
+- John Shahid
+- Karol Bucek
+- Lars Kanis
+- Sergio Arbeo
+- Timothy Elliott
+- Nobuyoshi Nakada

data/ext/java/nokogiri/HtmlDocument.java

@@ -36,7 +36,6 @@ import org.jruby.Ruby;
 import org.jruby.RubyClass;
 import org.jruby.anno.JRubyClass;
 import org.jruby.anno.JRubyMethod;
-import org.jruby.runtime.Arity;
 import org.jruby.runtime.Helpers;
 import org.jruby.runtime.ThreadContext;
 import org.jruby.runtime.builtin.IRubyObject;
@@ -48,6 +47,8 @@ import org.w3c.dom.NodeList;
 
 import nokogiri.internals.HtmlDomParserContext;
 
+import static nokogiri.internals.NokogiriHelpers.getNokogiriClass;
+
 /**
  * Class for Nokogiri::HTML::Document.
  *
@@ -65,21 +66,25 @@ public class HtmlDocument extends XmlDocument {
     public HtmlDocument(Ruby ruby, RubyClass klazz) {
         super(ruby, klazz);
     }
-    
+
+    public HtmlDocument(Ruby runtime, Document document) {
+        this(runtime, getNokogiriClass(runtime, "Nokogiri::XML::Document"), document);
+    }
+
     public HtmlDocument(Ruby ruby, RubyClass klazz, Document doc) {
         super(ruby, klazz, doc);
     }
 
     @JRubyMethod(name="new", meta = true, rest = true, required=0)
-    public static IRubyObject rbNew(ThreadContext context, IRubyObject klazz,
-                                    IRubyObject[] args) {
+    public static IRubyObject rbNew(ThreadContext context, IRubyObject klazz, IRubyObject[] args) {
+        final Ruby runtime = context.runtime;
         HtmlDocument htmlDocument;
         try {
-            Document docNode = createNewDocument();
-            htmlDocument = (HtmlDocument) NokogiriService.HTML_DOCUMENT_ALLOCATOR.allocate(context.getRuntime(), (RubyClass) klazz);
-            htmlDocument.setDocumentNode(context, docNode);
+            Document docNode = createNewDocument(runtime);
+            htmlDocument = (HtmlDocument) NokogiriService.HTML_DOCUMENT_ALLOCATOR.allocate(runtime, (RubyClass) klazz);
+            htmlDocument.setDocumentNode(context.runtime, docNode);
         } catch (Exception ex) {
-            throw context.getRuntime().newRuntimeError("couldn't create document: " + ex);
+            throw asRuntimeError(runtime, "couldn't create document: ", ex);
         }
 
         Helpers.invoke(context, htmlDocument, "initialize", args);
@@ -109,46 +114,29 @@ public class HtmlDocument extends XmlDocument {
         return internalSubset;
     }
 
-    public static IRubyObject do_parse(ThreadContext context,
-                                       IRubyObject klass,
-                                       IRubyObject[] args) {
-        Ruby ruby = context.getRuntime();
-        Arity.checkArgumentCount(ruby, args, 4, 4);
-        HtmlDomParserContext ctx =
-            new HtmlDomParserContext(ruby, args[2], args[3]);
-        ctx.setInputSource(context, args[0], args[1]);
-        return ctx.parse(context, klass, args[1]);
-    }
-    
-    public void setDocumentNode(ThreadContext context, Node node) {
-        super.setNode(context, node);
-        Ruby runtime = context.getRuntime();
-        if (node != null) {
-            Document document = (Document)node;
-            document.normalize();
-            stabilzeAttrValue(document.getDocumentElement());
-        }
+    @Override
+    void init(Ruby runtime, Document document) {
+        stabilizeTextContent(document);
+        document.normalize();
         setInstanceVariable("@decorators", runtime.getNil());
+        if (document.getDocumentElement() != null) {
+            stabilizeAttrs(document.getDocumentElement());
+        }
     }
-    
-    private void stabilzeAttrValue(Node node) {
-        if (node == null) return;
+
+    private static void stabilizeAttrs(Node node) {
         if (node.hasAttributes()) {
             NamedNodeMap nodeMap = node.getAttributes();
             for (int i=0; i<nodeMap.getLength(); i++) {
                 Node n = nodeMap.item(i);
                 if (n instanceof Attr) {
-                    Attr attr = (Attr)n;
-                    String attrName = attr.getName();
-                    // not sure, but need to get value always before document is referred.
-                    // or lose attribute value
-                    String attrValue = attr.getValue(); // don't delete this line
+                    stabilizeAttr((Attr) n);
                 }
             }
         }
         NodeList children = node.getChildNodes();
         for (int i=0; i<children.getLength(); i++) {
-            stabilzeAttrValue(children.item(i));
+            stabilizeAttrs(children.item(i));
         }
     }
     
@@ -167,11 +155,11 @@ public class HtmlDocument extends XmlDocument {
      * Read the HTML document from +io+ with given +url+, +encoding+,
      * and +options+.  See Nokogiri::HTML.parse
      */
-    @JRubyMethod(meta = true, rest = true)
-    public static IRubyObject read_io(ThreadContext context,
-                                      IRubyObject cls,
-                                      IRubyObject[] args) {
-        return do_parse(context, cls, args);
+    @JRubyMethod(meta = true, required = 4)
+    public static IRubyObject read_io(ThreadContext context, IRubyObject klass, IRubyObject[] args) {
+        HtmlDomParserContext ctx = new HtmlDomParserContext(context.runtime, args[2], args[3]);
+        ctx.setIOInputSource(context, args[0], args[1]);
+        return ctx.parse(context, (RubyClass) klass, args[1]);
     }
 
     /*
@@ -181,10 +169,10 @@ public class HtmlDocument extends XmlDocument {
      * Read the HTML document contained in +string+ with given +url+, +encoding+,
      * and +options+.  See Nokogiri::HTML.parse
      */
-    @JRubyMethod(meta = true, rest = true)
-    public static IRubyObject read_memory(ThreadContext context,
-                                          IRubyObject cls,
-                                          IRubyObject[] args) {
-        return do_parse(context, cls, args);
+    @JRubyMethod(meta = true, required = 4)
+    public static IRubyObject read_memory(ThreadContext context, IRubyObject klass, IRubyObject[] args) {
+        HtmlDomParserContext ctx = new HtmlDomParserContext(context.runtime, args[2], args[3]);
+        ctx.setStringInputSource(context, args[0], args[1]);
+        return ctx.parse(context, (RubyClass) klass, args[1]);
     }
 }