webtranslateit-hpricot 0.9.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b2c7f0e599b62be02967d46819ec60c457e8e7c2207752ef328d069d3ca3d627
-  data.tar.gz: 75205569719178f6b699114f54a042d491b4a3fc16248ace4a5f171460a720cf
+  metadata.gz: 23b27a1746fc1bbaf18618228aa7355bb333c6b6b7bef6ba3baa228c096307bc
+  data.tar.gz: 79c391d5c90b510b969557372e9c944fca144b3bc42241908f894ed20dcbea70
 SHA512:
-  metadata.gz: f7a5c3f9770659390d82c477c9ec45968c560e8c878399708a4a23a93c7a61fef9b46eaea6c0a0fff12b62ffbc318e8b03fd34c8095cb3687445badbc02999a5
-  data.tar.gz: 85afd161d4358033e9b4e32c69c35af264f88599849fc0022444f41da2cc91dade9d9949e07bf9e36980e3b613d1b5d92f1cb17253660f57b44205d9d5d16164
+  metadata.gz: 269e7a6e9f389c3045a5ee1f5eb824a5e99db52896b60b210d55ae51b66ac5894cebb2b3926837c3303ee6aec841289730f7e9e0ebfd653e06e94ecc0355f9ab
+  data.tar.gz: f0ac2d5cb9d96a7d2929e73d2ca94236fa067f1bd13832fb636eaad4638e73cac614f25ee5891cdf6bc8153545a8efea032994437adc7f065a41376e32ef1e96

data/CHANGELOG

@@ -1,3 +1,109 @@
+= 1.0.0
+=== 9 September 2026
+
+This release replaces the C extension with pure Ruby. No native code is
+compiled at install time and ragel is no longer needed to build or contribute.
+
+* Added an :html_void parse option. XML has no void elements, so parsing HTML
+  through Hpricot::XML opened <br> as a container and everything after it became
+  its children -- "<p>one<br>two</p>" put "two" inside the <br>, and to_html
+  emitted a "</br>" that was never in the source. With html_void: true the
+  twelve HTML void elements (area, base, basefont, br, col, hr, img, input,
+  isindex, link, meta, param) take no children and get no end tag, while the
+  rest of XML parsing is unchanged.
+
+  The set is derived from ElementContent rather than hardcoded, so it cannot
+  drift from what HTML mode already does.
+
+  A void element is serialized the way the source wrote it: "<br>" stays "<br>"
+  and "<hr />" stays "<hr />". HTML mode is untouched and still writes the XHTML
+  form. Without the option a <br> element remains an ordinary element, which
+  matters because XLIFF, DocBook and Android string resources contain them
+  legitimately and dropping the end tag would produce unbalanced output.
+
+Breaking changes
+* Requires Ruby >= 3.3. Ruby 3.2 reached end of life on 31 March 2026.
+* Hpricot.scan no longer accepts a block. The block form dereferenced a NULL
+  state pointer and segfaulted on any tag carrying an attribute; it had no
+  known callers.
+* Hpricot.css is removed. Its scanner dropped the last token of every selector,
+  so "div" matched nothing and "#id.cls" silently lost ".cls". Elements#search
+  has always used the Ruby implementation in elements.rb and is unaffected.
+* Node strings now carry the source document's encoding when it declares one,
+  rather than always Encoding.default_external. A binary source (File.binread,
+  Zip::File#read) is still tagged default_external, since its bytes carry no
+  encoding information and callers depend on that. Previously a Latin-1
+  document produced UTF-8-labelled strings that failed valid_encoding?. One
+  consequence: concatenating to_html output from documents of DIFFERENT
+  encodings now raises Encoding::CompatibilityError, where before every result
+  was default_external and always concatenated.
+* html_quote escapes an attribute quote as &quot; rather than a backslash.
+  HTML does not honour backslash escaping inside an attribute value, so the old
+  output did not re-parse as the same element -- <span style="font-family:\"MS
+  Mincho\""> came back as a text node and a bogus end tag -- and a quote in an
+  attribute became an injection point. to_html output therefore differs from
+  0.10.0 for any attribute value containing a literal quote.
+* Hpricot.uxs no longer raises on hostile entity references. &#99999999999;
+  raised RangeError, &#xD800; produced invalid UTF-8, &#0; emitted a NUL byte,
+  and any non-UTF-8 input raised Encoding::CompatibilityError. Out-of-range,
+  surrogate and XML-invalid codepoints now become "?".
+* BlankSlate is removed and Builder::CssProxy inherits from BasicObject.
+  BlankSlate hid nothing (its guard compared Symbols against a String, dead
+  since Ruby 1.9), installed an Object.method_added hook that fired for every
+  method definition anywhere in the host process, leaked method_added as a
+  public method on every class, and raised SystemStackError if loaded twice.
+
+Fixes
+* Parsing is deterministic. The C scanner read uninitialised memory in its
+  HTML-mode implicit-close path, so the same bytes could produce different
+  trees from one process to the next -- 20 runs of the test suite produced five
+  distinct outcomes.
+* Two reachable segfaults are gone: Hpricot.scan with a block on any tag with
+  an attribute, and IO input whose length is an exact multiple of 16384 bytes
+  (or any reader returning nil).
+* An unbounded leak is gone. Every exception path leaked the parser state plus
+  a permanently registered GC root: roughly 1.4KB retained per call, reachable
+  from any document containing an unknown encoding in its XML declaration.
+* An unknown encoding in an XML declaration no longer raises EncodingError.
+* Malformed input parses in linear time. Unquoted attribute values swallowing
+  "<", adjacent text-node merging, unmatched end tags and the HTML
+  implicit-close ancestor walk were all quadratic. The C scanner is quadratic
+  on the last two as well; on malformed input this release is 12-37x faster
+  than the extension it replaces.
+* Elements#at and its % alias work again. They referenced Fixnum, removed in
+  Ruby 3.2, and so raised NameError for every argument type.
+* Hpricot::DocType.new, Builder#doctype, #xhtml_transitional and #xhtml_strict
+  work again. The C node's attribute slot was nil when built from Ruby, so the
+  setters raised FrozenError.
+* Traverse#nodes_at no longer prints to stdout.
+* to_html no longer mutates frozen string literals.
+* String#fast_xs is reimplemented in Ruby, byte-for-byte identical to the C
+  version across all 256 single-byte inputs.
+
+Byte-identical round-tripping
+* Unterminated comments, CDATA sections and processing instructions keep their
+  source bytes. The C scanner dropped the opening delimiter of an unterminated
+  comment, so "<p>a</p><!-- never closed" came back four bytes short.
+* Unterminated attribute quotes round-trip; the C scanner prepended the tag
+  name to the resulting text node.
+
+Other
+* JRuby is supported by the same code. The separate Java scanner is gone.
+* CI covers every maintained Ruby (3.3, 3.4, 4.0) on Linux and macOS, and gates
+  on determinism, byte-identical round-tripping and sub-quadratic scaling.
+* setup.rb and the mswin32 cross-compile machinery are removed. The ragel
+  grammar is kept under docs/grammar/ as the specification the Ruby scanner was
+  ported from; the C implementation remains available at the tag
+  pre-pure-ruby-scanner.
+
+= 0.10.0
+=== 24 March 2026
+* Fix compilation with Ruby 4.
+* Use HTTPS source in Gemfile (HTTP disallowed by Bundler)
+* Fix incompatible function pointer types for ref_func/set_func arrays
+* Remove deprecated OBJ_TAINT/OBJ_TAINTED calls (removed in Ruby 3.2+)
+* Fix format string precision type and uninitialized variable in hpricot_css
+
 = 0.9.0
 === 23 April 2024
 * Fix issue compiling with clang 16.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/README.md

@@ -1,20 +1,196 @@
-# Hpricot is over.
+# webtranslateit-hpricot
 
-After years of lack of a proper maintainer for one of why's jewels, it has been
-decided to finally close the book on hpricot. Most users have migrated to alternatives
-and there is simply no time or energy to continue with the current codebase.
+A maintained fork of [hpricot](https://github.com/hpricot/hpricot), _why the
+lucky stiff's HTML/XML parser. Upstream was declared over in 2013; this fork
+exists because [WebTranslateIt](https://webtranslateit.com) still depends on it
+and intends to keep it working.
 
-If you feel that you have the time and wish to take it over, I suggest you instead
-think about making the hpricot-like API within nokogiri 100% compatible, that is a better
-use of your time.
+## Why this fork exists
 
-But if you still feel like "No damnit, I wanna work on hpricot itself still!" then fork
-this repo and start work. Send @evanphx or @nicksieger a message if you feel like you
-want to take over the gem name with new releases under the hpricot name.
+We parse translation and localisation files — `.resx`, `.xml`, `.ts`, `.tbx`,
+`.stringsdict`, `.xtb`, `.docx`, and a dozen more — and hand them back to
+customers after editing. That imposes a requirement most parsers do not meet:
+
+**Byte-identical round-tripping.** If a customer's file writes `…` we must
+give back `…`, not `…`. If it writes `"` we must give back `"`,
+not `"`. If it orders attributes a certain way, that order must survive. A
+translation tool that silently reformats the untouched 99% of a file is worse
+than useless — it turns every export into an unreviewable diff.
+
+We measured the alternatives against a real corpus:
+
+| Parser | Preserves `…` | Preserves `"` | Preserves attribute order | Byte-identical |
+|---|---|---|---|---|
+| hpricot | yes | yes | yes | **yes** |
+| Nokogiri | only via `encoding: 'US-ASCII'`, all-or-nothing | no | yes | no |
+| Oga | no | no | yes | no |
+| REXML (`raw: :all`) | yes | yes | **no** | no |
+
+Hpricot manages this because it does not decode. It records the source byte span
+of every node and emits those bytes verbatim for anything you did not modify.
+Fidelity is a property of *not* re-encoding, and re-encoding is exactly what a
+conformant serializer must do.
+
+The second requirement is **liberality**. Hpricot accepts malformed markup that
+a conformant parser rejects. Real uploaded files are frequently malformed, and
+"your file is invalid" is not an acceptable answer when the previous version of
+the product accepted it. A prior attempt to move to REXML foundered on precisely
+this — it correctly rejected files hpricot tolerated.
+
+Hpricot is also the fastest of the four we measured (~1.6x Nokogiri), though
+that is the least important reason: parsing is well under half the cost of a
+typical import job.
+
+## Current state
+
+The library works and is in production. The Ruby layer is actively maintained.
+
+**The C extension is being replaced with pure Ruby.** The scanner is a
+ragel-generated C state machine that has had no upstream maintenance since 2013,
+and a review turned up several memory-safety defects: an uninitialised read that
+makes HTML-mode parsing non-deterministic across processes, two reachable
+segfaults, an unbounded leak on parse errors, and GC roots registered against
+dead stack frames. It also no longer compiles on Ruby trunk.
+
+Since parsing is not the bottleneck, there is no reason to keep hand-patched C
+in a library that consumes untrusted input. A pure-Ruby scanner eliminates that
+entire class of defect by construction, removes the parallel Java implementation
+maintained for JRuby, and means no native extension to rebuild on every Ruby
+release. The plan is in
+[`docs/superpowers/plans/`](docs/superpowers/plans/2026-09-08-pure-ruby-scanner.md).
+
+Byte-identical round-tripping and liberality are the acceptance criteria for
+that work, verified by a differential harness that compares the new scanner
+against the old C one on a corpus of real files.
+
+## Installing
+
+    $ gem install webtranslateit-hpricot
+
+Or in a Gemfile:
+
+```ruby
+gem 'webtranslateit-hpricot'
+```
+
+The API is hpricot's — `require 'hpricot'` and everything below applies
+unchanged.
+
+## Performance
+
+Measured on Ruby 4.0.6, arm64-darwin, parsing a real Android `strings.xml`
+fixture scaled up by repeating its `<string>` elements. `Hpricot::XML`, mean of
+several runs.
+
+| Input | C/ragel scanner | pure Ruby | + YJIT | ratio (YJIT) |
+|---|---|---|---|---|
+| 26 KB | 0.4 ms | 2.3 ms | 1.8 ms | 4.5x |
+| 186 KB | 4.3 ms | 22.8 ms | 15.7 ms | 3.7x |
+| 1.5 MB | 45.4 ms | 190.6 ms | 151.3 ms | 3.3x |
+| 5 MB | 153.2 ms | 716.5 ms | 554.4 ms | 3.6x |
+
+Both are linear. YJIT is worth roughly 25-30% and needs no code change. Parsing
+is under half the cost of even parse-plus-extract, before encoding detection,
+entity decoding or database work, so it is not the bottleneck in any real
+pipeline.
+
+### Malformed input: faster than the C scanner
+
+The table above is well-formed input, where C wins. On malformed input the
+pure-Ruby scanner is now **considerably faster**, because several quadratic
+behaviours were fixed during the port that the C scanner still has:
+
+| Input | C/ragel scanner | pure Ruby |
+|---|---|---|
+| 97 KB of nested `<div>` | 1.73 s | 0.062 s |
+| 78 KB of `<a>` inside `<button>` | 1.58 s | 0.070 s |
+| 214 KB of unmatched end tags | 2.85 s | 0.077 s |
+
+These are not micro-optimisations: the same shapes were quadratic here too at
+various points during the port, at up to 77 seconds for 156 KB. Since this
+library parses untrusted uploads, an O(n²) input shape is a denial-of-service
+vector rather than a slow path, so `test/test_complexity.rb` asserts the
+scaling ratio for each of them and fails if any becomes quadratic again.
+
+### Where the time goes
+
+Scanning is ~80% of parse time and tree building ~20%. Profiling the scanner
+puts GC at 22% of samples (13.9% sweeping, 7.7% marking) and `String#byteslice`
+at another 10%, so allocation pressure dominates and that is where tuning has
+to aim.
+
+What worked:
+
+* Dispatching on the first byte, so a text token does not attempt four
+  construct regexes before falling through.
+* `StringScanner#skip` instead of `#scan` wherever the matched text is
+  discarded — `scan` allocates a String for the match even when only its
+  success matters.
+* Matching a whole attribute in one scan rather than six calls, using the
+  capture groups directly instead of separate byteslices (~5%).
+* Not copying the document. A source already valid in an ASCII-compatible
+  encoding is scanned in place: `StringScanner#pos` and `#skip` are
+  byte-oriented whatever the encoding, and the character classes here are all
+  ASCII-only. This removes the copies that used to be made per parse — two for
+  a BINARY source (the scanning buffer, plus a re-encoded copy to slice from),
+  one for a source carrying a declared encoding. `File.binread` and
+  `Zip::File#read` both give BINARY, so the two-copy case is the common one in
+  practice, and a 5 MB upload stops allocating 10 MB of copies. Peak RSS is
+  dominated by the parse tree, so this does not show up there; it is an
+  allocation win, not a footprint one.
+
+What did not work, measured and reverted: interning repeated tag names and
+attribute keys. `string` and `name` recur thousands of times in a translation
+file, but the lookup still allocates the temporary it searches with, so the
+allocation count was unchanged and the hash probe cost as much as it saved.
+
+### Why this is viable now and was not in 2006
+
+Hpricot was written in 2006 against Ruby 1.8, a tree-walking interpreter with
+no bytecode VM. A pure-Ruby scanner then would have been perhaps an order of
+magnitude slower than this one, which is a large part of why the scanner was
+written in C to begin with. Ruby 1.9 brought YARV, 2.x brought generational and
+incremental GC, and 3.x brought YJIT. The trade-off that justified a C extension
+in 2006 simply does not hold in 2026: this scanner parses a 1.5 MB document in
+130 ms, which is comfortably faster in absolute terms than the original C
+extension managed on the hardware it was written for.
+
+### "Pure Ruby" — what that does and does not mean
+
+The hot loop runs inside `StringScanner#scan`, and `strscan` is implemented in C
+in CRuby. So C still executes; what changed is *whose* C it is.
+
+What was removed is this project's own native code: roughly 7,000 lines of
+ragel-generated C plus a hand-written scanner, a parallel Java implementation
+for JRuby, and `fast_xs`. That code had no upstream maintainer after 2013, could
+not be regenerated without ragel (which nobody had installed), had to be
+recompiled for every Ruby and platform, and had been hand-patched in the
+generated output. A review of it found an uninitialised read, two reachable
+segfaults, an unbounded leak and GC roots registered on dead stack frames. It
+also stopped compiling on Ruby trunk.
+
+`strscan` is a different proposition: it ships with Ruby, is maintained and
+fuzzed by ruby-core, gets security fixes without us doing anything, and has an
+implementation on every engine — which is why JRuby now works from the same
+source with no Java of ours.
+
+So the accurate claim is **no native extension of our own**: nothing is
+compiled at install time, there is no `extensions` entry in the gemspec, no
+ragel, and no platform-specific builds. A character-by-character Ruby loop
+would be an order of magnitude slower than `StringScanner`, so keeping the hot
+loop inside it is a deliberate design constraint.
+
+## Contributing
+
+Issues and pull requests are welcome, though be aware this fork is maintained
+for a specific purpose and changes are weighed against that. If you want a
+general-purpose HTML parser, use
+[Nokogiri](https://github.com/sparklemotion/nokogiri) — it is better maintained,
+standards-compliant, and almost certainly what you want.
 
 Thanks to \_why for all the fun. We'll never forget it.
 
-## Now back to your original README content...
+---
 
 
 # Hpricot, Read Any HTML
@@ -62,13 +238,6 @@ not going to say "Use at your own risk" because I don't want this library to be
 risky.  If you trip on something, I'll share the liability by repairing things
 as quickly as I can.  Your responsibility is to report the inadequacies.
 
-## Installing Hpricot
-
-You may get the latest stable version from Rubyforge. Win32 binaries,
-Java binaries (for JRuby), and source gems are available.
-
-    $ gem install hpricot
-
 ## An Hpricot Showcase
 
 We're going to run through a big pile of examples to get you jump-started.

data/Rakefile

@@ -1,237 +1,43 @@
-require 'bundler/setup'
-ENV.delete('RUBYOPT')           # Don't propagate RUBYOPT/Bundler to subprocesses
-require 'rake/clean'
-require 'rubygems/package_task'
-require 'rdoc/task'
 require 'rake/testtask'
-begin
-  require 'rake/extensiontask'
-rescue LoadError
-  abort "To build, please first gem install rake-compiler"
-end
-
-RbConfig = Config unless defined?(RbConfig)
-
-NAME = "hpricot"
-REV = (`#{ENV['GIT'] || "git"} rev-list HEAD`.split.length + 1).to_s
-VERS = ENV['VERSION'] || "0.8" + (REV ? ".#{REV}" : "")
-PKG = "#{NAME}-#{VERS}"
-BIN = "*.{bundle,jar,so,o,obj,pdb,lib,def,exp,class,rbc}"
-CLEAN.include ["#{BIN}", "ext/**/#{BIN}", "lib/**/#{BIN}", "test/**/#{BIN}",
-               'ext/fast_xs/Makefile', 'ext/hpricot_scan/Makefile',
-               '**/.*.sw?', '*.gem', '.config', 'pkg', 'lib/hpricot_scan.rb', 'lib/fast_xs.rb']
-RDOC_OPTS = ['--quiet', '--title', 'The Hpricot Reference', '--main', 'README.md', '--inline-source']
-PKG_FILES = %w(CHANGELOG COPYING README.md Rakefile) +
-      Dir.glob("{bin,doc,test,extras}/**/*") +
-      (Dir.glob("lib/**/*.rb") - %w(lib/hpricot_scan.rb lib/fast_xs.rb)) +
-      Dir.glob("ext/**/*.{h,java,c,rb,rl}") +
-      %w[ext/hpricot_scan/hpricot_scan.c ext/hpricot_scan/hpricot_css.c ext/hpricot_scan/HpricotScanService.java] # needed because they are generated later
-RAGEL_C_CODE_GENERATION_STYLES = {
-  "table_driven" => 'T0',
-  "faster_table_driven" => 'T1',
-  "flat_table_driven" => 'F0',
-  "faster_flat_table_driven" => 'F1',
-  "goto_driven" => 'G0',
-  "faster_goto_driven" => 'G1',
-  "really_fast goto_driven" => 'G2'
-  # "n_way_split_really_fast_goto_driven" => 'P<N>'
-}
-DEFAULT_RAGEL_C_CODE_GENERATION = "really_fast goto_driven"
-SPEC =
-  Gem::Specification.new do |s|
-    s.name = NAME
-    s.version = VERS
-    s.platform = Gem::Platform::RUBY
-    s.has_rdoc = true
-    s.rdoc_options += RDOC_OPTS
-    s.extra_rdoc_files = ["README.md", "CHANGELOG", "COPYING"]
-    s.summary = "a swift, liberal HTML parser with a fantastic library"
-    s.description = s.summary
-    s.author = "why the lucky stiff"
-    s.email = 'why@ruby-lang.org'
-    s.homepage = 'http://code.whytheluckystiff.net/hpricot/'
-    s.rubyforge_project = 'hobix'
-    s.files = PKG_FILES
-    s.require_paths = ["lib"]
-    s.extensions = FileList["ext/**/extconf.rb"].to_a
-    s.bindir = "bin"
-  end
-# Dup the spec before any of its calculated ivars are set (e.g., #cache_file)
-Win32Spec = SPEC.dup
-JRubySpec = SPEC.dup
-
-# FAT cross-compile
-# Pass RUBY_CC_VERSION=1.8.7:1.9.2 when packaging for 1.8+1.9 mswin32 binaries
-%w(hpricot_scan fast_xs).each do |target|
-  Rake::ExtensionTask.new(target, SPEC) do |ext|
-    ext.lib_dir = File.join('lib', target) if ENV['RUBY_CC_VERSION']
-    ext.cross_compile = true                # enable cross compilation (requires cross compile toolchain)
-    ext.cross_platform = 'i386-mswin32'     # forces the Windows platform instead of the default one
-  end
-
-  # HACK around 1.9.2 cross .def file creation
-  def_file = "tmp/i386-mswin32/#{target}/1.9.2/#{target}-i386-mingw32.def"
-  directory File.dirname(def_file)
-  file def_file => File.dirname(def_file) do |t|
-    File.open(t.name, "w") do |f|
-      f << "EXPORTS\nInit_#{target}\n"
-    end
-  end
-
-  task File.join(File.dirname(def_file), "Makefile") => def_file
-  # END HACK
-  file "lib/#{target}.rb" do |t|
-    File.open(t.name, "w") do |f|
-      f.puts %{require "#{target}/\#{RUBY_VERSION.sub(/\\.\\d+$/, '')}/#{target}"}
-    end
-  end
-end
-file 'ext/hpricot_scan/extconf.rb' => :ragel
-
-desc "set environment variables to build and/or test with debug options"
-task :debug do
-  ENV['CFLAGS'] ||= ""
-  ENV['CFLAGS'] += " -g -DDEBUG"
-end
-
-desc "Does a full compile, test run"
-if defined?(JRUBY_VERSION)
-task :default => [:compile_java, :clean_fat_rb, :test]
-else
-task :default => [:compile, :clean_fat_rb, :test]
-end
-
-task :clean_fat_rb do
-  rm_f "lib/hpricot_scan.rb"
-  rm_f "lib/fast_xs.rb"
-end
-
-desc "Packages up Hpricot for all platforms."
-task :package => [:clean]
 
-desc "Run all the tests"
-Rake::TestTask.new do |t|
-    t.libs << "test"
-    t.test_files = FileList['test/test_*.rb']
-    t.verbose = true
+# test_differential needs a separately built checkout of the old C scanner as
+# an oracle (see test/differential_helper.rb); it is a development tool, not
+# part of the default suite.
+Rake::TestTask.new(:test) do |t|
+  t.libs = %w[lib test]
+  t.test_files = FileList['test/test_*.rb'].exclude(/differential/)
+  t.verbose = true
 end
 
-Rake::RDocTask.new do |rdoc|
-    rdoc.rdoc_dir = 'doc/rdoc'
-    rdoc.options += RDOC_OPTS
-    rdoc.main = "README.md"
-    rdoc.rdoc_files.add ['README.md', 'CHANGELOG', 'COPYING', 'lib/**/*.rb']
-end
-
-Gem::PackageTask.new(SPEC) do |p|
-    p.need_tar = true
-    p.gem_spec = SPEC
-end
-
-### Win32 Packages ###
-Win32Spec.platform = 'i386-mswin32'
-Win32Spec.files = PKG_FILES + %w(hpricot_scan fast_xs).map do |t|
-  unless ENV['RUBY_CC_VERSION']
-    file "lib/#{t}/1.8/#{t}.so" do
-      abort "ERROR while packaging: re-run for fat win32 gems:\nrake #{ARGV.join(' ')} RUBY_CC_VERSION=1.8.7:1.9.2"
-    end
-  end
-  ["lib/#{t}.rb", "lib/#{t}/1.8/#{t}.so", "lib/#{t}/1.9/#{t}.so"]
-end.flatten
-Win32Spec.extensions = []
-
-Gem::PackageTask.new(Win32Spec) do |p|
-  p.need_tar = false
-  p.gem_spec = Win32Spec
+Rake::TestTask.new(:differential) do |t|
+  t.libs = %w[lib test]
+  t.test_files = FileList['test/test_differential.rb']
+  t.verbose = true
 end
 
-JRubySpec.platform = 'java'
-JRubySpec.files = PKG_FILES + ["lib/hpricot_scan.jar", "lib/fast_xs.jar"]
-JRubySpec.extensions = []
-
-Gem::PackageTask.new(JRubySpec) do |p|
-  p.need_tar = false
-  p.gem_spec = JRubySpec
-end
-
-desc "Determines the Ragel version and displays it on the console along with the location of the Ragel binary."
-task :ragel_version do
-  @ragel_v = `ragel -v`[/(version )(\S*)/,2].to_f
-  puts "Using ragel version: #{@ragel_v}, location: #{`which ragel`}"
-  @ragel_v
-end
-
-desc "Generates the C scanner code with Ragel."
-task :ragel => [:ragel_version] do
-  if @ragel_v >= 6.1
-    @ragel_c_code_generation_style = RAGEL_C_CODE_GENERATION_STYLES[DEFAULT_RAGEL_C_CODE_GENERATION]
-    Dir.chdir("ext/hpricot_scan") do
-      sh %{ragel hpricot_scan.rl -#{@ragel_c_code_generation_style} -o hpricot_scan.c}
-      sh %{ragel hpricot_css.rl -#{@ragel_c_code_generation_style} -o hpricot_css.c}
-    end
-  else
-    STDERR.puts "Ragel 6.1 or greater is required."
-    exit(1)
-  end
-end
-
-# Java only supports the table-driven code
-# generation style at this point.
-desc "Generates the Java scanner code using the Ragel table-driven code generation style."
-task :ragel_java => [:ragel_version] do
-  if @ragel_v >= 6.1
-    puts "compiling with ragel version #{@ragel_v}"
-    Dir.chdir("ext/hpricot_scan") do
-      sh %{ragel -J -o HpricotCss.java hpricot_css.java.rl}
-      sh %{ragel -J -o HpricotScanService.java hpricot_scan.java.rl}
-    end
-  else
-    STDERR.puts "Ragel 6.1 or greater is required."
-    exit(1)
-  end
-end
-
-### JRuby Compile ###
-
-def java_classpath_arg # myriad of ways to discover JRuby classpath
-  begin
-    cpath  = Java::java.lang.System.getProperty('java.class.path').split(File::PATH_SEPARATOR)
-    cpath += Java::java.lang.System.getProperty('sun.boot.class.path').split(File::PATH_SEPARATOR)
-    jruby_cpath = cpath.compact.join(File::PATH_SEPARATOR)
-  rescue => e
-  end
-  unless jruby_cpath
-    jruby_cpath = ENV['JRUBY_PARENT_CLASSPATH'] || ENV['JRUBY_HOME'] &&
-      FileList["#{ENV['JRUBY_HOME']}/lib/*.jar"].join(File::PATH_SEPARATOR)
-  end
-  unless jruby_cpath || ENV['CLASSPATH'] =~ /jruby/
-    abort %{WARNING: No JRuby classpath has been set up.
-Define JRUBY_HOME=/path/to/jruby on the command line or in the environment}
-  end
-  "-cp \"#{jruby_cpath}\""
-end
-
-def compile_java(filenames, jarname)
-  sh %{javac -source 1.5 -target 1.5 #{java_classpath_arg} #{filenames.join(" ")}}
-  sh %{jar cf #{jarname} *.class}
-end
-
-task :hpricot_scan_java => [:ragel_java] do
-  Dir.chdir "ext/hpricot_scan" do
-    compile_java(["HpricotScanService.java", "HpricotCss.java"], "hpricot_scan.jar")
-  end
-end
-
-task :fast_xs_java do
-  Dir.chdir "ext/fast_xs" do
-    compile_java(["FastXsService.java"], "fast_xs.jar")
-  end
+desc 'Assert the suite produces identical results across 20 runs'
+task :determinism do
+  results = 20.times.map do
+    `ruby -Ilib -Itest test/test_parser.rb 2>&1`[/\d+ failures, \d+ errors/]
+  end.tally
+  abort "non-deterministic: #{results.inspect}" if results.size != 1
+  puts "deterministic across 20 runs: #{results.keys.first}"
 end
 
-%w(hpricot_scan fast_xs).each do |ext|
-  file "lib/#{ext}.jar" => "#{ext}_java" do |t|
-    mv "ext/#{ext}/#{ext}.jar", "lib"
+desc 'Assert every fixture round-trips byte-identically'
+task :fidelity do
+  $LOAD_PATH.unshift('lib')
+  require 'hpricot'
+  files = Dir['test/files/*'].select { |f| File.file?(f) }
+  # Compared as bytes: src comes from binread (ASCII-8BIT) while output
+  # carries the document's encoding, and String#== is false across
+  # incompatible encodings even when the bytes match.
+  bad = files.reject do |f|
+    src = File.binread(f)
+    Hpricot::XML(src).to_original_html.b == src.b
   end
-  task :compile_java => "lib/#{ext}.jar"
+  abort "not byte-identical: #{bad.inspect}" unless bad.empty?
+  puts "#{files.size} fixtures round-trip byte-identically"
 end
 
+task default: %i[test fidelity determinism]