parse5 0.6.1 → 1.0.0

package/.npmignore

@@ -1,3 +1,5 @@
 .idea
 node_modules
-benchmark
+benchmark
+test
+logo.png

package/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2013 Ivan Nikulin (ifaaan@gmail.com)
+Copyright (c) 2013-2014 Ivan Nikulin (ifaaan@gmail.com)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -1,77 +1,166 @@
-parse5
-======
-Fast full-featured HTML parser for Node. Based on WHATWG HTML5 specification.  
-To build [TestCafé](http://testcafe.devexpress.com/) we needed fast and ready for production HTML parser for node.js, which will parse HTML as a modern browser's parser.
-Existing solutions were either too slow or their output was too inaccurate. So, this is how parse5 was born.
-
-Install
--------
-```
-$ npm install parse5
-```
-
-Usage and API
--------------
-```js
-var Parser = require('parse5').Parser;
-
-//Instantiate parser
-var parser = new Parser();
-
-//Then feed it with an HTML document
-var document = parser.parse('<!DOCTYPE html><html><head></head><body>Hi there!</body></html>')
-
-//Now let's parse HTML-snippet
-var fragment = parser.parseFragment('<title>Parse5 is &#102;&#117;&#99;&#107;ing awesome!</title><h1>42</h1>');
-
-```
-
-Is it fast?
------------
-Check out [this benchmark](https://github.com/inikulin/node-html-parser-bench).
-
-```
-Starting benchmark. Fasten your seatbelts...
-html5 (https://github.com/aredridel/html5) x 0.18 ops/sec ±5.92% (5 runs sampled)
-htmlparser (https://github.com/tautologistics/node-htmlparser/) x 3.83 ops/sec ±42.43% (14 runs sampled)
-htmlparser2 (https://github.com/fb55/htmlparser2) x 4.05 ops/sec ±39.27% (15 runs sampled)
-parse5 (https://github.com/inikulin/parse5) x 3.04 ops/sec ±51.81% (13 runs sampled)
-Fastest is htmlparser2 (https://github.com/fb55/htmlparser2),parse5 (https://github.com/inikulin/parse5)
-```
-
-So, parse5 is as fast as simple specification incompatible parsers and ~15-times(!) faster than the current specification compatible parser available for the node.
-
-Testing
--------
-Test data is adopted from [html5lib project](https://github.com/html5lib). Parser is covered by more than 8000 test cases.
-To run tests:
-```
-$ node test/run_tests.js
-```
-
-Custom tree adapter
--------------------
-You can create a custom tree adapter so parse5 can work with your own DOM-tree implementation.
-Just pass your adapter implementation to the parser's constructor as an argument:
-
-```js
-var Parser = require('parse5').Parser;
-
-var myTreeAdapter = {
-   //Adapter methods...
-};
-
-//Instantiate parser
-var parser = new Parser(myTreeAdapter);
-```
-
-Sample implementation can be found [here](https://github.com/inikulin/parse5/blob/master/lib/default_tree_adapter.js).
-The custom tree adapter should implement all methods exposed via `exports` in the sample implementation.
-
-Questions or suggestions?
--------------------------
-If you have any questions, please feel free to create an issue [here on github](https://github.com/inikulin/parse5/issues).
-
-Author
-------
-[Ivan Nikulin](https://github.com/inikulin) (ifaaan@gmail.com)
+![logo](https://raw.github.com/inikulin/parse5/master/logo.png)  
+
+Fast full-featured HTML parsing/serialization toolset for Node. Based on WHATWG HTML5 specification.  
+To build [TestCafé](http://testcafe.devexpress.com/) we needed fast and ready for production HTML parser, which will parse HTML as a modern browser's parser.
+Existing solutions were either too slow or their output was too inaccurate. So, this is how parse5 was born.
+
+##Install
+```
+$ npm install parse5
+```
+
+
+##Simple usage
+```js
+var Parser = require('parse5').Parser;
+
+//Instantiate parser
+var parser = new Parser();
+
+//Then feed it with an HTML document
+var document = parser.parse('<!DOCTYPE html><html><head></head><body>Hi there!</body></html>')
+
+//Now let's parse HTML-snippet
+var fragment = parser.parseFragment('<title>Parse5 is &#102;&#117;&#99;&#107;ing awesome!</title><h1>42</h1>');
+
+```
+
+##Is it fast?
+Check out [this benchmark](https://github.com/inikulin/node-html-parser-bench).
+
+```
+Starting benchmark. Fasten your seatbelts...
+html5 (https://github.com/aredridel/html5) x 0.18 ops/sec ±5.92% (5 runs sampled)
+htmlparser (https://github.com/tautologistics/node-htmlparser/) x 3.83 ops/sec ±42.43% (14 runs sampled)
+htmlparser2 (https://github.com/fb55/htmlparser2) x 4.05 ops/sec ±39.27% (15 runs sampled)
+parse5 (https://github.com/inikulin/parse5) x 3.04 ops/sec ±51.81% (13 runs sampled)
+Fastest is htmlparser2 (https://github.com/fb55/htmlparser2),parse5 (https://github.com/inikulin/parse5)
+```
+
+So, parse5 is as fast as simple specification incompatible parsers and ~15-times(!) faster than the current specification compatible parser available for the node.
+
+
+##API reference
+
+###Enum: TreeAdapters
+Provides built-in tree adapters which can be passed as an optional argument to the `Parser` and `TreeSerializer` constructors.   
+
+####&bull; TreeAdapters.default 
+Default tree format for parse5.
+
+
+####&bull; TreeAdapters.htmlparser2
+Quite popular [htmlparser2](https://github.com/fb55/htmlparser2) tree format (e.g. used in [cheerio](https://github.com/MatthewMueller/cheerio) and [jsdom](https://github.com/tmpvar/jsdom)).  
+
+---------------------------------------
+    
+    
+###Class: Parser
+Provides HTML parsing functionality.
+
+####&bull; Parser.ctor([treeAdapter])
+Creates new reusable instance of the `Parser`. Optional `treeAdapter` argument specifies resulting tree format. If `treeAdapter` argument is not specified, `default` tree adapter will be used.
+
+*Example:*
+```js
+var parse5 = require('parse5');
+
+//Instantiate new parser with default tree adapter
+var parser1 = new parse5.Parser();
+
+//Instantiate new parser with htmlparser2 tree adapter
+var parser2 = new parse5.Parser(parse5.TreeAdapters.htmlparser2);
+```
+
+
+
+####&bull; Parser.parse(html)
+Parses specified `html` string. Returns `document` node.
+
+*Example:*
+```js
+var document = parser.parse('<!DOCTYPE html><html><head></head><body>Hi there!</body></html>');
+```
+
+
+####&bull; Parser.parseFragment(htmlFragment, [contextElement])
+Parses given `htmlFragment`. Returns `documentFragment` node. Optional `contextElement` argument specifies resulting tree format. If `contextElement` argument is not specified, `<div>` element will be used.
+
+*Example:*
+```js
+var documentFragment = parser.parseFragment('<table></table>');
+
+//Parse html fragment in context of the parsed <table> element
+var trFragment = parser.parseFragment('<tr><td>Shake it, baby</td></tr>', documentFragment.childNodes[0]);
+```
+
+---------------------------------------
+
+###Class: TreeSerializer
+Provides tree-to-HTML serialization functionality.
+
+####&bull; TreeSerializer.ctor([treeAdapter])
+Creates new reusable instance of the `TreeSerializer`. Optional `treeAdapter` argument specifies input tree format. If `treeAdapter` argument is not specified, `default` tree adapter will be used.
+
+*Example:*
+```js
+var parse5 = require('parse5');
+
+//Instantiate new serializer with default tree adapter
+var serializer1 = new parse5.TreeSerializer();
+
+//Instantiate new serializer with htmlparser2 tree adapter
+var serializer2 = new parse5.TreeSerializer(parse5.TreeAdapters.htmlparser2);
+```
+
+
+####&bull; TreeSerializer.serialize(node)
+Serializes the given `node`. Returns HTML string.
+
+*Example:*
+```js
+var document = parser.parse('<!DOCTYPE html><html><head></head><body>Hi there!</body></html>');
+
+//Serialize document
+var html = serializer.serialize(document);
+
+//Serialize <body> element content
+var bodyInnerHtml = serializer.serialize(document.childNodes[0].childNodes[1]);
+```
+
+---------------------------------------
+
+
+##Testing
+Test data is adopted from [html5lib project](https://github.com/html5lib). Parser is covered by more than 8000 test cases.
+To run tests:
+```
+$ node test/run_tests.js
+```
+
+
+##Custom tree adapter
+You can create a custom tree adapter so parse5 can work with your own DOM-tree implementation.
+Just pass your adapter implementation to the parser's constructor as an argument:
+
+```js
+var Parser = require('parse5').Parser;
+
+var myTreeAdapter = {
+   //Adapter methods...
+};
+
+//Instantiate parser
+var parser = new Parser(myTreeAdapter);
+```
+
+Sample implementation can be found [here](https://github.com/inikulin/parse5/blob/master/lib/tree_adapters/default.js).
+The custom tree adapter should implement all methods exposed via `exports` in the sample implementation.
+
+##Questions or suggestions?
+If you have any questions, please feel free to create an issue [here on github](https://github.com/inikulin/parse5/issues).
+
+
+##Author
+[Ivan Nikulin](https://github.com/inikulin) (ifaaan@gmail.com)
+

package/index.js

@@ -0,0 +1,7 @@
+exports.Parser = require('./lib/tree_construction_stage/parser');
+exports.TreeSerializer = require('./lib/tree_construction_stage/tree_serializer');
+
+exports.TreeAdapters = {
+    default: require('./lib/tree_adapters/default'),
+    htmlparser2: require('./lib/tree_adapters/htmlparser2')
+};

package/lib/{named_entity_trie.js → tokenization_stage/named_entity_trie.js}

@@ -1,7 +1,7 @@
 //NOTE: this file contains auto generated trie structure that is used for named entity references consumption
 //(see: http://www.whatwg.org/specs/web-apps/current-work/multipage/tokenization.html#tokenizing-character-references and
 //http://www.whatwg.org/specs/web-apps/current-work/multipage/named-character-references.html#named-character-references)
-exports.TRIE = {
+module.exports = {
     0x41: {l: {0x61: {l: {0x63: {l: {0x75: {l: {0x74: {l: {0x65: {l: {0x3B: {c: [193]}}, c: [193]}}}}}}}}}, 0x62: {l: {0x72: {l: {0x65: {l: {0x76: {l: {0x65: {l: {0x3B: {c: [258]}}}}}}}}}}}, 0x63: {l: {0x69: {l: {0x72: {l: {0x63: {l: {0x3B: {c: [194]}}, c: [194]}}}}}, 0x79: {l: {0x3B: {c: [1040]}}}}}, 0x45: {l: {0x6C: {l: {0x69: {l: {0x67: {l: {0x3B: {c: [198]}}, c: [198]}}}}}}}, 0x66: {l: {0x72: {l: {0x3B: {c: [120068]}}}}}, 0x67: {l: {0x72: {l: {0x61: {l: {0x76: {l: {0x65: {l: {0x3B: {c: [192]}}, c: [192]}}}}}}}}}, 0x6C: {l: {0x70: {l: {0x68: {l: {0x61: {l: {0x3B: {c: [913]}}}}}}}}}, 0x6D: {l: {0x61: {l: {0x63: {l: {0x72: {l: {0x3B: {c: [256]}}}}}}}}}, 0x4D: {l: {0x50: {l: {0x3B: {c: [38]}}, c: [38]}}}, 0x6E: {l: {0x64: {l: {0x3B: {c: [10835]}}}}}, 0x6F: {l: {0x67: {l: {0x6F: {l: {0x6E: {l: {0x3B: {c: [260]}}}}}}}, 0x70: {l: {0x66: {l: {0x3B: {c: [120120]}}}}}}}, 0x70: {l: {0x70: {l: {0x6C: {l: {0x79: {l: {0x46: {l: {0x75: {l: {0x6E: {l: {0x63: {l: {0x74: {l: {0x69: {l: {0x6F: {l: {0x6E: {l: {0x3B: {c: [8289]}}}}}}}}}}}}}}}}}}}}}}}}}, 0x72: {l: {0x69: {l: {0x6E: {l: {0x67: {l: {0x3B: {c: [197]}}, c: [197]}}}}}}}, 0x73: {l: {0x63: {l: {0x72: {l: {0x3B: {c: [119964]}}}}}, 0x73: {l: {0x69: {l: {0x67: {l: {0x6E: {l: {0x3B: {c: [8788]}}}}}}}}}}}, 0x74: {l: {0x69: {l: {0x6C: {l: {0x64: {l: {0x65: {l: {0x3B: {c: [195]}}, c: [195]}}}}}}}}}, 0x75: {l: {0x6D: {l: {0x6C: {l: {0x3B: {c: [196]}}, c: [196]}}}}}}},
     0x61: {l: {0x61: {l: {0x63: {l: {0x75: {l: {0x74: {l: {0x65: {l: {0x3B: {c: [225]}}, c: [225]}}}}}}}}}, 0x62: {l: {0x72: {l: {0x65: {l: {0x76: {l: {0x65: {l: {0x3B: {c: [259]}}}}}}}}}}}, 0x63: {l: {0x3B: {c: [8766]}, 0x64: {l: {0x3B: {c: [8767]}}}, 0x45: {l: {0x3B: {c: [8766, 819]}}}, 0x69: {l: {0x72: {l: {0x63: {l: {0x3B: {c: [226]}}, c: [226]}}}}}, 0x75: {l: {0x74: {l: {0x65: {l: {0x3B: {c: [180]}}, c: [180]}}}}}, 0x79: {l: {0x3B: {c: [1072]}}}}}, 0x65: {l: {0x6C: {l: {0x69: {l: {0x67: {l: {0x3B: {c: [230]}}, c: [230]}}}}}}}, 0x66: {l: {0x3B: {c: [8289]}, 0x72: {l: {0x3B: {c: [120094]}}}}}, 0x67: {l: {0x72: {l: {0x61: {l: {0x76: {l: {0x65: {l: {0x3B: {c: [224]}}, c: [224]}}}}}}}}}, 0x6C: {l: {0x65: {l: {0x66: {l: {0x73: {l: {0x79: {l: {0x6D: {l: {0x3B: {c: [8501]}}}}}}}}}, 0x70: {l: {0x68: {l: {0x3B: {c: [8501]}}}}}}}, 0x70: {l: {0x68: {l: {0x61: {l: {0x3B: {c: [945]}}}}}}}}}, 0x6D: {l: {0x61: {l: {0x63: {l: {0x72: {l: {0x3B: {c: [257]}}}}}, 0x6C: {l: {0x67: {l: {0x3B: {c: [10815]}}}}}}}, 0x70: {l: {0x3B: {c: [38]}}, c: [38]}}}, 0x6E: {l: {0x64: {l: {0x61: {l: {0x6E: {l: {0x64: {l: {0x3B: {c: [10837]}}}}}}}, 0x3B: {c: [8743]}, 0x64: {l: {0x3B: {c: [10844]}}}, 0x73: {l: {0x6C: {l: {0x6F: {l: {0x70: {l: {0x65: {l: {0x3B: {c: [10840]}}}}}}}}}}}, 0x76: {l: {0x3B: {c: [10842]}}}}}, 0x67: {l: {0x3B: {c: [8736]}, 0x65: {l: {0x3B: {c: [10660]}}}, 0x6C: {l: {0x65: {l: {0x3B: {c: [8736]}}}}}, 0x6D: {l: {0x73: {l: {0x64: {l: {0x61: {l: {0x61: {l: {0x3B: {c: [10664]}}}, 0x62: {l: {0x3B: {c: [10665]}}}, 0x63: {l: {0x3B: {c: [10666]}}}, 0x64: {l: {0x3B: {c: [10667]}}}, 0x65: {l: {0x3B: {c: [10668]}}}, 0x66: {l: {0x3B: {c: [10669]}}}, 0x67: {l: {0x3B: {c: [10670]}}}, 0x68: {l: {0x3B: {c: [10671]}}}}}, 0x3B: {c: [8737]}}}}}}}, 0x72: {l: {0x74: {l: {0x3B: {c: [8735]}, 0x76: {l: {0x62: {l: {0x3B: {c: [8894]}, 0x64: {l: {0x3B: {c: [10653]}}}}}}}}}}}, 0x73: {l: {0x70: {l: {0x68: {l: {0x3B: {c: [8738]}}}}}, 0x74: {l: {0x3B: {c: [197]}}}}}, 0x7A: {l: {0x61: {l: {0x72: {l: {0x72: {l: {0x3B: {c: [9084]}}}}}}}}}}}}}, 0x6F: {l: {0x67: {l: {0x6F: {l: {0x6E: {l: {0x3B: {c: [261]}}}}}}}, 0x70: {l: {0x66: {l: {0x3B: {c: [120146]}}}}}}}, 0x70: {l: {0x61: {l: {0x63: {l: {0x69: {l: {0x72: {l: {0x3B: {c: [10863]}}}}}}}}}, 0x3B: {c: [8776]}, 0x45: {l: {0x3B: {c: [10864]}}}, 0x65: {l: {0x3B: {c: [8778]}}}, 0x69: {l: {0x64: {l: {0x3B: {c: [8779]}}}}}, 0x6F: {l: {0x73: {l: {0x3B: {c: [39]}}}}}, 0x70: {l: {0x72: {l: {0x6F: {l: {0x78: {l: {0x3B: {c: [8776]}, 0x65: {l: {0x71: {l: {0x3B: {c: [8778]}}}}}}}}}}}}}}}, 0x72: {l: {0x69: {l: {0x6E: {l: {0x67: {l: {0x3B: {c: [229]}}, c: [229]}}}}}}}, 0x73: {l: {0x63: {l: {0x72: {l: {0x3B: {c: [119990]}}}}}, 0x74: {l: {0x3B: {c: [42]}}}, 0x79: {l: {0x6D: {l: {0x70: {l: {0x3B: {c: [8776]}, 0x65: {l: {0x71: {l: {0x3B: {c: [8781]}}}}}}}}}}}}}, 0x74: {l: {0x69: {l: {0x6C: {l: {0x64: {l: {0x65: {l: {0x3B: {c: [227]}}, c: [227]}}}}}}}}}, 0x75: {l: {0x6D: {l: {0x6C: {l: {0x3B: {c: [228]}}, c: [228]}}}}}, 0x77: {l: {0x63: {l: {0x6F: {l: {0x6E: {l: {0x69: {l: {0x6E: {l: {0x74: {l: {0x3B: {c: [8755]}}}}}}}}}}}}}, 0x69: {l: {0x6E: {l: {0x74: {l: {0x3B: {c: [10769]}}}}}}}}}}},
     0x62: {l: {0x61: {l: {0x63: {l: {0x6B: {l: {0x63: {l: {0x6F: {l: {0x6E: {l: {0x67: {l: {0x3B: {c: [8780]}}}}}}}}}, 0x65: {l: {0x70: {l: {0x73: {l: {0x69: {l: {0x6C: {l: {0x6F: {l: {0x6E: {l: {0x3B: {c: [1014]}}}}}}}}}}}}}}}, 0x70: {l: {0x72: {l: {0x69: {l: {0x6D: {l: {0x65: {l: {0x3B: {c: [8245]}}}}}}}}}}}, 0x73: {l: {0x69: {l: {0x6D: {l: {0x3B: {c: [8765]}, 0x65: {l: {0x71: {l: {0x3B: {c: [8909]}}}}}}}}}}}}}}}, 0x72: {l: {0x76: {l: {0x65: {l: {0x65: {l: {0x3B: {c: [8893]}}}}}}}, 0x77: {l: {0x65: {l: {0x64: {l: {0x3B: {c: [8965]}, 0x67: {l: {0x65: {l: {0x3B: {c: [8965]}}}}}}}}}}}}}}}, 0x62: {l: {0x72: {l: {0x6B: {l: {0x3B: {c: [9141]}, 0x74: {l: {0x62: {l: {0x72: {l: {0x6B: {l: {0x3B: {c: [9142]}}}}}}}}}}}}}}}, 0x63: {l: {0x6F: {l: {0x6E: {l: {0x67: {l: {0x3B: {c: [8780]}}}}}}}, 0x79: {l: {0x3B: {c: [1073]}}}}}, 0x64: {l: {0x71: {l: {0x75: {l: {0x6F: {l: {0x3B: {c: [8222]}}}}}}}}}, 0x65: {l: {0x63: {l: {0x61: {l: {0x75: {l: {0x73: {l: {0x3B: {c: [8757]}, 0x65: {l: {0x3B: {c: [8757]}}}}}}}}}}}, 0x6D: {l: {0x70: {l: {0x74: {l: {0x79: {l: {0x76: {l: {0x3B: {c: [10672]}}}}}}}}}}}, 0x70: {l: {0x73: {l: {0x69: {l: {0x3B: {c: [1014]}}}}}}}, 0x72: {l: {0x6E: {l: {0x6F: {l: {0x75: {l: {0x3B: {c: [8492]}}}}}}}}}, 0x74: {l: {0x61: {l: {0x3B: {c: [946]}}}, 0x68: {l: {0x3B: {c: [8502]}}}, 0x77: {l: {0x65: {l: {0x65: {l: {0x6E: {l: {0x3B: {c: [8812]}}}}}}}}}}}}}, 0x66: {l: {0x72: {l: {0x3B: {c: [120095]}}}}}, 0x69: {l: {0x67: {l: {0x63: {l: {0x61: {l: {0x70: {l: {0x3B: {c: [8898]}}}}}, 0x69: {l: {0x72: {l: {0x63: {l: {0x3B: {c: [9711]}}}}}}}, 0x75: {l: {0x70: {l: {0x3B: {c: [8899]}}}}}}}, 0x6F: {l: {0x64: {l: {0x6F: {l: {0x74: {l: {0x3B: {c: [10752]}}}}}}}, 0x70: {l: {0x6C: {l: {0x75: {l: {0x73: {l: {0x3B: {c: [10753]}}}}}}}}}, 0x74: {l: {0x69: {l: {0x6D: {l: {0x65: {l: {0x73: {l: {0x3B: {c: [10754]}}}}}}}}}}}}}, 0x73: {l: {0x71: {l: {0x63: {l: {0x75: {l: {0x70: {l: {0x3B: {c: [10758]}}}}}}}}}, 0x74: {l: {0x61: {l: {0x72: {l: {0x3B: {c: [9733]}}}}}}}}}, 0x74: {l: {0x72: {l: {0x69: {l: {0x61: {l: {0x6E: {l: {0x67: {l: {0x6C: {l: {0x65: {l: {0x64: {l: {0x6F: {l: {0x77: {l: {0x6E: {l: {0x3B: {c: [9661]}}}}}}}}}, 0x75: {l: {0x70: {l: {0x3B: {c: [9651]}}}}}}}}}}}}}}}}}}}}}, 0x75: {l: {0x70: {l: {0x6C: {l: {0x75: {l: {0x73: {l: {0x3B: {c: [10756]}}}}}}}}}}}, 0x76: {l: {0x65: {l: {0x65: {l: {0x3B: {c: [8897]}}}}}}}, 0x77: {l: {0x65: {l: {0x64: {l: {0x67: {l: {0x65: {l: {0x3B: {c: [8896]}}}}}}}}}}}}}}}, 0x6B: {l: {0x61: {l: {0x72: {l: {0x6F: {l: {0x77: {l: {0x3B: {c: [10509]}}}}}}}}}}}, 0x6C: {l: {0x61: {l: {0x63: {l: {0x6B: {l: {0x6C: {l: {0x6F: {l: {0x7A: {l: {0x65: {l: {0x6E: {l: {0x67: {l: {0x65: {l: {0x3B: {c: [10731]}}}}}}}}}}}}}}}, 0x73: {l: {0x71: {l: {0x75: {l: {0x61: {l: {0x72: {l: {0x65: {l: {0x3B: {c: [9642]}}}}}}}}}}}}}, 0x74: {l: {0x72: {l: {0x69: {l: {0x61: {l: {0x6E: {l: {0x67: {l: {0x6C: {l: {0x65: {l: {0x3B: {c: [9652]}, 0x64: {l: {0x6F: {l: {0x77: {l: {0x6E: {l: {0x3B: {c: [9662]}}}}}}}}}, 0x6C: {l: {0x65: {l: {0x66: {l: {0x74: {l: {0x3B: {c: [9666]}}}}}}}}}, 0x72: {l: {0x69: {l: {0x67: {l: {0x68: {l: {0x74: {l: {0x3B: {c: [9656]}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}, 0x6E: {l: {0x6B: {l: {0x3B: {c: [9251]}}}}}}}, 0x6B: {l: {0x31: {l: {0x32: {l: {0x3B: {c: [9618]}}}, 0x34: {l: {0x3B: {c: [9617]}}}}}, 0x33: {l: {0x34: {l: {0x3B: {c: [9619]}}}}}}}, 0x6F: {l: {0x63: {l: {0x6B: {l: {0x3B: {c: [9608]}}}}}}}}}, 0x6E: {l: {0x65: {l: {0x3B: {c: [61, 8421]}, 0x71: {l: {0x75: {l: {0x69: {l: {0x76: {l: {0x3B: {c: [8801, 8421]}}}}}}}}}}}, 0x6F: {l: {0x74: {l: {0x3B: {c: [8976]}}}}}}}, 0x4E: {l: {0x6F: {l: {0x74: {l: {0x3B: {c: [10989]}}}}}}}, 0x6F: {l: {0x70: {l: {0x66: {l: {0x3B: {c: [120147]}}}}}, 0x74: {l: {0x3B: {c: [8869]}, 0x74: {l: {0x6F: {l: {0x6D: {l: {0x3B: {c: [8869]}}}}}}}}}, 0x77: {l: {0x74: {l: {0x69: {l: {0x65: {l: {0x3B: {c: [8904]}}}}}}}}}, 0x78: {l: {0x62: {l: {0x6F: {l: {0x78: {l: {0x3B: {c: [10697]}}}}}}}, 0x64: {l: {0x6C: {l: {0x3B: {c: [9488]}}}, 0x4C: {l: {0x3B: {c: [9557]}}}, 0x72: {l: {0x3B: {c: [9484]}}}, 0x52: {l: {0x3B: {c: [9554]}}}}}, 0x44: {l: {0x6C: {l: {0x3B: {c: [9558]}}}, 0x4C: {l: {0x3B: {c: [9559]}}}, 0x72: {l: {0x3B: {c: [9555]}}}, 0x52: {l: {0x3B: {c: [9556]}}}}}, 0x68: {l: {0x3B: {c: [9472]}, 0x64: {l: {0x3B: {c: [9516]}}}, 0x44: {l: {0x3B: {c: [9573]}}}, 0x75: {l: {0x3B: {c: [9524]}}}, 0x55: {l: {0x3B: {c: [9576]}}}}}, 0x48: {l: {0x3B: {c: [9552]}, 0x64: {l: {0x3B: {c: [9572]}}}, 0x44: {l: {0x3B: {c: [9574]}}}, 0x75: {l: {0x3B: {c: [9575]}}}, 0x55: {l: {0x3B: {c: [9577]}}}}}, 0x6D: {l: {0x69: {l: {0x6E: {l: {0x75: {l: {0x73: {l: {0x3B: {c: [8863]}}}}}}}}}}}, 0x70: {l: {0x6C: {l: {0x75: {l: {0x73: {l: {0x3B: {c: [8862]}}}}}}}}}, 0x74: {l: {0x69: {l: {0x6D: {l: {0x65: {l: {0x73: {l: {0x3B: {c: [8864]}}}}}}}}}}}, 0x75: {l: {0x6C: {l: {0x3B: {c: [9496]}}}, 0x4C: {l: {0x3B: {c: [9563]}}}, 0x72: {l: {0x3B: {c: [9492]}}}, 0x52: {l: {0x3B: {c: [9560]}}}}}, 0x55: {l: {0x6C: {l: {0x3B: {c: [9564]}}}, 0x4C: {l: {0x3B: {c: [9565]}}}, 0x72: {l: {0x3B: {c: [9561]}}}, 0x52: {l: {0x3B: {c: [9562]}}}}}, 0x76: {l: {0x3B: {c: [9474]}, 0x68: {l: {0x3B: {c: [9532]}}}, 0x48: {l: {0x3B: {c: [9578]}}}, 0x6C: {l: {0x3B: {c: [9508]}}}, 0x4C: {l: {0x3B: {c: [9569]}}}, 0x72: {l: {0x3B: {c: [9500]}}}, 0x52: {l: {0x3B: {c: [9566]}}}}}, 0x56: {l: {0x3B: {c: [9553]}, 0x68: {l: {0x3B: {c: [9579]}}}, 0x48: {l: {0x3B: {c: [9580]}}}, 0x6C: {l: {0x3B: {c: [9570]}}}, 0x4C: {l: {0x3B: {c: [9571]}}}, 0x72: {l: {0x3B: {c: [9567]}}}, 0x52: {l: {0x3B: {c: [9568]}}}}}}}}}, 0x70: {l: {0x72: {l: {0x69: {l: {0x6D: {l: {0x65: {l: {0x3B: {c: [8245]}}}}}}}}}}}, 0x72: {l: {0x65: {l: {0x76: {l: {0x65: {l: {0x3B: {c: [728]}}}}}}}, 0x76: {l: {0x62: {l: {0x61: {l: {0x72: {l: {0x3B: {c: [166]}}, c: [166]}}}}}}}}}, 0x73: {l: {0x63: {l: {0x72: {l: {0x3B: {c: [119991]}}}}}, 0x65: {l: {0x6D: {l: {0x69: {l: {0x3B: {c: [8271]}}}}}}}, 0x69: {l: {0x6D: {l: {0x3B: {c: [8765]}, 0x65: {l: {0x3B: {c: [8909]}}}}}}}, 0x6F: {l: {0x6C: {l: {0x62: {l: {0x3B: {c: [10693]}}}, 0x3B: {c: [92]}, 0x68: {l: {0x73: {l: {0x75: {l: {0x62: {l: {0x3B: {c: [10184]}}}}}}}}}}}}}}}, 0x75: {l: {0x6C: {l: {0x6C: {l: {0x3B: {c: [8226]}, 0x65: {l: {0x74: {l: {0x3B: {c: [8226]}}}}}}}}}, 0x6D: {l: {0x70: {l: {0x3B: {c: [8782]}, 0x45: {l: {0x3B: {c: [10926]}}}, 0x65: {l: {0x3B: {c: [8783]}, 0x71: {l: {0x3B: {c: [8783]}}}}}}}}}}}}},

package/lib/tokenization_stage/parser_simulator.js

@@ -0,0 +1,2 @@
+//TODO
+//https://github.com/yoavweiss/Blink/blob/6012e8e8453fb0f802c11d5968f6663e2010042e/Source/core/html/parser/HTMLTreeBuilderSimulator.cpp

package/lib/{preprocessor.js → tokenization_stage/preprocessor.js}

@@ -1,10 +1,10 @@
-var unicode = require('./unicode');
+var UNICODE = require('../const/unicode');
 
 //Aliases
-var $ = unicode.CODE_POINTS;
+var $ = UNICODE.CODE_POINTS;
 
 //Const
-var CARRIAGE_RETURN_NEW_LINE_REG_EXP = /\r\n?/g;
+var CARRIAGE_RETURN_NEW_LINE_REGEX = /\r\n?/g;
 
 //Utils
 
@@ -24,11 +24,11 @@ function getSurrogatePairCodePoint(cp1, cp2) {
 }
 
 //Preprocessor
-var Preprocessor = exports.Preprocessor = function (html) {
+var Preprocessor = module.exports = function (html) {
     //NOTE: All U+000D CARRIAGE RETURN (CR) characters must be converted to U+000A LINE FEED (LF) characters.
     //Any U+000A LINE FEED (LF) characters that immediately follow a U+000D CARRIAGE RETURN (CR) character
     //must be ignored.
-    this.html = html.replace(CARRIAGE_RETURN_NEW_LINE_REG_EXP, '\n');
+    this.html = html.replace(CARRIAGE_RETURN_NEW_LINE_REGEX, '\n');
 
     //NOTE: one leading U+FEFF BYTE ORDER MARK character must be ignored if any are present in the input stream.
     this.pos = this.html.charCodeAt(0) === $.BOM ? 0 : -1;

package/lib/{tokenizer.js → tokenization_stage/tokenizer.js}

@@ -1,10 +1,10 @@
-var Preprocessor = require('./preprocessor').Preprocessor,
-    unicode = require('./unicode'),
-    NAMED_ENTITY_TRIE = require('./named_entity_trie').TRIE;
+var Preprocessor = require('./preprocessor'),
+    UNICODE = require('../const/unicode'),
+    NAMED_ENTITY_TRIE = require('./named_entity_trie');
 
 //Aliases
-var $ = unicode.CODE_POINTS,
-    $$ = unicode.CODE_POINT_SEQUENCES;
+var $ = UNICODE.CODE_POINTS,
+    $$ = UNICODE.CODE_POINT_SEQUENCES;
 
 //Replacement code points for numeric entities
 var NUMERIC_ENTITY_REPLACEMENTS = {
@@ -139,7 +139,7 @@ function toAsciiLowerChar(cp) {
 }
 
 //Tokenizer
-var Tokenizer = exports.Tokenizer = function (html) {
+var Tokenizer = module.exports = function (html) {
     this.preprocessor = new Preprocessor(html);
 
     this.tokenQueue = [];
@@ -567,7 +567,7 @@ _[RCDATA_STATE] = function rcdataState(cp) {
         this.state = RCDATA_LESS_THAN_SIGN_STATE;
 
     else if (cp === $.NULL)
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
 
     else if (cp === $.EOF)
         this._emitEOFToken();
@@ -599,7 +599,7 @@ _[RAWTEXT_STATE] = function rawtextState(cp) {
         this.state = RAWTEXT_LESS_THAN_SIGN_STATE;
 
     else if (cp === $.NULL)
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
 
     else if (cp === $.EOF)
         this._emitEOFToken();
@@ -616,7 +616,7 @@ _[SCRIPT_DATA_STATE] = function scriptDataState(cp) {
         this.state = SCRIPT_DATA_LESS_THAN_SIGN_STATE;
 
     else if (cp === $.NULL)
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
 
     else if (cp === $.EOF)
         this._emitEOFToken();
@@ -630,7 +630,7 @@ _[SCRIPT_DATA_STATE] = function scriptDataState(cp) {
 //------------------------------------------------------------------
 _[PLAINTEXT_STATE] = function plaintextState(cp) {
     if (cp === $.NULL)
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
 
     else if (cp === $.EOF)
         this._emitEOFToken();
@@ -718,7 +718,7 @@ _[TAG_NAME_STATE] = function tagNameState(cp) {
         this.currentToken.tagName += toAsciiLowerChar(cp);
 
     else if (cp === $.NULL)
-        this.currentToken.tagName += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.tagName += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF)
         this._reconsumeInState(DATA_STATE);
@@ -1006,7 +1006,7 @@ _[SCRIPT_DATA_ESCAPED_STATE] = function scriptDataEscapedState(cp) {
         this.state = SCRIPT_DATA_ESCAPED_LESS_THAN_SIGN_STATE;
 
     else if (cp === $.NULL)
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
 
     else if (cp === $.EOF)
         this._reconsumeInState(DATA_STATE);
@@ -1029,7 +1029,7 @@ _[SCRIPT_DATA_ESCAPED_DASH_STATE] = function scriptDataEscapedDashState(cp) {
 
     else if (cp === $.NULL) {
         this.state = SCRIPT_DATA_ESCAPED_STATE;
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
     }
 
     else if (cp === $.EOF)
@@ -1058,7 +1058,7 @@ _[SCRIPT_DATA_ESCAPED_DASH_DASH_STATE] = function scriptDataEscapedDashDashState
 
     else if (cp === $.NULL) {
         this.state = SCRIPT_DATA_ESCAPED_STATE;
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
     }
 
     else if (cp === $.EOF)
@@ -1202,7 +1202,7 @@ _[SCRIPT_DATA_DOUBLE_ESCAPED_STATE] = function scriptDataDoubleEscapedState(cp)
     }
 
     else if (cp === $.NULL)
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
 
     else if (cp === $.EOF)
         this._reconsumeInState(DATA_STATE);
@@ -1227,7 +1227,7 @@ _[SCRIPT_DATA_DOUBLE_ESCAPED_DASH_STATE] = function scriptDataDoubleEscapedDashS
 
     else if (cp === $.NULL) {
         this.state = SCRIPT_DATA_DOUBLE_ESCAPED_STATE;
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
     }
 
     else if (cp === $.EOF)
@@ -1258,7 +1258,7 @@ _[SCRIPT_DATA_DOUBLE_ESCAPED_DASH_DASH_STATE] = function scriptDataDoubleEscaped
 
     else if (cp === $.NULL) {
         this.state = SCRIPT_DATA_DOUBLE_ESCAPED_STATE;
-        this._emitChar(unicode.REPLACEMENT_CHARACTER);
+        this._emitChar(UNICODE.REPLACEMENT_CHARACTER);
     }
 
     else if (cp === $.EOF)
@@ -1329,7 +1329,7 @@ _[BEFORE_ATTRIBUTE_NAME_STATE] = function beforeAttributeNameState(cp) {
     }
 
     else if (cp === $.NULL) {
-        this._createAttr(unicode.REPLACEMENT_CHARACTER);
+        this._createAttr(UNICODE.REPLACEMENT_CHARACTER);
         this.state = ATTRIBUTE_NAME_STATE;
     }
 
@@ -1372,7 +1372,7 @@ _[ATTRIBUTE_NAME_STATE] = function attributeNameState(cp) {
         this.currentAttr.name += toChar(cp);
 
     else if (cp === $.NULL)
-        this.currentAttr.name += unicode.REPLACEMENT_CHARACTER;
+        this.currentAttr.name += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF)
         this._reconsumeInState(DATA_STATE);
@@ -1405,7 +1405,7 @@ _[AFTER_ATTRIBUTE_NAME_STATE] = function afterAttributeNameState(cp) {
     }
 
     else if (cp === $.NULL) {
-        this._createAttr(unicode.REPLACEMENT_CHARACTER);
+        this._createAttr(UNICODE.REPLACEMENT_CHARACTER);
         this.state = ATTRIBUTE_NAME_STATE;
     }
 
@@ -1440,7 +1440,7 @@ _[BEFORE_ATTRIBUTE_VALUE_STATE] = function beforeAttributeValueState(cp) {
         this.state = ATTRIBUTE_VALUE_SINGLE_QUOTED_STATE;
 
     else if (cp === $.NULL) {
-        this.currentAttr.value += unicode.REPLACEMENT_CHARACTER;
+        this.currentAttr.value += UNICODE.REPLACEMENT_CHARACTER;
         this.state = ATTRIBUTE_VALUE_UNQUOTED_STATE;
     }
 
@@ -1477,7 +1477,7 @@ _[ATTRIBUTE_VALUE_DOUBLE_QUOTED_STATE] = function attributeValueDoubleQuotedStat
     }
 
     else if (cp === $.NULL)
-        this.currentAttr.value += unicode.REPLACEMENT_CHARACTER;
+        this.currentAttr.value += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF)
         this._reconsumeInState(DATA_STATE);
@@ -1500,7 +1500,7 @@ _[ATTRIBUTE_VALUE_SINGLE_QUOTED_STATE] = function attributeValueSingleQuotedStat
     }
 
     else if (cp === $.NULL)
-        this.currentAttr.value += unicode.REPLACEMENT_CHARACTER;
+        this.currentAttr.value += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF)
         this._reconsumeInState(DATA_STATE);
@@ -1528,7 +1528,7 @@ _[ATTRIBUTE_VALUE_UNQUOTED_STATE] = function attributeValueUnquotedState(cp) {
     }
 
     else if (cp === $.NULL)
-        this.currentAttr.value += unicode.REPLACEMENT_CHARACTER;
+        this.currentAttr.value += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.QUOTATION_MARK || cp === $.APOSTROPHE || cp === $.LESS_THAN_SIGN ||
              cp === $.EQUALS_SIGN || cp === $.GRAVE_ACCENT) {
@@ -1614,7 +1614,7 @@ _[BOGUS_COMMENT_STATE] = function bogusCommentState(cp) {
         }
 
         else {
-            this.currentToken.data += cp === $.NULL ? unicode.REPLACEMENT_CHARACTER : toChar(cp);
+            this.currentToken.data += cp === $.NULL ? UNICODE.REPLACEMENT_CHARACTER : toChar(cp);
             cp = this._consume();
         }
     }
@@ -1651,7 +1651,7 @@ _[COMMENT_START_STATE] = function commentStartState(cp) {
         this.state = COMMENT_START_DASH_STATE;
 
     else if (cp === $.NULL) {
-        this.currentToken.data += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.data += UNICODE.REPLACEMENT_CHARACTER;
         this.state = COMMENT_STATE;
     }
 
@@ -1680,7 +1680,7 @@ _[COMMENT_START_DASH_STATE] = function commentStartDashState(cp) {
 
     else if (cp === $.NULL) {
         this.currentToken.data += '-';
-        this.currentToken.data += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.data += UNICODE.REPLACEMENT_CHARACTER;
         this.state = COMMENT_STATE;
     }
 
@@ -1709,7 +1709,7 @@ _[COMMENT_STATE] = function commentState(cp) {
         this.state = COMMENT_END_DASH_STATE;
 
     else if (cp === $.NULL)
-        this.currentToken.data += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.data += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF) {
         this._emitCurrentToken();
@@ -1729,7 +1729,7 @@ _[COMMENT_END_DASH_STATE] = function commentEndDashState(cp) {
 
     else if (cp === $.NULL) {
         this.currentToken.data += '-';
-        this.currentToken.data += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.data += UNICODE.REPLACEMENT_CHARACTER;
         this.state = COMMENT_STATE;
     }
 
@@ -1762,7 +1762,7 @@ _[COMMENT_END_STATE] = function commentEndState(cp) {
 
     else if (cp === $.NULL) {
         this.currentToken.data += '--';
-        this.currentToken.data += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.data += UNICODE.REPLACEMENT_CHARACTER;
         this.state = COMMENT_STATE;
     }
 
@@ -1794,7 +1794,7 @@ _[COMMENT_END_BANG_STATE] = function commentEndBangState(cp) {
 
     else if (cp === $.NULL) {
         this.currentToken.data += '--!';
-        this.currentToken.data += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.data += UNICODE.REPLACEMENT_CHARACTER;
         this.state = COMMENT_STATE;
     }
 
@@ -1855,7 +1855,7 @@ _[BEFORE_DOCTYPE_NAME_STATE] = function beforeDoctypeNameState(cp) {
     }
 
     else if (cp === $.NULL) {
-        this._createDoctypeToken(unicode.REPLACEMENT_CHARACTER);
+        this._createDoctypeToken(UNICODE.REPLACEMENT_CHARACTER);
         this.state = DOCTYPE_NAME_STATE;
     }
 
@@ -1881,7 +1881,7 @@ _[DOCTYPE_NAME_STATE] = function doctypeNameState(cp) {
         this.currentToken.name += toAsciiLowerChar(cp);
 
     else if (cp === $.NULL)
-        this.currentToken.name += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.name += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF) {
         this.currentToken.forceQuirks = true;
@@ -2001,7 +2001,7 @@ _[DOCTYPE_PUBLIC_IDENTIFIER_DOUBLE_QUOTED_STATE] = function doctypePublicIdentif
         this.state = AFTER_DOCTYPE_PUBLIC_IDENTIFIER_STATE;
 
     else if (cp === $.NULL)
-        this.currentToken.publicId += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.publicId += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.GREATER_THAN_SIGN) {
         this.currentToken.forceQuirks = true;
@@ -2027,7 +2027,7 @@ _[DOCTYPE_PUBLIC_IDENTIFIER_SINGLE_QUOTED_STATE] = function doctypePublicIdentif
         this.state = AFTER_DOCTYPE_PUBLIC_IDENTIFIER_STATE;
 
     else if (cp === $.NULL)
-        this.currentToken.publicId += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.publicId += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.GREATER_THAN_SIGN) {
         this.currentToken.forceQuirks = true;
@@ -2198,7 +2198,7 @@ _[DOCTYPE_SYSTEM_IDENTIFIER_DOUBLE_QUOTED_STATE] = function doctypeSystemIdentif
     }
 
     else if (cp === $.NULL)
-        this.currentToken.systemId += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.systemId += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF) {
         this.currentToken.forceQuirks = true;
@@ -2224,7 +2224,7 @@ _[DOCTYPE_SYSTEM_IDENTIFIER_SINGLE_QUOTED_STATE] = function doctypeSystemIdentif
     }
 
     else if (cp === $.NULL)
-        this.currentToken.systemId += unicode.REPLACEMENT_CHARACTER;
+        this.currentToken.systemId += UNICODE.REPLACEMENT_CHARACTER;
 
     else if (cp === $.EOF) {
         this.currentToken.forceQuirks = true;