xml-twig 1.3.11 → 1.3.13

package/README.md

@@ -18,7 +18,7 @@ When you need to read a XML file, then you have two principles:
 This module tries to combine both principles. The XML document can be read in chunks and within a chunk you have all the nice features and functions you know from a DOM based parser.
 
 ## Dependencies
-XML documents are read either with [sax](https://www.npmjs.com/package/sax), [node-expat](https://www.npmjs.com/package/node-expat) or [saxophone](https://www.npmjs.com/package/saxophone) parser. More parser may be added in future releases. By default the `sax` parser is used.
+XML documents are read either with [sax](https://www.npmjs.com/package/sax), [node-expat](https://www.npmjs.com/package/node-expat) or [saxophone](https://www.npmjs.com/package/saxophone) parser. More parser may be added in future releases. By default the `sax` parser is used. However, I clearly recommend using the `node-expat` parser. All other parsers I tested, are not compliant to XML standards.
 
 **NOTE: The `node-expat` and `saxophone` modules are not automatically installed with this module. Install the parser by yourself, if you like to use it**
 
@@ -33,7 +33,7 @@ npm install node-expat
 npm install saxophone
 
 ```
-In my tests I parsed a 900 MB big XML file, the `node-expat` is faster than `sax` (node-expat: around 2:30 Minutes, sax: around 3:40 Minutes). However, you may run into problems when you try to install the `node-expat` parser. That's the reason why `node-expat` parser is not installed automatically. `saxophone` is even a little faster (around 2:10 Minutes) than `node-expat`, however `saxophone` lacks some functions and is not fully compliant to XML standards.
+In my tests I parsed a 900 MB big XML file, the `node-expat` is faster than `sax` (node-expat: around 2:30 Minutes, sax: around 3:40 Minutes). However, you may run into problems when you try to install the `node-expat` parser. That's the reason why `node-expat` parser is not installed automatically. `saxophone` is even a little faster (around 2:10 Minutes) than `node-expat`.
 
 ## How to use it
 
@@ -69,7 +69,7 @@ API Documentation: see [Twig](./doc/twig.md)
    If you prefer [events](https://nodejs.org/api/events.html), then use `event` property instead of `function` in handler declaration:
 
    ```js
-   const parser = twig.createParser({ tag: twig.Root, event: 'rootElement' }, { method: 'sax' })
+   const parser = twig.createParser({ tag: twig.Root, event: 'rootElement' }, { method: 'expat' })
    fs.createReadStream(`${__dirname}/bookstore.xml`).pipe(parser)
 
    parser.on('rootElement', (elt) => {
@@ -96,6 +96,7 @@ API Documentation: see [Twig](./doc/twig.md)
       { tag: 'book', function: bookHandler },
       { tag: 'ebook', function: bookHandler }
    ];
+   handle_book = [ { tag: ['book', 'ebook'], function: bookHandler } ];
    handle_book = { tag: /book$/, function: bookHandler };
    handle_book = [{
       tag: function(name, elt) { return name.endsWith('book') },
@@ -392,7 +393,12 @@ This `xml-twig` module focus on reading a XML files. In principle it would be po
 
 Accessing Twig-Elements by [XML-Path](https://www.w3.org/TR/xpath/) language is not supported. One reason it, the `Twig` class models more an [Element](https://www.w3schools.com/xml/xml_elements.asp) rather than a [Node](https://www.w3schools.com/xml/dom_nodes.asp) which would be more generic.
 
-Despite [W3C Recommendations](https://www.w3.org/TR/xml/#charencoding) ("All XML processors MUST be able to read entities in both the UTF-8 and UTF-16 encodings"), the `sax` and `saxophone` parsers do not support UTF-16 encodings. When you have a XML-File encoded in UTF-16, then you must use the `expat` parser. 
+As already mentioned above, I recommend the `expat` parser. The other parser may work for your purpose, however they have several limitations and bugs:
+
+- `sax` and `saxophone` do not support UTF-16 encoding. I did not test other encodings, because [W3C Recommendations](https://www.w3.org/TR/xml/#charencoding) defines only UTF-8 and UTF-16 as required
+- `sax` misinterpret character entities
+- `saxophone` fails on `<!DOCTYPE>` element
+- Properties `currentLine` and `currentColumn` are not available with `saxophone` 
 
 
 

package/doc/twig.md

@@ -53,11 +53,11 @@
 Element can be specified as string, Regular Expression, custom function, <code>Twig.Root</code> or <code>Twig.Any</code><br> 
 You can specify a <code>function</code> or a <code>event</code> name</p>
 </dd>
-<dt><a href="#HandlerCondition">HandlerCondition</a> : <code>string</code> | <code>RegExp</code> | <code><a href="#HandlerConditionFilter">HandlerConditionFilter</a></code> | <code><a href="#Root">Root</a></code> | <code><a href="#Any">Any</a></code> | <code>undefined</code></dt>
+<dt><a href="#HandlerCondition">HandlerCondition</a> : <code>string</code> | <code>Array.&lt;string&gt;</code> | <code>RegExp</code> | <code><a href="#HandlerConditionFilter">HandlerConditionFilter</a></code> | <code><a href="#Root">Root</a></code> | <code><a href="#Any">Any</a></code></dt>
 <dd><p>Condition to specify when handler shall be called<br> </p>
 <ul>
-<li>If <code>undefined</code>, then all elements are returned.<br> </li>
 <li>If <code>string</code> then the element name must be equal to the string</li>
+<li>If <code>string[]</code> then the element name must be included in string array</li>
 <li>If <code>RegExp</code> then the element name must match the Regular Expression</li>
 <li>If <a href="#HandlerConditionFilter">HandlerConditionFilter</a> then function must return <code>true</code></li>
 <li>Use <code>Twig.Root</code> to call the handler on root element, i.e. when the end of document is reached</li>
@@ -141,6 +141,7 @@ You can specify a <code>function</code> or a <code>event</code> name</p>
     * [.pinned](#Twig+pinned) ⇒ <code>boolean</code>
     * [.close](#Twig+close)
     * [.debug](#Twig+debug) ⇒ <code>string</code>
+    * [.toString](#Twig+toString) ⇒ <code>string</code>
     * [.addChild](#Twig+addChild) ℗
     * [.writer](#Twig+writer) ⇒ [<code>XMLWriter</code>](https://www.npmjs.com/package/xml-writer)
     * [.attr](#Twig+attr) ⇒ <code>string</code> \| <code>number</code> \| <code>object</code>
@@ -360,6 +361,13 @@ XML-Twig for dummies :-)
 
 **Kind**: instance property of [<code>Twig</code>](#Twig)  
 **Returns**: <code>string</code> - The XML-Tree which is currently available in RAM - no valid XML Structure  
+<a name="Twig+toString"></a>
+
+### twig.toString ⇒ <code>string</code>
+Returns XML string of the element
+
+**Kind**: instance property of [<code>Twig</code>](#Twig)  
+**Returns**: <code>string</code> - The XML-Element as string  
 <a name="Twig+addChild"></a>
 
 ### twig.addChild ℗
@@ -758,6 +766,7 @@ Common function to filter Twig element
     * [.pinned](#Twig+pinned) ⇒ <code>boolean</code>
     * [.close](#Twig+close)
     * [.debug](#Twig+debug) ⇒ <code>string</code>
+    * [.toString](#Twig+toString) ⇒ <code>string</code>
     * [.addChild](#Twig+addChild) ℗
     * [.writer](#Twig+writer) ⇒ [<code>XMLWriter</code>](https://www.npmjs.com/package/xml-writer)
     * [.attr](#Twig+attr) ⇒ <code>string</code> \| <code>number</code> \| <code>object</code>
@@ -977,6 +986,13 @@ XML-Twig for dummies :-)
 
 **Kind**: instance property of [<code>Twig</code>](#Twig)  
 **Returns**: <code>string</code> - The XML-Tree which is currently available in RAM - no valid XML Structure  
+<a name="Twig+toString"></a>
+
+### twig.toString ⇒ <code>string</code>
+Returns XML string of the element
+
+**Kind**: instance property of [<code>Twig</code>](#Twig)  
+**Returns**: <code>string</code> - The XML-Element as string  
 <a name="Twig+addChild"></a>
 
 ### twig.addChild ℗
@@ -1494,8 +1510,8 @@ Reference to handler functions for Twig objects.<br>
 
 <a name="HandlerCondition"></a>
 
-## HandlerCondition : <code>string</code> \| <code>RegExp</code> \| [<code>HandlerConditionFilter</code>](#HandlerConditionFilter) \| [<code>Root</code>](#Root) \| [<code>Any</code>](#Any) \| <code>undefined</code>
-Condition to specify when handler shall be called<br> 
- If `undefined`, then all elements are returned.<br> 
- If `string` then the element name must be equal to the string
- If `RegExp` then the element name must match the Regular Expression
- If [HandlerConditionFilter](#HandlerConditionFilter) then function must return `true`
- Use `Twig.Root` to call the handler on root element, i.e. when the end of document is reached
- Use `Twig.Any` to call the handler on every element
- If `string` then the element name must be equal to the string
- If `string[]` then the element name must be included in string array
- If `RegExp` then the element name must match the Regular Expression
- If [HandlerConditionFilter](#HandlerConditionFilter) then function must return `true`
- Use `Twig.Root` to call the handler on root element, i.e. when the end of document is reached
- Use `Twig.Any` to call the handler on every element
+## HandlerCondition : <code>string</code> \| <code>Array.&lt;string&gt;</code> \| <code>RegExp</code> \| [<code>HandlerConditionFilter</code>](#HandlerConditionFilter) \| [<code>Root</code>](#Root) \| [<code>Any</code>](#Any)
+Condition to specify when handler shall be called<br> 
 
 **Kind**: global typedef  
 <a name="HandlerFunction"></a>

package/package.json

@@ -5,15 +5,15 @@
   },
   "name": "xml-twig",
   "description": "Node module for processing huge XML documents in tree mode",
-  "version": "1.3.11",
+  "version": "1.3.13",
   "main": "twig.js",
   "directories": {
     "doc": "doc"
   },
   "devDependencies": {
-    "jsdoc-to-markdown": "^8.0.0",
+    "jsdoc-to-markdown": "^8.0.1",
     "luxon": "^3.4.4",
-    "node-expat": "^2.4.0",
+    "node-expat": "^2.4.1",
     "saxophone": "^0.8.0"
   },
   "scripts": {

package/samples/memory-test.js

@@ -6,6 +6,7 @@ let Entry = 0;
 
 let parser = twig.createParser([{ tag: 'Entry', function: EntryHandler }], { method: 'expat' })
 // http://aiweb.cs.washington.edu/research/projects/xmltk/xmldata/data/SwissProt/SwissProt.xml.gz
+// For more files see http://aiweb.cs.washington.edu/research/projects/xmltk/xmldata/www/repository.html
 let reader = fs.createReadStream(`SwissProt.xml`);
 reader.pipe(parser);
 

package/twig.js

@@ -1,5 +1,5 @@
 const SAX = 'sax';
-const EXPAT = 'expat';
+const EXPAT = ['expat', 'node-expat'];
 const SAXOPHONE = 'saxophone';
 
 let tree;
@@ -35,10 +35,13 @@ let current;
 * @see {@link https://www.npmjs.com/package/libxmljs|libxmljs}
 */
 
-/**
-* @external sax-wasm
-* @todo Not yet implemented, maybe later. Not sure if `WritableStream` is supported
-* @see {@link https://www.npmjs.com/package/sax-wasm|sax-wasm}
+/*
+* Other parsers I had a look at:
+* {@link https://www.npmjs.com/package/sax-wasm|sax-wasm}: not a 'stream.Writable'
+* {@link https://www.npmjs.com/package/@rubensworks/saxes|saxes}: not a 'stream.Writable'
+* {@link https://www.npmjs.com/package/node-xml-stream|node-xml-stream}: should work, but not implemented
+* {@link https://www.npmjs.com/package/saxes-stream|saxes-stream}: not a 'stream.Writable'
+* {@link https://www.npmjs.com/package/xml-streamer|xml-streamer}: based on 'node-expat', does not add any benefit
 */
 
 
@@ -83,13 +86,13 @@ const Any = new AnyHandler();
 
 /**
 * Condition to specify when handler shall be called<br> 
-* - If `undefined`, then all elements are returned.<br> 
 * - If `string` then the element name must be equal to the string
+* - If `string[]` then the element name must be included in string array
 * - If `RegExp` then the element name must match the Regular Expression
 * - If [HandlerConditionFilter](#HandlerConditionFilter) then function must return `true`
 * - Use `Twig.Root` to call the handler on root element, i.e. when the end of document is reached
 * - Use `Twig.Any` to call the handler on every element
-* @typedef {string|RegExp|HandlerConditionFilter|Root|Any|undefined} HandlerCondition 
+* @typedef {string|string[]|RegExp|HandlerConditionFilter|Root|Any} HandlerCondition 
 */
 
 /**
@@ -130,7 +133,6 @@ const Any = new AnyHandler();
 * @returns {external:sax|external:node-expat|external:saxophone} The parser Object
 */
 
-
 /**
 * Create a new Twig parser
 * @param {TwigHandler|TwigHandler[]} handler - Object or array of element specification and function to handle elements
@@ -143,12 +145,11 @@ function createParser(handler, options = {}) {
    let parser;
    let namespaces = {};
 
-   if (options.partial) {
-      const handle1 = Array.isArray(handler) ? handler : [handler];
-      let any = handle1.find(x => x.tag instanceof AnyHandler);
-      if (any !== undefined)
-         console.warn(`Using option '{ partial: true }' and handler '{ tag: Any, function: ${any.function.toString()} }' does not make much sense`);
-   }
+   const handlerCheck = Array.isArray(handler) ? handler : [handler];
+   if (handlerCheck.find(x => x.tag === undefined) != null || handlerCheck.find(x => x.tag.length == 0) != null)
+      throw new ReferenceError(`'handler.tag' is not defined`);
+   if (options.partial && handlerCheck.find(x => x.tag instanceof AnyHandler) != null)
+      console.warn(`Using option '{ partial: true }' and handler '{ tag: Any, function: ${any.function.toString()} }' does not make much sense`);
 
    // `parser.on("...", err =>  {...}` does not work, because I need access to 'this'
    if (options.method === SAX) {
@@ -221,7 +222,7 @@ function createParser(handler, options = {}) {
          parser.emit("close");
       });
 
-   } else if (options.method === EXPAT) {
+   } else if (EXPAT.includes(options.method)) {
       parser = require("node-expat").createParser();
       Object.defineProperty(parser, 'currentLine', {
          enumerable: true,
@@ -413,6 +414,9 @@ function onStart(binds, node, attrs) {
                if (typeof hndl.tag === 'string' && name === hndl.tag) {
                   elt.pin();
                   break;
+               } else if (typeof Array.isArray(hndl.tag) && hndl.tag.includes(name)) {
+                  elt.pin();
+                  break;
                } else if (hndl.tag instanceof RegExp && hndl.tag.test(name)) {
                   elt.pin();
                   break;
@@ -426,7 +430,7 @@ function onStart(binds, node, attrs) {
    }
 
    if (options.xmlns) {
-      if ([EXPAT, SAXOPHONE].includes(options.method)) {
+      if (EXPAT.concat(SAXOPHONE).includes(options.method)) {
          for (let key of Object.keys(attrs).filter(x => x.startsWith('xmlns:')))
             namespaces[key.split(':')[1]] = attrs[key];
       }
@@ -466,6 +470,10 @@ function onClose(handler, options, name) {
          if (typeof hndl.function === 'function') hndl.function(tree);
          if (typeof hndl.event === 'string') parser.emit(hndl.event, tree);
          purge = false;
+      } else if (typeof Array.isArray(hndl.tag) && hndl.tag.includes(name)) {
+         if (typeof hndl.function === 'function') hndl.function(current ?? tree);
+         if (typeof hndl.event === 'string') parser.emit(hndl.event, current ?? tree);
+         purge = false;
       } else if (typeof hndl.tag === 'string' && name === hndl.tag) {
          if (typeof hndl.function === 'function') hndl.function(current ?? tree);
          if (typeof hndl.event === 'string') parser.emit(hndl.event, current ?? tree);