xml-tokenizer 0.0.12 → 0.0.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +55 -49
  2. package/package.json +2 -2
package/README.md CHANGED
@@ -19,7 +19,7 @@
19
19
 
20
20
  > Status: Experimental
21
21
 
22
- `xml-tokenizer` is a straightforward and typesafe XML tokenizer that streams tokens through a callback mechanism.
22
+ `xml-tokenizer` is a straightforward and typesafe XML tokenizer that streams tokens through a callback mechanism.
23
23
  The implementation is based on the [roxmltree](https://github.com/RazrFalcon/roxmltree) [`tokenizer.rs`](https://github.com/RazrFalcon/roxmltree/blob/master/src/tokenizer.rs). See the [FAQ](#-faq) why we did not embed the [roxmltree](https://github.com/RazrFalcon/roxmltree) crate as WASM.
24
24
 
25
25
  - **XML Token Stream**: Processes XML documents as a stream, emitting tokens on the fly similar to the [`SAX`](https://www.baeldung.com/java-sax-parser) approach
@@ -46,38 +46,42 @@ My goal was to develop an efficient & flexible alternative by porting [roxmltree
46
46
  ## 📖 Usage
47
47
 
48
48
  ```ts
49
- import { xmlToObject, xmlToSimplifiedObject, tokenize, select } from 'xml-tokenizer';
49
+ import { select, tokenize, xmlToObject, xmlToSimplifiedObject } from 'xml-tokenizer';
50
50
 
51
51
  // Parse XML to Javascript object without information lost (uses `tokenize` under the hood)
52
- const xmlObject = xmlToObject("<p>Hello World</p>");
52
+ const xmlObject = xmlToObject('<p>Hello World</p>');
53
53
 
54
54
  // Or, parse XML to easy to queryable Javascript object
55
- const simplifiedXmlObject = xmlToSimplifiedObject("<p>Hello World</p>");
56
-
57
- // Or, parse XML to a stream of tokens
58
- tokenize("<p>Hello World</p>", false, (token) => {
59
- switch (token.type) {
60
- case 'ElementStart':
61
- console.log('Start of element:', token);
62
- break;
63
- case 'Text':
64
- console.log('Text content:', token.text);
65
- break;
66
- // Handle other token types as needed
67
- default:
68
- console.log('Token:', token);
69
- }
55
+ const simplifiedXmlObject = xmlToSimplifiedObject('<p>Hello World</p>');
56
+
57
+ // Or, parse XML to a stream of tokens
58
+ tokenize('<p>Hello World</p>', false, (token) => {
59
+ switch (token.type) {
60
+ case 'ElementStart':
61
+ console.log('Start of element:', token);
62
+ break;
63
+ case 'Text':
64
+ console.log('Text content:', token.text);
65
+ break;
66
+ // Handle other token types as needed
67
+ default:
68
+ console.log('Token:', token);
69
+ }
70
70
  });
71
71
 
72
72
  // Or, stream only a selection of tokens
73
- select(xml, [
74
- [
75
- { axis: 'child', local: 'bookstore' },
76
- { axis: 'child', local: 'book', attributes: [{ local: 'category', value: 'COOKING' }] }
77
- ]
78
- ], (selectedToken) => {
79
- // Handle selected token
80
- });
73
+ select(
74
+ xml,
75
+ [
76
+ [
77
+ { axis: 'child', local: 'bookstore' },
78
+ { axis: 'child', local: 'book', attributes: [{ local: 'category', value: 'COOKING' }] }
79
+ ]
80
+ ],
81
+ (selectedToken) => {
82
+ // Handle selected token
83
+ }
84
+ );
81
85
  ```
82
86
 
83
87
  ### Token Types
@@ -100,35 +104,37 @@ The following token types are supported:
100
104
 
101
105
  - **Attribute Value Handling:**
102
106
  - **XML 1.0:** Attributes must be explicitly assigned a value in the format `Name="Value"`. An attribute without a value is not valid XML.
103
- - **Parser Behavior:** Attributes without an explicit value are interpreted as `true` (e.g., `<element attribute/>` is parsed as `attribute="true"`).
107
+ - **Parser Behavior:** Attributes without an explicit value are interpreted as `true` (e.g., `<element attribute/>` is parsed as `attribute="true"`).
104
108
  - **Reason**: This behavior aligns with HTML-style parsing, which was necessary to handle HTML attributes without explicit values.
105
109
 
106
110
  ## 🚀 Benchmark
107
111
 
108
- The performance of `xml-tokenizer` was benchmarked against other popular XML parsers. These tests focus on XML to object conversion and node counting. Interestingly, the version of `xml-tokenizer` imported directly from npm performed significantly better. The reason for this discrepancy is unclear, but the results seem accurate based on external testing.
112
+ The performance of `xml-tokenizer` was benchmarked against other popular XML parsers. These tests focus on XML to object conversion and node counting. Interestingly, the version of `xml-tokenizer` imported directly from npm performed significantly better. The reason for this discrepancy is unclear, but the results seem accurate based on external testing.
109
113
 
110
114
  ### XML to Object Conversion
111
115
 
112
- | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
113
- |---------------------|---------------------------------|---------------|---------------|----------------|--------------------------------|
114
- | xml-tokenizer | 46.87 | 19.47 | 24.57 | 21.33 | ±2.06% |
115
- | xml-tokenizer (dist)| 53.70 | 17.31 | 25.20 | 18.62 | ±3.28% |
116
- | xml-tokenizer (npm) | 163.00 | 5.03 | 8.50 | 6.13 | ±2.32% |
117
- | fast-xml-parser | 66.00 | 14.01 | 20.73 | 15.15 | ±3.34% |
118
- | txml | 234.52 | 3.38 | 7.61 | 4.26 | ±4.00% |
119
- | xml2js | 36.21 | 25.58 | 37.28 | 27.61 | ±4.39% |
116
+ | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
117
+ | -------------------- | ------------------------------- | ------------- | ------------- | -------------- | ------------------------------ |
118
+ | xml-tokenizer | 46.87 | 19.47 | 24.57 | 21.33 | ±2.06% |
119
+ | xml-tokenizer (dist) | 53.70 | 17.31 | 25.20 | 18.62 | ±3.28% |
120
+ | xml-tokenizer (npm) | 163.00 | 5.03 | 8.50 | 6.13 | ±2.32% |
121
+ | fast-xml-parser | 66.00 | 14.01 | 20.73 | 15.15 | ±3.34% |
122
+ | txml | 234.52 | 3.38 | 7.61 | 4.26 | ±4.00% |
123
+ | xml2js | 36.21 | 25.58 | 37.28 | 27.61 | ±4.39% |
120
124
 
121
125
  ### Node Counting
122
126
 
123
127
  | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
124
- |---------------------|---------------------------------|---------------|---------------|----------------|--------------------------------|
128
+ | ------------------- | ------------------------------- | ------------- | ------------- | -------------- | ------------------------------ |
125
129
  | xml-tokenizer | 53.03 | 18.30 | 19.45 | 18.86 | ±0.81% |
126
130
  | xml-tokenizer (npm) | 166.61 | 5.62 | 7.16 | 6.00 | ±0.88% |
127
131
  | saxen | 500.99 | 1.83 | 4.79 | 2.00 | ±1.52% |
128
132
  | sax | 64.44 | 14.96 | 16.34 | 15.52 | ±0.67% |
129
133
 
130
134
  ### Running the Benchmarks
135
+
131
136
  The benchmarks can be found in the [`__tests__`](https://github.com/builder-group/community/tree/develop/packages/xml-tokenizer/src/__tests__) directory and can be executed by running:
137
+
132
138
  ```bash
133
139
  pnpm run bench
134
140
  ```
@@ -143,11 +149,11 @@ Calling a TypeScript function from Rust on every token event (`wasmMix` benchmar
143
149
 
144
150
  The `roxmltree` package with the Rust implementation can be found in the `_deprecated` folder ([`packages/_deprecated/roxmltree_wasm`](https://github.com/builder-group/community/tree/develop/packages/_deprecated/roxmltree_wasm)).
145
151
 
146
- | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
147
- |---------------------|---------------------------------|---------------|---------------|----------------|--------------------------------|
148
- | roxmltree:text | 67.12 | 14.33 | 83.29 | 80.08 | ±1.27% |
149
- | roxmltree:wasmMix | 28.17 | 34.83 | 36.71 | 35.49 | ±0.91% |
150
- | roxmltree:wasm | 109.30 | 8.30 | 13.16 | 9.15 | ±3.31% |
152
+ | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
153
+ | ----------------- | ------------------------------- | ------------- | ------------- | -------------- | ------------------------------ |
154
+ | roxmltree:text | 67.12 | 14.33 | 83.29 | 80.08 | ±1.27% |
155
+ | roxmltree:wasmMix | 28.17 | 34.83 | 36.71 | 35.49 | ±0.91% |
156
+ | roxmltree:wasm | 109.30 | 8.30 | 13.16 | 9.15 | ±3.31% |
151
157
 
152
158
  ### Why ported `tokenizer.rs` to TypeScript?
153
159
 
@@ -159,16 +165,16 @@ We removed the byte-based implementation to enhance maintainability and because
159
165
 
160
166
  Decoding `Uint8Array` snippets to JavaScript strings is frequently necessary, nearly on every token event. This decoding process is slow, making this approach less efficient than working directly with strings.
161
167
 
162
- | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
163
- |---------------------|---------------------------------|---------------|---------------|----------------|--------------------------------|
164
- | roxmltree:text | 67.12 | 14.33 | 83.29 | 80.08 | ±1.27% |
165
- | roxmltree:byte | 12.48 | 78.65 | 16.45 | 14.90 | ±1.15% |
168
+ | Parser | Operations per Second (ops/sec) | Min Time (ms) | Max Time (ms) | Mean Time (ms) | Relative Margin of Error (rme) |
169
+ | -------------- | ------------------------------- | ------------- | ------------- | -------------- | ------------------------------ |
170
+ | roxmltree:text | 67.12 | 14.33 | 83.29 | 80.08 | ±1.27% |
171
+ | roxmltree:byte | 12.48 | 78.65 | 16.45 | 14.90 | ±1.15% |
166
172
 
167
173
  The `roxmltree` package with the Byte-Based implementation can be found in the `_deprecated` folder ([`packages/_deprecated/roxmltree_byte-only`](https://github.com/builder-group/community/tree/develop/packages/_deprecated/roxmltree_byte-only)).
168
174
 
169
175
  ### Why not use a Generator?
170
176
 
171
- While generators can improve developer experience, they introduce significant performance overhead. Our benchmarks show that using a generator dramatically increases the execution time compared to the callback approach. Given our focus on performance, we chose to maintain the callback implementation.
177
+ While generators can improve developer experience, they introduce significant performance overhead. Our benchmarks show that using a generator dramatically increases the execution time compared to the callback approach. Given our focus on performance, we chose to maintain the callback implementation.
172
178
 
173
179
  See [Generator vs Iterator vs Callback](https://observablehq.com/@domoritz/yield-vs-iterator-vs-callback) for more details.
174
180
 
@@ -190,8 +196,8 @@ See [Generator vs Iterator vs Callback](https://observablehq.com/@domoritz/yield
190
196
 
191
197
  [Benchmark implementation in Vanilla Profiler](https://github.com/builder-group/monorepo/tree/develop/examples/xml-tokenizer/vanilla/profiler)
192
198
 
193
-
194
199
  ## 💡 Resources
200
+
195
201
  - [How I developed the fastest XML parser](https://tnickel.de/2020/08/30/2020-08-how-the-fastest-xml-parser-is-build/)
196
202
  - [txml](https://github.com/TobiasNickel/tXml)
197
- - [roxmltree](https://github.com/RazrFalcon/roxmltree)
203
+ - [roxmltree](https://github.com/RazrFalcon/roxmltree)
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "xml-tokenizer",
3
- "version": "0.0.12",
3
+ "version": "0.0.13",
4
4
  "description": "Straightforward and typesafe XML tokenizer that streams tokens through a callback mechanism",
5
5
  "private": false,
6
6
  "source": "./src/index.ts",
@@ -28,7 +28,7 @@
28
28
  "saxen": "^10.0.0",
29
29
  "txml": "^5.1.1",
30
30
  "xml2js": "^0.6.2",
31
- "@blgc/config": "0.0.22"
31
+ "@blgc/config": "0.0.23"
32
32
  },
33
33
  "files": [
34
34
  "dist",