binja 0.4.2 → 0.4.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -58
- package/dist/cli.js +44 -354
- package/dist/filters/index.d.ts.map +1 -1
- package/dist/index.js +105 -390
- package/dist/lexer/hybrid.d.ts +1 -1
- package/dist/lexer/hybrid.d.ts.map +1 -1
- package/dist/lexer/index.d.ts.map +1 -1
- package/dist/native/index.d.ts +10 -1
- package/dist/native/index.d.ts.map +1 -1
- package/dist/native/index.js +76 -1
- package/dist/runtime/context.d.ts.map +1 -1
- package/dist/runtime/index.d.ts.map +1 -1
- package/native/darwin-arm64/libbinja.dylib +0 -0
- package/native/darwin-x64/libbinja.dylib +0 -0
- package/native/linux-arm64/libbinja.so +0 -0
- package/native/linux-x64/libbinja.so +0 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,20 +1,19 @@
|
|
|
1
1
|
<h1 align="center">binja</h1>
|
|
2
2
|
|
|
3
3
|
<p align="center">
|
|
4
|
-
<strong>High-performance Jinja2/Django template engine for Bun
|
|
4
|
+
<strong>High-performance Jinja2/Django template engine for Bun - 2-4x faster than Nunjucks</strong>
|
|
5
5
|
</p>
|
|
6
6
|
|
|
7
7
|
<p align="center">
|
|
8
8
|
<a href="#installation">Installation</a> •
|
|
9
9
|
<a href="#quick-start">Quick Start</a> •
|
|
10
10
|
<a href="#features">Features</a> •
|
|
11
|
-
<a href="#
|
|
11
|
+
<a href="#benchmarks">Benchmarks</a> •
|
|
12
12
|
<a href="#filters">Filters</a>
|
|
13
13
|
</p>
|
|
14
14
|
|
|
15
15
|
<p align="center">
|
|
16
16
|
<img src="https://img.shields.io/badge/bun-%23000000.svg?style=for-the-badge&logo=bun&logoColor=white" alt="Bun" />
|
|
17
|
-
<img src="https://img.shields.io/badge/Zig-F7A41D?style=for-the-badge&logo=zig&logoColor=white" alt="Zig Native" />
|
|
18
17
|
<img src="https://img.shields.io/badge/TypeScript-007ACC?style=for-the-badge&logo=typescript&logoColor=white" alt="TypeScript" />
|
|
19
18
|
<img src="https://img.shields.io/badge/Django-092E20?style=for-the-badge&logo=django&logoColor=white" alt="Django Compatible" />
|
|
20
19
|
<img src="https://img.shields.io/badge/license-BSD--3--Clause-blue.svg?style=for-the-badge" alt="BSD-3-Clause License" />
|
|
@@ -26,7 +25,7 @@
|
|
|
26
25
|
|
|
27
26
|
| Feature | Binja | Other JS engines |
|
|
28
27
|
|---------|-----------|------------------|
|
|
29
|
-
| **
|
|
28
|
+
| **Runtime Performance** | ✅ 2-4x faster | ❌ |
|
|
30
29
|
| **AOT Compilation** | ✅ 160x faster | ❌ |
|
|
31
30
|
| Django DTL Compatible | ✅ 100% | ❌ Partial |
|
|
32
31
|
| Jinja2 Compatible | ✅ Full | ⚠️ Limited |
|
|
@@ -43,70 +42,34 @@
|
|
|
43
42
|
|
|
44
43
|
## Benchmarks
|
|
45
44
|
|
|
46
|
-
Tested on Mac Studio M1 Max, Bun 1.3.5
|
|
45
|
+
Tested on Mac Studio M1 Max, Bun 1.3.5.
|
|
47
46
|
|
|
48
47
|
### Two Rendering Modes
|
|
49
48
|
|
|
50
49
|
| Mode | Function | Best For | vs Nunjucks |
|
|
51
50
|
|------|----------|----------|-------------|
|
|
52
|
-
| **Runtime** | `render()` | Development | **
|
|
51
|
+
| **Runtime** | `render()` | Development | **2-4x faster** |
|
|
53
52
|
| **AOT** | `compile()` | Production | **160x faster** |
|
|
54
53
|
|
|
55
|
-
### Performance
|
|
54
|
+
### Runtime Performance (vs Nunjucks)
|
|
56
55
|
|
|
57
|
-
| Benchmark |
|
|
58
|
-
|
|
59
|
-
| Simple Template |
|
|
60
|
-
| Complex Template |
|
|
61
|
-
|
|
|
62
|
-
|
|
|
63
|
-
| Conditionals |
|
|
64
|
-
|
|
|
56
|
+
| Benchmark | binja | Nunjucks | Speedup |
|
|
57
|
+
|-----------|-------|----------|---------|
|
|
58
|
+
| Simple Template | 371K ops/s | 96K ops/s | **3.9x** |
|
|
59
|
+
| Complex Template | 44K ops/s | 23K ops/s | **2.0x** |
|
|
60
|
+
| Multiple Filters | 246K ops/s | 63K ops/s | **3.9x** |
|
|
61
|
+
| Nested Loops | 76K ops/s | 26K ops/s | **3.0x** |
|
|
62
|
+
| Conditionals | 84K ops/s | 25K ops/s | **3.4x** |
|
|
63
|
+
| HTML Escaping | 985K ops/s | 242K ops/s | **4.1x** |
|
|
64
|
+
| Large Dataset | 9.6K ops/s | 6.6K ops/s | **1.5x** |
|
|
65
65
|
|
|
66
|
-
###
|
|
67
|
-
|
|
68
|
-
```bash
|
|
69
|
-
bun run full-benchmark.ts
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
---
|
|
73
|
-
|
|
74
|
-
## Native Acceleration
|
|
75
|
-
|
|
76
|
-
Binja includes a **native Zig lexer** that provides **7x faster** tokenization through Bun's FFI. The native library is automatically used when available.
|
|
77
|
-
|
|
78
|
-
### Supported Platforms
|
|
79
|
-
|
|
80
|
-
| Platform | Architecture | Status |
|
|
81
|
-
|----------|--------------|--------|
|
|
82
|
-
| macOS | Apple Silicon (arm64) | ✅ |
|
|
83
|
-
| macOS | Intel (x64) | ✅ |
|
|
84
|
-
| Linux | x64 | ✅ |
|
|
85
|
-
| Linux | arm64 | ✅ |
|
|
86
|
-
|
|
87
|
-
### Check Native Status
|
|
88
|
-
|
|
89
|
-
```typescript
|
|
90
|
-
import { isNativeAccelerated } from 'binja/lexer'
|
|
91
|
-
|
|
92
|
-
console.log('Using native Zig:', isNativeAccelerated())
|
|
93
|
-
// Output: Using native Zig: true
|
|
94
|
-
```
|
|
95
|
-
|
|
96
|
-
### Performance Comparison
|
|
97
|
-
|
|
98
|
-
| Template Size | TypeScript Lexer | Zig Native | Speedup |
|
|
99
|
-
|--------------|------------------|------------|---------|
|
|
100
|
-
| Small (100B) | 290K ops/s | 1.2M ops/s | **4x** |
|
|
101
|
-
| Medium (1KB) | 85K ops/s | 450K ops/s | **5x** |
|
|
102
|
-
| Large (10KB) | 12K ops/s | 85K ops/s | **7x** |
|
|
66
|
+
### AOT Compilation (Maximum Performance)
|
|
103
67
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
- ✅ Error handling with line numbers
|
|
68
|
+
| Benchmark | binja AOT | binja Runtime | Speedup |
|
|
69
|
+
|-----------|-----------|---------------|---------|
|
|
70
|
+
| Simple Template | **14.3M ops/s** | 371K ops/s | 39x |
|
|
71
|
+
| Complex Template | **1.07M ops/s** | 44K ops/s | 24x |
|
|
72
|
+
| Nested Loops | **1.75M ops/s** | 76K ops/s | 23x |
|
|
110
73
|
|
|
111
74
|
---
|
|
112
75
|
|
package/dist/cli.js
CHANGED
|
@@ -1,286 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env bun
|
|
2
2
|
// @bun
|
|
3
|
-
var __defProp = Object.defineProperty;
|
|
4
|
-
var __getOwnPropNames = Object.getOwnPropertyNames;
|
|
5
|
-
var __getOwnPropDesc = Object.getOwnPropertyDescriptor;
|
|
6
|
-
var __hasOwnProp = Object.prototype.hasOwnProperty;
|
|
7
|
-
var __moduleCache = /* @__PURE__ */ new WeakMap;
|
|
8
|
-
var __toCommonJS = (from) => {
|
|
9
|
-
var entry = __moduleCache.get(from), desc;
|
|
10
|
-
if (entry)
|
|
11
|
-
return entry;
|
|
12
|
-
entry = __defProp({}, "__esModule", { value: true });
|
|
13
|
-
if (from && typeof from === "object" || typeof from === "function")
|
|
14
|
-
__getOwnPropNames(from).map((key) => !__hasOwnProp.call(entry, key) && __defProp(entry, key, {
|
|
15
|
-
get: () => from[key],
|
|
16
|
-
enumerable: !(desc = __getOwnPropDesc(from, key)) || desc.enumerable
|
|
17
|
-
}));
|
|
18
|
-
__moduleCache.set(from, entry);
|
|
19
|
-
return entry;
|
|
20
|
-
};
|
|
21
|
-
var __export = (target, all) => {
|
|
22
|
-
for (var name in all)
|
|
23
|
-
__defProp(target, name, {
|
|
24
|
-
get: all[name],
|
|
25
|
-
enumerable: true,
|
|
26
|
-
configurable: true,
|
|
27
|
-
set: (newValue) => all[name] = () => newValue
|
|
28
|
-
});
|
|
29
|
-
};
|
|
30
|
-
var __esm = (fn, res) => () => (fn && (res = fn(fn = 0)), res);
|
|
31
|
-
|
|
32
|
-
// src/native/index.ts
|
|
33
|
-
var exports_native = {};
|
|
34
|
-
__export(exports_native, {
|
|
35
|
-
tokenizeCount: () => tokenizeCount,
|
|
36
|
-
tokenize: () => tokenize,
|
|
37
|
-
nativeVersion: () => nativeVersion,
|
|
38
|
-
isNativeAvailable: () => isNativeAvailable,
|
|
39
|
-
TokenType: () => TokenType,
|
|
40
|
-
NativeLexer: () => NativeLexer
|
|
41
|
-
});
|
|
42
|
-
import { dlopen, FFIType, ptr, CString } from "bun:ffi";
|
|
43
|
-
import { join, basename } from "path";
|
|
44
|
-
import { existsSync } from "fs";
|
|
45
|
-
function getLibraryPath() {
|
|
46
|
-
const platform = process.platform;
|
|
47
|
-
const arch = process.arch;
|
|
48
|
-
const libExt = platform === "darwin" ? "dylib" : platform === "win32" ? "dll" : "so";
|
|
49
|
-
const libName = `libbinja.${libExt}`;
|
|
50
|
-
const dirName = basename(import.meta.dir);
|
|
51
|
-
const projectRoot = dirName === "native" ? join(import.meta.dir, "..", "..") : join(import.meta.dir, "..");
|
|
52
|
-
const searchPaths = [
|
|
53
|
-
join(projectRoot, "native", `${platform}-${arch}`, libName),
|
|
54
|
-
join(projectRoot, "native", libName),
|
|
55
|
-
join(projectRoot, "zig-native", "zig-out", "lib", libName),
|
|
56
|
-
join(projectRoot, "zig-native", libName),
|
|
57
|
-
join(import.meta.dir, libName)
|
|
58
|
-
];
|
|
59
|
-
for (const p of searchPaths) {
|
|
60
|
-
if (existsSync(p)) {
|
|
61
|
-
return p;
|
|
62
|
-
}
|
|
63
|
-
}
|
|
64
|
-
return null;
|
|
65
|
-
}
|
|
66
|
-
function loadLibrary() {
|
|
67
|
-
if (_loadAttempted) {
|
|
68
|
-
return _lib;
|
|
69
|
-
}
|
|
70
|
-
_loadAttempted = true;
|
|
71
|
-
const libPath = getLibraryPath();
|
|
72
|
-
if (!libPath) {
|
|
73
|
-
console.warn("[binja] Native library not found, using pure JS fallback");
|
|
74
|
-
return null;
|
|
75
|
-
}
|
|
76
|
-
try {
|
|
77
|
-
_lib = dlopen(libPath, symbols);
|
|
78
|
-
_nativeAvailable = true;
|
|
79
|
-
return _lib;
|
|
80
|
-
} catch (e) {
|
|
81
|
-
console.warn(`[binja] Failed to load native library: ${e}`);
|
|
82
|
-
return null;
|
|
83
|
-
}
|
|
84
|
-
}
|
|
85
|
-
function isNativeAvailable() {
|
|
86
|
-
loadLibrary();
|
|
87
|
-
return _nativeAvailable;
|
|
88
|
-
}
|
|
89
|
-
function nativeVersion() {
|
|
90
|
-
const lib = loadLibrary();
|
|
91
|
-
if (!lib)
|
|
92
|
-
return null;
|
|
93
|
-
const versionPtr = lib.symbols.binja_version();
|
|
94
|
-
if (!versionPtr)
|
|
95
|
-
return null;
|
|
96
|
-
return new CString(versionPtr).toString();
|
|
97
|
-
}
|
|
98
|
-
function tokenizeCount(source) {
|
|
99
|
-
if (source.length === 0) {
|
|
100
|
-
return 1;
|
|
101
|
-
}
|
|
102
|
-
const lib = loadLibrary();
|
|
103
|
-
if (!lib) {
|
|
104
|
-
throw new Error("Native library not available");
|
|
105
|
-
}
|
|
106
|
-
const bytes = new TextEncoder().encode(source);
|
|
107
|
-
return Number(lib.symbols.binja_tokenize_count(ptr(bytes), bytes.length));
|
|
108
|
-
}
|
|
109
|
-
function tokenize(source) {
|
|
110
|
-
const lexer = new NativeLexer(source);
|
|
111
|
-
try {
|
|
112
|
-
return lexer.getAllTokens();
|
|
113
|
-
} finally {
|
|
114
|
-
lexer.free();
|
|
115
|
-
}
|
|
116
|
-
}
|
|
117
|
-
var TokenType, symbols, _lib = null, _loadAttempted = false, _nativeAvailable = false, NativeLexer;
|
|
118
|
-
var init_native = __esm(() => {
|
|
119
|
-
TokenType = {
|
|
120
|
-
TEXT: 0,
|
|
121
|
-
VAR_START: 1,
|
|
122
|
-
VAR_END: 2,
|
|
123
|
-
BLOCK_START: 3,
|
|
124
|
-
BLOCK_END: 4,
|
|
125
|
-
COMMENT_START: 5,
|
|
126
|
-
COMMENT_END: 6,
|
|
127
|
-
IDENTIFIER: 7,
|
|
128
|
-
STRING: 8,
|
|
129
|
-
NUMBER: 9,
|
|
130
|
-
OPERATOR: 10,
|
|
131
|
-
DOT: 11,
|
|
132
|
-
COMMA: 12,
|
|
133
|
-
PIPE: 13,
|
|
134
|
-
COLON: 14,
|
|
135
|
-
LPAREN: 15,
|
|
136
|
-
RPAREN: 16,
|
|
137
|
-
LBRACKET: 17,
|
|
138
|
-
RBRACKET: 18,
|
|
139
|
-
LBRACE: 19,
|
|
140
|
-
RBRACE: 20,
|
|
141
|
-
ASSIGN: 21,
|
|
142
|
-
EOF: 22
|
|
143
|
-
};
|
|
144
|
-
symbols = {
|
|
145
|
-
binja_lexer_new: {
|
|
146
|
-
args: [FFIType.ptr, FFIType.u64],
|
|
147
|
-
returns: FFIType.ptr
|
|
148
|
-
},
|
|
149
|
-
binja_lexer_free: {
|
|
150
|
-
args: [FFIType.ptr],
|
|
151
|
-
returns: FFIType.void
|
|
152
|
-
},
|
|
153
|
-
binja_lexer_token_count: {
|
|
154
|
-
args: [FFIType.ptr],
|
|
155
|
-
returns: FFIType.u64
|
|
156
|
-
},
|
|
157
|
-
binja_lexer_token_type: {
|
|
158
|
-
args: [FFIType.ptr, FFIType.u64],
|
|
159
|
-
returns: FFIType.u8
|
|
160
|
-
},
|
|
161
|
-
binja_lexer_token_start: {
|
|
162
|
-
args: [FFIType.ptr, FFIType.u64],
|
|
163
|
-
returns: FFIType.u32
|
|
164
|
-
},
|
|
165
|
-
binja_lexer_token_end: {
|
|
166
|
-
args: [FFIType.ptr, FFIType.u64],
|
|
167
|
-
returns: FFIType.u32
|
|
168
|
-
},
|
|
169
|
-
binja_lexer_has_error: {
|
|
170
|
-
args: [FFIType.ptr],
|
|
171
|
-
returns: FFIType.bool
|
|
172
|
-
},
|
|
173
|
-
binja_lexer_error_code: {
|
|
174
|
-
args: [FFIType.ptr],
|
|
175
|
-
returns: FFIType.u8
|
|
176
|
-
},
|
|
177
|
-
binja_lexer_error_line: {
|
|
178
|
-
args: [FFIType.ptr],
|
|
179
|
-
returns: FFIType.u32
|
|
180
|
-
},
|
|
181
|
-
binja_tokenize_count: {
|
|
182
|
-
args: [FFIType.ptr, FFIType.u64],
|
|
183
|
-
returns: FFIType.u64
|
|
184
|
-
},
|
|
185
|
-
binja_version: {
|
|
186
|
-
args: [],
|
|
187
|
-
returns: FFIType.ptr
|
|
188
|
-
}
|
|
189
|
-
};
|
|
190
|
-
NativeLexer = class NativeLexer {
|
|
191
|
-
lexerPtr = 0;
|
|
192
|
-
source;
|
|
193
|
-
sourceBuffer;
|
|
194
|
-
lib;
|
|
195
|
-
_tokenCount = 0;
|
|
196
|
-
_isEmpty = false;
|
|
197
|
-
constructor(source) {
|
|
198
|
-
const lib = loadLibrary();
|
|
199
|
-
if (!lib) {
|
|
200
|
-
throw new Error("Native library not available. Use isNativeAvailable() to check first.");
|
|
201
|
-
}
|
|
202
|
-
this.lib = lib;
|
|
203
|
-
this.source = source;
|
|
204
|
-
if (source.length === 0) {
|
|
205
|
-
this._isEmpty = true;
|
|
206
|
-
this._tokenCount = 1;
|
|
207
|
-
this.sourceBuffer = new Uint8Array(0);
|
|
208
|
-
return;
|
|
209
|
-
}
|
|
210
|
-
this.sourceBuffer = new TextEncoder().encode(source);
|
|
211
|
-
const result = this.lib.symbols.binja_lexer_new(ptr(this.sourceBuffer), this.sourceBuffer.length);
|
|
212
|
-
if (!result) {
|
|
213
|
-
throw new Error("Failed to create native lexer");
|
|
214
|
-
}
|
|
215
|
-
this.lexerPtr = result;
|
|
216
|
-
this._tokenCount = Number(this.lib.symbols.binja_lexer_token_count(this.lexerPtr));
|
|
217
|
-
}
|
|
218
|
-
get tokenCount() {
|
|
219
|
-
return this._tokenCount;
|
|
220
|
-
}
|
|
221
|
-
getTokenType(index) {
|
|
222
|
-
if (this._isEmpty)
|
|
223
|
-
return TokenType.EOF;
|
|
224
|
-
return Number(this.lib.symbols.binja_lexer_token_type(this.lexerPtr, index));
|
|
225
|
-
}
|
|
226
|
-
getTokenStart(index) {
|
|
227
|
-
if (this._isEmpty)
|
|
228
|
-
return 0;
|
|
229
|
-
return Number(this.lib.symbols.binja_lexer_token_start(this.lexerPtr, index));
|
|
230
|
-
}
|
|
231
|
-
getTokenEnd(index) {
|
|
232
|
-
if (this._isEmpty)
|
|
233
|
-
return 0;
|
|
234
|
-
return Number(this.lib.symbols.binja_lexer_token_end(this.lexerPtr, index));
|
|
235
|
-
}
|
|
236
|
-
hasError() {
|
|
237
|
-
if (this._isEmpty)
|
|
238
|
-
return false;
|
|
239
|
-
return Boolean(this.lib.symbols.binja_lexer_has_error(this.lexerPtr));
|
|
240
|
-
}
|
|
241
|
-
getErrorCode() {
|
|
242
|
-
if (this._isEmpty)
|
|
243
|
-
return 0;
|
|
244
|
-
return Number(this.lib.symbols.binja_lexer_error_code(this.lexerPtr));
|
|
245
|
-
}
|
|
246
|
-
getErrorLine() {
|
|
247
|
-
if (this._isEmpty)
|
|
248
|
-
return 1;
|
|
249
|
-
return Number(this.lib.symbols.binja_lexer_error_line(this.lexerPtr));
|
|
250
|
-
}
|
|
251
|
-
getTokenValue(index) {
|
|
252
|
-
if (this._isEmpty)
|
|
253
|
-
return "";
|
|
254
|
-
const start = this.getTokenStart(index);
|
|
255
|
-
const end = this.getTokenEnd(index);
|
|
256
|
-
return new TextDecoder().decode(this.sourceBuffer.slice(start, end));
|
|
257
|
-
}
|
|
258
|
-
getToken(index) {
|
|
259
|
-
return {
|
|
260
|
-
type: this.getTokenType(index),
|
|
261
|
-
start: this.getTokenStart(index),
|
|
262
|
-
end: this.getTokenEnd(index),
|
|
263
|
-
value: this.getTokenValue(index)
|
|
264
|
-
};
|
|
265
|
-
}
|
|
266
|
-
getAllTokens() {
|
|
267
|
-
const tokens = [];
|
|
268
|
-
for (let i = 0;i < this._tokenCount; i++) {
|
|
269
|
-
tokens.push(this.getToken(i));
|
|
270
|
-
}
|
|
271
|
-
return tokens;
|
|
272
|
-
}
|
|
273
|
-
free() {
|
|
274
|
-
if (this.lexerPtr) {
|
|
275
|
-
this.lib.symbols.binja_lexer_free(this.lexerPtr);
|
|
276
|
-
this.lexerPtr = null;
|
|
277
|
-
}
|
|
278
|
-
}
|
|
279
|
-
[Symbol.dispose]() {
|
|
280
|
-
this.free();
|
|
281
|
-
}
|
|
282
|
-
};
|
|
283
|
-
});
|
|
284
3
|
|
|
285
4
|
// src/cli.ts
|
|
286
5
|
import * as fs from "fs";
|
|
@@ -302,21 +21,8 @@ var KEYWORDS = {
|
|
|
302
21
|
};
|
|
303
22
|
|
|
304
23
|
// src/lexer/hybrid.ts
|
|
305
|
-
var
|
|
306
|
-
var _nativeAvailable2 = false;
|
|
307
|
-
var NativeLexerClass = null;
|
|
24
|
+
var _tokenizeBatchFn = null;
|
|
308
25
|
function checkNative() {
|
|
309
|
-
if (_nativeChecked)
|
|
310
|
-
return _nativeAvailable2;
|
|
311
|
-
_nativeChecked = true;
|
|
312
|
-
try {
|
|
313
|
-
const native = (init_native(), __toCommonJS(exports_native));
|
|
314
|
-
if (typeof native.isNativeAvailable === "function" && native.isNativeAvailable()) {
|
|
315
|
-
_nativeAvailable2 = true;
|
|
316
|
-
NativeLexerClass = native.NativeLexer;
|
|
317
|
-
return true;
|
|
318
|
-
}
|
|
319
|
-
} catch {}
|
|
320
26
|
return false;
|
|
321
27
|
}
|
|
322
28
|
var NATIVE_TO_TS = {
|
|
@@ -363,71 +69,52 @@ var KEYWORD_TO_TYPE = {
|
|
|
363
69
|
or: "OR" /* OR */,
|
|
364
70
|
not: "NOT" /* NOT */
|
|
365
71
|
};
|
|
366
|
-
var ERROR_MESSAGES = {
|
|
367
|
-
1: "Unterminated string",
|
|
368
|
-
2: "Unclosed template tag",
|
|
369
|
-
3: "Invalid operator",
|
|
370
|
-
4: "Unexpected character"
|
|
371
|
-
};
|
|
372
72
|
function isNativeAccelerated() {
|
|
373
73
|
return checkNative();
|
|
374
74
|
}
|
|
375
75
|
function tokenizeNative(source) {
|
|
376
|
-
if (!checkNative() || !
|
|
76
|
+
if (!checkNative() || !_tokenizeBatchFn)
|
|
377
77
|
return null;
|
|
378
78
|
if (source.length === 0) {
|
|
379
79
|
return [{ type: "EOF" /* EOF */, value: "", line: 1, column: 1 }];
|
|
380
80
|
}
|
|
381
|
-
const
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
const errorLine = lexer.getErrorLine();
|
|
386
|
-
const message = ERROR_MESSAGES[errorCode] ?? "Unknown error";
|
|
387
|
-
throw new Error(`${message} at line ${errorLine}`);
|
|
388
|
-
}
|
|
389
|
-
const tokens = [];
|
|
390
|
-
const count = lexer.tokenCount;
|
|
391
|
-
const lineStarts = [0];
|
|
392
|
-
for (let i = 0;i < source.length; i++) {
|
|
393
|
-
if (source[i] === `
|
|
81
|
+
const rawTokens = _tokenizeBatchFn(source);
|
|
82
|
+
const lineStarts = [0];
|
|
83
|
+
for (let i = 0;i < source.length; i++) {
|
|
84
|
+
if (source[i] === `
|
|
394
85
|
`)
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
return tokens;
|
|
428
|
-
} finally {
|
|
429
|
-
lexer.free();
|
|
430
|
-
}
|
|
86
|
+
lineStarts.push(i + 1);
|
|
87
|
+
}
|
|
88
|
+
const tokens = new Array(rawTokens.length);
|
|
89
|
+
for (let i = 0;i < rawTokens.length; i++) {
|
|
90
|
+
const [nativeType, start, end] = rawTokens[i];
|
|
91
|
+
let value = source.slice(start, end);
|
|
92
|
+
let lo = 0, hi = lineStarts.length - 1;
|
|
93
|
+
while (lo < hi) {
|
|
94
|
+
const mid = lo + hi + 1 >> 1;
|
|
95
|
+
if (lineStarts[mid] <= start)
|
|
96
|
+
lo = mid;
|
|
97
|
+
else
|
|
98
|
+
hi = mid - 1;
|
|
99
|
+
}
|
|
100
|
+
const line = lo + 1;
|
|
101
|
+
const column = start - lineStarts[lo] + 1;
|
|
102
|
+
let type = NATIVE_TO_TS[nativeType] ?? "NAME" /* NAME */;
|
|
103
|
+
if (nativeType === 10 && OPERATOR_TO_TYPE[value]) {
|
|
104
|
+
type = OPERATOR_TO_TYPE[value];
|
|
105
|
+
} else if (type === "NAME" /* NAME */ && KEYWORD_TO_TYPE[value]) {
|
|
106
|
+
type = KEYWORD_TO_TYPE[value];
|
|
107
|
+
}
|
|
108
|
+
if (type === "STRING" /* STRING */ && value.length >= 2) {
|
|
109
|
+
const first = value[0];
|
|
110
|
+
const last = value[value.length - 1];
|
|
111
|
+
if (first === '"' && last === '"' || first === "'" && last === "'") {
|
|
112
|
+
value = value.slice(1, -1);
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
tokens[i] = { type, value, line, column };
|
|
116
|
+
}
|
|
117
|
+
return tokens;
|
|
431
118
|
}
|
|
432
119
|
|
|
433
120
|
// src/lexer/index.ts
|
|
@@ -827,13 +514,16 @@ class Lexer {
|
|
|
827
514
|
` || c === "\r";
|
|
828
515
|
}
|
|
829
516
|
isDigit(c) {
|
|
830
|
-
|
|
517
|
+
const code = c.charCodeAt(0);
|
|
518
|
+
return code >= 48 && code <= 57;
|
|
831
519
|
}
|
|
832
520
|
isAlpha(c) {
|
|
833
|
-
|
|
521
|
+
const code = c.charCodeAt(0);
|
|
522
|
+
return code >= 97 && code <= 122 || code >= 65 && code <= 90;
|
|
834
523
|
}
|
|
835
524
|
isAlphaNumeric(c) {
|
|
836
|
-
|
|
525
|
+
const code = c.charCodeAt(0);
|
|
526
|
+
return code >= 48 && code <= 57 || code >= 97 && code <= 122 || code >= 65 && code <= 90;
|
|
837
527
|
}
|
|
838
528
|
addToken(type, value) {
|
|
839
529
|
this.state.tokens.push({
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/filters/index.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,MAAM,MAAM,cAAc,GAAG,CAAC,KAAK,EAAE,GAAG,EAAE,GAAG,IAAI,EAAE,GAAG,EAAE,KAAK,GAAG,CAAA;AAuBhE,eAAO,MAAM,KAAK,EAAE,cAAuD,CAAA;AAE3E,eAAO,MAAM,KAAK,EAAE,cAAuD,CAAA;AAE3E,eAAO,MAAM,UAAU,EAAE,cAGxB,CAAA;AAED,eAAO,MAAM,QAAQ,EAAE,cAGtB,CAAA;AAED,eAAO,MAAM,KAAK,EAAE,cACwC,CAAA;AAE5D,eAAO,MAAM,IAAI,EAAE,cAAgD,CAAA;AAEnE,eAAO,MAAM,SAAS,EAAE,cACoB,CAAA;AAE5C,eAAO,MAAM,MAAM,EAAE,cAWpB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAKlB,CAAA;AAED,eAAO,MAAM,QAAQ,EAAE,cACqB,CAAA;AAE5C,eAAO,MAAM,UAAU,EAAE,cAaxB,CAAA;AAED,eAAO,MAAM,YAAY,EAAE,cAM1B,CAAA;AAED,eAAO,MAAM,aAAa,EAAE,cAI3B,CAAA;AAED,eAAO,MAAM,aAAa,EAAE,cAI3B,CAAA;AAED,eAAO,MAAM,SAAS,EAAE,cACsC,CAAA;AAE9D,eAAO,MAAM,MAAM,EAAE,cAMpB,CAAA;AAED,eAAO,MAAM,KAAK,EAAE,cACS,CAAA;AAE7B,eAAO,MAAM,KAAK,EAAE,cACW,CAAA;AAE/B,eAAO,MAAM,GAAG,EAAE,cACiB,CAAA;AAEnC,eAAO,MAAM,OAAO,EAAE,cAKc,CAAA;AAIpC,eAAO,MAAM,GAAG,EAAE,cAAmD,CAAA;AAErE,eAAO,MAAM,KAAK,EAAE,cACsB,CAAA;AAE1C,eAAO,MAAM,GAAG,EAAE,cAA4D,CAAA;AAE9E,eAAO,MAAM,KAAK,EAAE,cAA4D,CAAA;AAGhF,eAAO,MAAM,WAAW,EAAE,cAWzB,CAAA;AAGD,eAAO,MAAM,GAAG,EAAE,cASjB,CAAA;AAGD,eAAO,MAAM,WAAW,EAAE,cACS,CAAA;AAEnC,eAAO,MAAM,cAAc,EAAE,cAY5B,CAAA;AAID,eAAO,MAAM,MAAM,EAAE,cAUpB,CAAA;AAED,eAAO,MAAM,SAAS,EAAE,cACO,CAAA;AAE/B,eAAO,MAAM,KAAK,EAAE,cAInB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAIlB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAGlB,CAAA;AAED,eAAO,MAAM,KAAK,EAAE,cAanB,CAAA;AAED,eAAO,MAAM,OAAO,EAAE,cAIrB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAIlB,CAAA;AAED,eAAO,MAAM,MAAM,EAAE,cAGpB,CAAA;AAGD,eAAO,MAAM,SAAS,EAAE,cAGvB,CAAA;AAGD,eAAO,MAAM,QAAQ,EAAE,cAOtB,CAAA;AAGD,eAAO,MAAM,gBAAgB,EAAE,cAG9B,CAAA;AAGD,eAAO,MAAM,OAAO,EAAE,cAUrB,CAAA;AAoCD,eAAO,MAAM,IAAI,EAAE,cAKlB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAA+D,CAAA;AAElF,eAAO,MAAM,SAAS,EAAE,cAqBvB,CAAA;AAED,eAAO,MAAM,SAAS,EAAE,cAqBvB,CAAA;AAKD,QAAA,MAAM,aAAa,EAAE,cAKpB,CAAA;AACD,OAAO,EAAE,aAAa,IAAI,OAAO,EAAE,CAAA;AAGnC,eAAO,MAAM,eAAe,EAAE,cACgC,CAAA;AAG9D,eAAO,MAAM,KAAK,EAAE,cAKnB,CAAA;AAGD,eAAO,MAAM,SAAS,EAAE,cAIvB,CAAA;AAID,eAAO,MAAM,SAAS,EAAE,cAA6D,CAAA;AAErF,eAAO,MAAM,MAAM,EAAE,cAMpB,CAAA;AAID,eAAO,MAAM,IAAI,EAAE,cAUlB,CAAA;AAKD,eAAO,MAAM,MAAM,EAAE,cAKpB,CAAA;AAGD,eAAO,MAAM,KAAK,EAAE,cAcnB,CAAA;AAID,eAAO,MAAM,OAAO,EAAE,cAkBrB,CAAA;AAKD,eAAO,MAAM,QAAQ,EAAE,cA6BtB,CAAA;AAID,eAAO,MAAM,MAAM,EAAE,cAoBpB,CAAA;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/filters/index.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,MAAM,MAAM,cAAc,GAAG,CAAC,KAAK,EAAE,GAAG,EAAE,GAAG,IAAI,EAAE,GAAG,EAAE,KAAK,GAAG,CAAA;AAuBhE,eAAO,MAAM,KAAK,EAAE,cAAuD,CAAA;AAE3E,eAAO,MAAM,KAAK,EAAE,cAAuD,CAAA;AAE3E,eAAO,MAAM,UAAU,EAAE,cAGxB,CAAA;AAED,eAAO,MAAM,QAAQ,EAAE,cAGtB,CAAA;AAED,eAAO,MAAM,KAAK,EAAE,cACwC,CAAA;AAE5D,eAAO,MAAM,IAAI,EAAE,cAAgD,CAAA;AAEnE,eAAO,MAAM,SAAS,EAAE,cACoB,CAAA;AAE5C,eAAO,MAAM,MAAM,EAAE,cAWpB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAKlB,CAAA;AAED,eAAO,MAAM,QAAQ,EAAE,cACqB,CAAA;AAE5C,eAAO,MAAM,UAAU,EAAE,cAaxB,CAAA;AAED,eAAO,MAAM,YAAY,EAAE,cAM1B,CAAA;AAED,eAAO,MAAM,aAAa,EAAE,cAI3B,CAAA;AAED,eAAO,MAAM,aAAa,EAAE,cAI3B,CAAA;AAED,eAAO,MAAM,SAAS,EAAE,cACsC,CAAA;AAE9D,eAAO,MAAM,MAAM,EAAE,cAMpB,CAAA;AAED,eAAO,MAAM,KAAK,EAAE,cACS,CAAA;AAE7B,eAAO,MAAM,KAAK,EAAE,cACW,CAAA;AAE/B,eAAO,MAAM,GAAG,EAAE,cACiB,CAAA;AAEnC,eAAO,MAAM,OAAO,EAAE,cAKc,CAAA;AAIpC,eAAO,MAAM,GAAG,EAAE,cAAmD,CAAA;AAErE,eAAO,MAAM,KAAK,EAAE,cACsB,CAAA;AAE1C,eAAO,MAAM,GAAG,EAAE,cAA4D,CAAA;AAE9E,eAAO,MAAM,KAAK,EAAE,cAA4D,CAAA;AAGhF,eAAO,MAAM,WAAW,EAAE,cAWzB,CAAA;AAGD,eAAO,MAAM,GAAG,EAAE,cASjB,CAAA;AAGD,eAAO,MAAM,WAAW,EAAE,cACS,CAAA;AAEnC,eAAO,MAAM,cAAc,EAAE,cAY5B,CAAA;AAID,eAAO,MAAM,MAAM,EAAE,cAUpB,CAAA;AAED,eAAO,MAAM,SAAS,EAAE,cACO,CAAA;AAE/B,eAAO,MAAM,KAAK,EAAE,cAInB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAIlB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAGlB,CAAA;AAED,eAAO,MAAM,KAAK,EAAE,cAanB,CAAA;AAED,eAAO,MAAM,OAAO,EAAE,cAIrB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAIlB,CAAA;AAED,eAAO,MAAM,MAAM,EAAE,cAGpB,CAAA;AAGD,eAAO,MAAM,SAAS,EAAE,cAGvB,CAAA;AAGD,eAAO,MAAM,QAAQ,EAAE,cAOtB,CAAA;AAGD,eAAO,MAAM,gBAAgB,EAAE,cAG9B,CAAA;AAGD,eAAO,MAAM,OAAO,EAAE,cAUrB,CAAA;AAoCD,eAAO,MAAM,IAAI,EAAE,cAKlB,CAAA;AAED,eAAO,MAAM,IAAI,EAAE,cAA+D,CAAA;AAElF,eAAO,MAAM,SAAS,EAAE,cAqBvB,CAAA;AAED,eAAO,MAAM,SAAS,EAAE,cAqBvB,CAAA;AAKD,QAAA,MAAM,aAAa,EAAE,cAKpB,CAAA;AACD,OAAO,EAAE,aAAa,IAAI,OAAO,EAAE,CAAA;AAGnC,eAAO,MAAM,eAAe,EAAE,cACgC,CAAA;AAG9D,eAAO,MAAM,KAAK,EAAE,cAKnB,CAAA;AAGD,eAAO,MAAM,SAAS,EAAE,cAIvB,CAAA;AAID,eAAO,MAAM,SAAS,EAAE,cAA6D,CAAA;AAErF,eAAO,MAAM,MAAM,EAAE,cAMpB,CAAA;AAID,eAAO,MAAM,IAAI,EAAE,cAUlB,CAAA;AAKD,eAAO,MAAM,MAAM,EAAE,cAKpB,CAAA;AAGD,eAAO,MAAM,KAAK,EAAE,cAcnB,CAAA;AAID,eAAO,MAAM,OAAO,EAAE,cAkBrB,CAAA;AAKD,eAAO,MAAM,QAAQ,EAAE,cA6BtB,CAAA;AAID,eAAO,MAAM,MAAM,EAAE,cAoBpB,CAAA;AAID,eAAO,MAAM,OAAO,EAAE,cA4BrB,CAAA;AAID,eAAO,MAAM,MAAM,EAAE,cAapB,CAAA;AAGD,eAAO,MAAM,MAAM,EAAE,cAAyC,CAAA;AAG9D,eAAO,MAAM,IAAI,EAAE,cAMlB,CAAA;AAGD,eAAO,MAAM,GAAG,EAAE,cAOjB,CAAA;AAGD,eAAO,MAAM,MAAM,EAAE,cAOpB,CAAA;AAGD,eAAO,MAAM,MAAM,EAAE,cAMpB,CAAA;AAGD,eAAO,MAAM,UAAU,EAAE,cAmBxB,CAAA;AAGD,eAAO,MAAM,UAAU,EAAE,cAIxB,CAAA;AAGD,eAAO,MAAM,IAAI,EAAE,cAGlB,CAAA;AAID,eAAO,MAAM,GAAG,EAAE,cAYjB,CAAA;AAID,eAAO,MAAM,GAAG,EAAE,cAYjB,CAAA;AAID,eAAO,MAAM,GAAG,EAAE,cAQjB,CAAA;AAID,eAAO,MAAM,MAAM,EAAE,cAUpB,CAAA;AAGD,eAAO,MAAM,WAAW,EAAE,cAOzB,CAAA;AAcD,eAAO,MAAM,aAAa,EAAE,cAQ3B,CAAA;AAID,eAAO,MAAM,WAAW,EAAE,cASzB,CAAA;AAGD,eAAO,MAAM,cAAc,EAAE,cA8B5B,CAAA;AAID,eAAO,MAAM,cAAc,EAAE,MAAM,CAAC,MAAM,EAAE,cAAc,CAkGzD,CAAA"}
|