lua-lexer 1.1.1 → 1.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +157 -157
- package/package.json +1 -1
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Andmarski
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,157 +1,157 @@
|
|
|
1
|
-
# lua-lexer
|
|
2
|
-
|
|
3
|
-
[](https://npmjs.org/package/lua-lexer)
|
|
4
|
-
[](https://opensource.org/licenses/MIT)
|
|
5
|
-
[](https://www.typescriptlang.org/)
|
|
6
|
-
[](https://npmjs.org/package/lua-lexer)
|
|
7
|
-
|
|
8
|
-
A fast, zero-dependency, and lightweight Lua lexer written in TypeScript.
|
|
9
|
-
|
|
10
|
-
This library is designed solely for tokenizing Lua source code into an array of tokens, making it a solid foundation for building parsers, formatters, syntax highlighters, or static analysis tools without the overhead of building an Abstract Syntax Tree (AST).
|
|
11
|
-
|
|
12
|
-
## Features
|
|
13
|
-
|
|
14
|
-
* **High Performance:** Optimized with zero-allocation strategies and tight loops to minimize execution time.
|
|
15
|
-
* **Zero Dependencies:** Keeps your dependency tree small and secure.
|
|
16
|
-
* **Dual API:** Choose between a standard `tokenize()` function or a Streaming API optimized for large files.
|
|
17
|
-
* **Lua 5.3+ Support:** Fully handles modern Lua lexical syntax, including bitwise operators and hexadecimal floats.
|
|
18
|
-
* **TypeScript Support:** Includes comprehensive type definitions.
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
## Installation
|
|
22
|
-
|
|
23
|
-
Install via npm:
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
npm install lua-lexer
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
## Usage
|
|
31
|
-
|
|
32
|
-
### ES Modules (ESM) / TypeScript
|
|
33
|
-
|
|
34
|
-
```typescript
|
|
35
|
-
import { tokenize, LuaTokenType } from "lua-lexer";
|
|
36
|
-
|
|
37
|
-
const tokens = tokenize("i = 0");
|
|
38
|
-
console.log(JSON.stringify(tokens, null, 2));
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
### CommonJS
|
|
42
|
-
|
|
43
|
-
```javascript
|
|
44
|
-
const { tokenize, LuaTokenType } = require("lua-lexer");
|
|
45
|
-
|
|
46
|
-
const tokens = tokenize("i = 0");
|
|
47
|
-
console.log(JSON.stringify(tokens, null, 2));
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
## Lexer Interface
|
|
51
|
-
|
|
52
|
-
### 1. The `tokenize` Helper
|
|
53
|
-
|
|
54
|
-
The standard way to use the library is the `tokenize` function, which processes the entire string and returns an array of `LuaToken` objects:
|
|
55
|
-
|
|
56
|
-
```typescript
|
|
57
|
-
tokenize(code, options?);
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
**Available Options:**
|
|
61
|
-
* `skipComments?: boolean` *(default: `false`)*: Skips parsing and yielding comment tokens to save memory and CPU cycles.
|
|
62
|
-
* `luaVersion?: "5.1" | "5.2" | "5.3" | "5.4" | "5.5"` *(default: `"5.4"`)*: The specific Lua language version to tokenize against. Determines the availability of version-specific operators, keywords, and escape sequences.
|
|
63
|
-
|
|
64
|
-
### 2. Streaming / Manual Iteration Interface
|
|
65
|
-
|
|
66
|
-
A streaming interface is also available. This is preferable when dealing with large files or when building a parser that consumes tokens on-demand.
|
|
67
|
-
|
|
68
|
-
```typescript
|
|
69
|
-
import { LuaLexer, LuaTokenType } from "lua-lexer";
|
|
70
|
-
|
|
71
|
-
const lexer = new LuaLexer({ skipComments: true });
|
|
72
|
-
lexer.load("foo = 'bar'");
|
|
73
|
-
|
|
74
|
-
let token;
|
|
75
|
-
while (true) {
|
|
76
|
-
token = lexer.next();
|
|
77
|
-
if (token.type === LuaTokenType.END_OF_FILE) break;
|
|
78
|
-
console.log(token);
|
|
79
|
-
}
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
## Token Format
|
|
84
|
-
|
|
85
|
-
If the following code is executed:
|
|
86
|
-
|
|
87
|
-
```typescript
|
|
88
|
-
tokenize("i = 0");
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
The returned array will look like:
|
|
92
|
-
|
|
93
|
-
```js
|
|
94
|
-
[
|
|
95
|
-
{
|
|
96
|
-
"type": 60,
|
|
97
|
-
"value": "i",
|
|
98
|
-
"line": 1,
|
|
99
|
-
"column": 1,
|
|
100
|
-
"start": 0,
|
|
101
|
-
"length": 1
|
|
102
|
-
},
|
|
103
|
-
{
|
|
104
|
-
"type": 44,
|
|
105
|
-
"value": undefined,
|
|
106
|
-
"line": 1,
|
|
107
|
-
"column": 3,
|
|
108
|
-
"start": 2,
|
|
109
|
-
"length": 1
|
|
110
|
-
},
|
|
111
|
-
{
|
|
112
|
-
"type": 59,
|
|
113
|
-
"value": 0,
|
|
114
|
-
"line": 1,
|
|
115
|
-
"column": 5,
|
|
116
|
-
"start": 4,
|
|
117
|
-
"length": 1
|
|
118
|
-
},
|
|
119
|
-
{
|
|
120
|
-
"type": 0,
|
|
121
|
-
"line": 1,
|
|
122
|
-
"column": 6,
|
|
123
|
-
"start": 5,
|
|
124
|
-
"length": 0
|
|
125
|
-
}
|
|
126
|
-
]
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
### Token Properties:
|
|
130
|
-
* **`type`**: Expressed as an enum integer which can be matched with the exported `LuaTokenType` enum (e.g., `60` equals `LuaTokenType.IDENTIFIER`).
|
|
131
|
-
* **`value`**: The parsed value of the token (e.g., numeric value for numbers, string values for strings and identifiers).
|
|
132
|
-
* **`line`**: The 1-indexed line number where the token starts.
|
|
133
|
-
* **`column`**: The 1-indexed column number where the token starts.
|
|
134
|
-
* **`start`**: The 0-indexed character offset where the token begins in the raw source string.
|
|
135
|
-
* **`length`**: The length of the token in characters. Slicing `code.substring(start, start + length)` will return the raw token string.
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
## Performance
|
|
139
|
-
|
|
140
|
-
Below is a benchmark comparing the raw tokenization speed of `lua-lexer` against the tokenization phase of the `luaparse` package.
|
|
141
|
-
|
|
142
|
-
| Library | Avg Time (Raw Lexing) | Relative Performance |
|
|
143
|
-
| :--- | :--- | :--- |
|
|
144
|
-
| **lua-lexer** | **~0.83 ms** | **~1.50x Faster** |
|
|
145
|
-
| luaparse | ~1.24 ms | Baseline (1.0x) |
|
|
146
|
-
|
|
147
|
-
> **Note:** For full methodology, reproduction instructions
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
## Acknowledgements
|
|
151
|
-
|
|
152
|
-
This project was heavily inspired by the excellent `luaparse` library created by Oskar Schöldström. Ultimately, `lua-lexer` started as a fun experiment with code
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
## License
|
|
156
|
-
|
|
157
|
-
MIT
|
|
1
|
+
# lua-lexer
|
|
2
|
+
|
|
3
|
+
[](https://npmjs.org/package/lua-lexer)
|
|
4
|
+
[](https://opensource.org/licenses/MIT)
|
|
5
|
+
[](https://www.typescriptlang.org/)
|
|
6
|
+
[](https://npmjs.org/package/lua-lexer)
|
|
7
|
+
|
|
8
|
+
A fast, zero-dependency, and lightweight Lua lexer written in TypeScript.
|
|
9
|
+
|
|
10
|
+
This library is designed solely for tokenizing Lua source code into an array of tokens, making it a solid foundation for building parsers, formatters, syntax highlighters, or static analysis tools without the overhead of building an Abstract Syntax Tree (AST).
|
|
11
|
+
|
|
12
|
+
## Features
|
|
13
|
+
|
|
14
|
+
* **High Performance:** Optimized with zero-allocation strategies and tight loops to minimize execution time.
|
|
15
|
+
* **Zero Dependencies:** Keeps your dependency tree small and secure.
|
|
16
|
+
* **Dual API:** Choose between a standard `tokenize()` function or a Streaming API optimized for large files.
|
|
17
|
+
* **Lua 5.3+ Support:** Fully handles modern Lua lexical syntax, including bitwise operators and hexadecimal floats.
|
|
18
|
+
* **TypeScript Support:** Includes comprehensive type definitions.
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
## Installation
|
|
22
|
+
|
|
23
|
+
Install via npm:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
npm install lua-lexer
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
## Usage
|
|
31
|
+
|
|
32
|
+
### ES Modules (ESM) / TypeScript
|
|
33
|
+
|
|
34
|
+
```typescript
|
|
35
|
+
import { tokenize, LuaTokenType } from "lua-lexer";
|
|
36
|
+
|
|
37
|
+
const tokens = tokenize("i = 0");
|
|
38
|
+
console.log(JSON.stringify(tokens, null, 2));
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
### CommonJS
|
|
42
|
+
|
|
43
|
+
```javascript
|
|
44
|
+
const { tokenize, LuaTokenType } = require("lua-lexer");
|
|
45
|
+
|
|
46
|
+
const tokens = tokenize("i = 0");
|
|
47
|
+
console.log(JSON.stringify(tokens, null, 2));
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Lexer Interface
|
|
51
|
+
|
|
52
|
+
### 1. The `tokenize` Helper
|
|
53
|
+
|
|
54
|
+
The standard way to use the library is the `tokenize` function, which processes the entire string and returns an array of `LuaToken` objects:
|
|
55
|
+
|
|
56
|
+
```typescript
|
|
57
|
+
tokenize(code, options?);
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Available Options:**
|
|
61
|
+
* `skipComments?: boolean` *(default: `false`)*: Skips parsing and yielding comment tokens to save memory and CPU cycles.
|
|
62
|
+
* `luaVersion?: "5.1" | "5.2" | "5.3" | "5.4" | "5.5"` *(default: `"5.4"`)*: The specific Lua language version to tokenize against. Determines the availability of version-specific operators, keywords, and escape sequences.
|
|
63
|
+
|
|
64
|
+
### 2. Streaming / Manual Iteration Interface
|
|
65
|
+
|
|
66
|
+
A streaming interface is also available. This is preferable when dealing with large files or when building a parser that consumes tokens on-demand.
|
|
67
|
+
|
|
68
|
+
```typescript
|
|
69
|
+
import { LuaLexer, LuaTokenType } from "lua-lexer";
|
|
70
|
+
|
|
71
|
+
const lexer = new LuaLexer({ skipComments: true });
|
|
72
|
+
lexer.load("foo = 'bar'");
|
|
73
|
+
|
|
74
|
+
let token;
|
|
75
|
+
while (true) {
|
|
76
|
+
token = lexer.next();
|
|
77
|
+
if (token.type === LuaTokenType.END_OF_FILE) break;
|
|
78
|
+
console.log(token);
|
|
79
|
+
}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
|
|
83
|
+
## Token Format
|
|
84
|
+
|
|
85
|
+
If the following code is executed:
|
|
86
|
+
|
|
87
|
+
```typescript
|
|
88
|
+
tokenize("i = 0");
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
The returned array will look like:
|
|
92
|
+
|
|
93
|
+
```js
|
|
94
|
+
[
|
|
95
|
+
{
|
|
96
|
+
"type": 60,
|
|
97
|
+
"value": "i",
|
|
98
|
+
"line": 1,
|
|
99
|
+
"column": 1,
|
|
100
|
+
"start": 0,
|
|
101
|
+
"length": 1
|
|
102
|
+
},
|
|
103
|
+
{
|
|
104
|
+
"type": 44,
|
|
105
|
+
"value": undefined,
|
|
106
|
+
"line": 1,
|
|
107
|
+
"column": 3,
|
|
108
|
+
"start": 2,
|
|
109
|
+
"length": 1
|
|
110
|
+
},
|
|
111
|
+
{
|
|
112
|
+
"type": 59,
|
|
113
|
+
"value": 0,
|
|
114
|
+
"line": 1,
|
|
115
|
+
"column": 5,
|
|
116
|
+
"start": 4,
|
|
117
|
+
"length": 1
|
|
118
|
+
},
|
|
119
|
+
{
|
|
120
|
+
"type": 0,
|
|
121
|
+
"line": 1,
|
|
122
|
+
"column": 6,
|
|
123
|
+
"start": 5,
|
|
124
|
+
"length": 0
|
|
125
|
+
}
|
|
126
|
+
]
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
### Token Properties:
|
|
130
|
+
* **`type`**: Expressed as an enum integer which can be matched with the exported `LuaTokenType` enum (e.g., `60` equals `LuaTokenType.IDENTIFIER`).
|
|
131
|
+
* **`value`**: The parsed value of the token (e.g., numeric value for numbers, string values for strings and identifiers).
|
|
132
|
+
* **`line`**: The 1-indexed line number where the token starts.
|
|
133
|
+
* **`column`**: The 1-indexed column number where the token starts.
|
|
134
|
+
* **`start`**: The 0-indexed character offset where the token begins in the raw source string.
|
|
135
|
+
* **`length`**: The length of the token in characters. Slicing `code.substring(start, start + length)` will return the raw token string.
|
|
136
|
+
|
|
137
|
+
|
|
138
|
+
## Performance
|
|
139
|
+
|
|
140
|
+
Below is a benchmark comparing the raw tokenization speed of `lua-lexer` against the tokenization phase of the `luaparse` package.
|
|
141
|
+
|
|
142
|
+
| Library | Avg Time (Raw Lexing) | Relative Performance |
|
|
143
|
+
| :--- | :--- | :--- |
|
|
144
|
+
| **lua-lexer** | **~0.83 ms** | **~1.50x Faster** |
|
|
145
|
+
| luaparse | ~1.24 ms | Baseline (1.0x) |
|
|
146
|
+
|
|
147
|
+
> **Note:** For full methodology, reproduction instructions and details please read the [Performance Benchmarks](./benchmarks/README.md) documentation.
|
|
148
|
+
|
|
149
|
+
|
|
150
|
+
## Acknowledgements
|
|
151
|
+
|
|
152
|
+
This project was heavily inspired by the excellent [`luaparse`](https://github.com/fstirlitz/luaparse/) library created by Oskar Schöldström. Ultimately, `lua-lexer` started as a fun experiment with code - a playground to see just how fast and optimized a zero-dependency Lua lexer could be written in TypeScript.
|
|
153
|
+
|
|
154
|
+
|
|
155
|
+
## License
|
|
156
|
+
|
|
157
|
+
MIT
|