@kaeawc/auto-mobile 0.0.89 → 0.0.91
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -0
- package/README.md.backup +7 -0
- package/dist/build-info.json +12 -10
- package/dist/schemas/prototype-platform-permissions.json +75 -0
- package/dist/schemas/test-plan.schema.json +2 -2
- package/dist/src/index.js +946 -874
- package/dist/src/utils/workers/iosBundleExtractWorker.js +3 -3
- package/mcp-skills/auto-mobile-usage/SKILL.md +1 -0
- package/node_modules/strtok3/README.md +172 -113
- package/node_modules/strtok3/lib/BufferSizeError.js +1 -1
- package/node_modules/strtok3/lib/TokenWriter.js +151 -0
- package/node_modules/strtok3/lib/core.js +1 -0
- package/node_modules/strtok3/package.json +2 -2
- package/node_modules/token-types/LICENSE.txt +9 -9
- package/node_modules/token-types/README.md +160 -120
- package/node_modules/token-types/{lib → dist/lib}/index.js +21 -3
- package/node_modules/token-types/package.json +81 -81
- package/package.json +11 -9
- package/schemas/prototype-platform-permissions.json +75 -0
- package/schemas/test-plan.schema.json +2 -2
|
@@ -8,24 +8,25 @@
|
|
|
8
8
|
[](https://codecov.io/gh/Borewit/strtok3)
|
|
9
9
|
# strtok3
|
|
10
10
|
|
|
11
|
-
A promise-based streaming [
|
|
11
|
+
A promise-based streaming [_tokenizer_](#tokenizer-object) for [Node.js](https://nodejs.org) and browsers.
|
|
12
12
|
|
|
13
|
-
The `strtok3` module provides several methods for creating a [
|
|
13
|
+
The `strtok3` module provides several methods for creating a [_tokenizer_](#tokenizer-object) from various input sources.
|
|
14
14
|
Designed for:
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
15
|
+
- Seamless support in streaming environments.
|
|
16
|
+
- Efficiently decode binary data, strings, and numbers.
|
|
17
|
+
- Reading [predefined](https://github.com/Borewit/token-types) or custom tokens.
|
|
18
|
+
- Offering [_tokenizers_](#tokenizer-object) for reading from [files](#fromfile-function), [streams](#fromstream-function) or [Uint8Arrays](#frombuffer-function).
|
|
19
19
|
|
|
20
20
|
### Features
|
|
21
|
+
|
|
21
22
|
`strtok3` can read from:
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
23
|
+
- Files, using a file path as input.
|
|
24
|
+
- Node.js [streams](https://nodejs.org/api/stream.html).
|
|
25
|
+
- WHATWG [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) objects containing `Uint8Array` chunks.
|
|
26
|
+
- [Buffer](https://nodejs.org/api/buffer.html) or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array).
|
|
27
|
+
- [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) objects.
|
|
28
|
+
- HTTP chunked transfer provided by [@tokenizer/http](https://github.com/Borewit/tokenizer-http).
|
|
29
|
+
- [Amazon S3](https://aws.amazon.com/s3) chunks with [@tokenizer/s3](https://github.com/Borewit/tokenizer-s3).
|
|
29
30
|
|
|
30
31
|
## Installation
|
|
31
32
|
|
|
@@ -51,6 +52,7 @@ Coverage is collected in a separate Node.js 24 job and uploaded to Codecov.
|
|
|
51
52
|
For local development, running the TypeScript tests with `tsx` requires Node.js 18.19+ (18.x) or Node.js ≥ 20.6.0.
|
|
52
53
|
|
|
53
54
|
## Support the Project
|
|
55
|
+
|
|
54
56
|
If you find this project useful and would like to support its development, consider sponsoring or contributing:
|
|
55
57
|
|
|
56
58
|
- [Become a sponsor to Borewit](https://github.com/sponsors/Borewit)
|
|
@@ -63,7 +65,7 @@ If you find this project useful and would like to support its development, consi
|
|
|
63
65
|
|
|
64
66
|
### strtok3 methods
|
|
65
67
|
|
|
66
|
-
Use one of the methods to instantiate an [
|
|
68
|
+
Use one of the methods to instantiate an [_abstract tokenizer_](#tokenizer-object):
|
|
67
69
|
- [fromBlob](#fromblob-function)
|
|
68
70
|
- [fromBuffer](#frombuffer-function)
|
|
69
71
|
- [fromFile](#fromfile-function)
|
|
@@ -84,15 +86,15 @@ without Node.js file-system APIs. In browsers, `strtok3` resolves to this core e
|
|
|
84
86
|
Create a tokenizer from a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob).
|
|
85
87
|
|
|
86
88
|
```ts
|
|
87
|
-
function fromBlob(blob: Blob, options?: ITokenizerOptions): BlobTokenizer
|
|
89
|
+
function fromBlob(blob: Blob, options?: ITokenizerOptions): BlobTokenizer;
|
|
88
90
|
```
|
|
89
91
|
|
|
90
|
-
| Parameter | Optional
|
|
91
|
-
|
|
92
|
-
| blob | no
|
|
93
|
-
| options | yes
|
|
92
|
+
| Parameter | Optional | Type | Description |
|
|
93
|
+
| --------- | -------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
94
|
+
| blob | no | [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) | [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) to read from |
|
|
95
|
+
| options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
|
|
94
96
|
|
|
95
|
-
Returns a [
|
|
97
|
+
Returns a [_tokenizer_](#tokenizer-object).
|
|
96
98
|
|
|
97
99
|
```js
|
|
98
100
|
import { fromBlob } from 'strtok3';
|
|
@@ -104,11 +106,10 @@ async function parse() {
|
|
|
104
106
|
const tokenizer = fromBlob(blob);
|
|
105
107
|
|
|
106
108
|
const myUint8Number = await tokenizer.readToken(Token.UINT8);
|
|
107
|
-
console.log(`My number: ${myUint8Number}`);
|
|
109
|
+
console.log(`My number: ${myUint8Number}`);
|
|
108
110
|
}
|
|
109
111
|
|
|
110
112
|
parse();
|
|
111
|
-
|
|
112
113
|
```
|
|
113
114
|
|
|
114
115
|
#### `fromBuffer()` function
|
|
@@ -116,15 +117,15 @@ parse();
|
|
|
116
117
|
Create a tokenizer from memory ([Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) or Node.js [Buffer](https://nodejs.org/api/buffer.html)).
|
|
117
118
|
|
|
118
119
|
```ts
|
|
119
|
-
function fromBuffer(uint8Array: Uint8Array, options?: ITokenizerOptions): BufferTokenizer
|
|
120
|
+
function fromBuffer(uint8Array: Uint8Array, options?: ITokenizerOptions): BufferTokenizer;
|
|
120
121
|
```
|
|
121
122
|
|
|
122
|
-
| Parameter | Optional | Type
|
|
123
|
-
|
|
123
|
+
| Parameter | Optional | Type | Description |
|
|
124
|
+
| ---------- | -------- | --------------------------------------------------------------------------------------------------------- | --------------------------------- |
|
|
124
125
|
| uint8Array | no | [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) | Buffer or Uint8Array to read from |
|
|
125
|
-
| options | yes | [ITokenizerOptions](#itokenizeroptions-interface)
|
|
126
|
+
| options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
|
|
126
127
|
|
|
127
|
-
Returns a [
|
|
128
|
+
Returns a [_tokenizer_](#tokenizer-object).
|
|
128
129
|
|
|
129
130
|
```js
|
|
130
131
|
import { fromBuffer } from 'strtok3';
|
|
@@ -142,22 +143,22 @@ parse();
|
|
|
142
143
|
|
|
143
144
|
#### `fromFile` function
|
|
144
145
|
|
|
145
|
-
Creates a [
|
|
146
|
+
Creates a [_tokenizer_](#tokenizer-object) from a local file.
|
|
146
147
|
|
|
147
148
|
```ts
|
|
148
|
-
function fromFile(sourceFilePath: string, options?: IFromFileOptions): Promise<FileTokenizer
|
|
149
|
-
```
|
|
149
|
+
function fromFile(sourceFilePath: string, options?: IFromFileOptions): Promise<FileTokenizer>;
|
|
150
|
+
```
|
|
150
151
|
|
|
151
|
-
| Parameter | Type
|
|
152
|
-
|
|
153
|
-
| sourceFilePath | `string`
|
|
154
|
-
| options
|
|
152
|
+
| Parameter | Type | Description |
|
|
153
|
+
| -------------- | ----------------------------------------------- | ------------------------------- |
|
|
154
|
+
| sourceFilePath | `string` | Path to file to read from |
|
|
155
|
+
| options | [IFromFileOptions](#ifromfileoptions-interface) | Optional file tokenizer options |
|
|
155
156
|
|
|
156
157
|
> [!NOTE]
|
|
157
158
|
> - Only available for Node.js engines
|
|
158
159
|
> - `fromFile` automatically embeds [file information](#ifileinfo-interface)
|
|
159
160
|
|
|
160
|
-
A Promise resolving to a [
|
|
161
|
+
A Promise resolving to a [_tokenizer_](#tokenizer-object) which can be used to parse a file.
|
|
161
162
|
|
|
162
163
|
```js
|
|
163
164
|
import { fromFile } from 'strtok3';
|
|
@@ -174,8 +175,6 @@ async function parse() {
|
|
|
174
175
|
}
|
|
175
176
|
|
|
176
177
|
parse();
|
|
177
|
-
|
|
178
|
-
|
|
179
178
|
```
|
|
180
179
|
|
|
181
180
|
#### `fromStream()` function
|
|
@@ -183,15 +182,15 @@ parse();
|
|
|
183
182
|
Create a tokenizer from a Node.js [readable stream](https://nodejs.org/api/stream.html#readable-streams) containing binary data.
|
|
184
183
|
|
|
185
184
|
```ts
|
|
186
|
-
function fromStream(stream: Readable, options?: ITokenizerOptions): Promise<ReadStreamTokenizer
|
|
185
|
+
function fromStream(stream: Readable, options?: ITokenizerOptions): Promise<ReadStreamTokenizer>;
|
|
187
186
|
```
|
|
188
187
|
|
|
189
|
-
| Parameter | Optional | Type
|
|
190
|
-
|
|
191
|
-
| stream
|
|
192
|
-
| options
|
|
188
|
+
| Parameter | Optional | Type | Description |
|
|
189
|
+
| --------- | -------- | ------------------------------------------------- | ------------------------------------ |
|
|
190
|
+
| stream | no | Readable | Node.js readable stream to read from |
|
|
191
|
+
| options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
|
|
193
192
|
|
|
194
|
-
When importing from `strtok3` in Node.js, returns a promise resolving to a [
|
|
193
|
+
When importing from `strtok3` in Node.js, returns a promise resolving to a [_tokenizer_](#tokenizer-object).
|
|
195
194
|
For streams created by `createReadStream`, the file path and size are added to `tokenizer.fileInfo`.
|
|
196
195
|
The `strtok3/core` entry point also exports `fromStream`, but returns the tokenizer synchronously without looking up file information.
|
|
197
196
|
|
|
@@ -225,15 +224,15 @@ Create a tokenizer from a [WHATWG ReadableStream](https://developer.mozilla.org/
|
|
|
225
224
|
Uses a BYOB reader when supported by the stream, otherwise a default reader. Source cleanup differs between these readers; see [`close()`](#close-function) and [`abort()`](#abort-function).
|
|
226
225
|
|
|
227
226
|
```ts
|
|
228
|
-
function fromWebStream(webStream: AnyWebByteStream, options?: ITokenizerOptions): ReadStreamTokenizer
|
|
227
|
+
function fromWebStream(webStream: AnyWebByteStream, options?: ITokenizerOptions): ReadStreamTokenizer;
|
|
229
228
|
```
|
|
230
229
|
|
|
231
|
-
| Parameter
|
|
232
|
-
|
|
233
|
-
| webStream
|
|
234
|
-
| options
|
|
230
|
+
| Parameter | Optional | Type | Description |
|
|
231
|
+
| --------- | -------- | --------------------------------------------------------------------------------- | ---------------------------------- |
|
|
232
|
+
| webStream | no | [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) | WHATWG ReadableStream to read from |
|
|
233
|
+
| options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
|
|
235
234
|
|
|
236
|
-
Returns a [
|
|
235
|
+
Returns a [_tokenizer_](#tokenizer-object).
|
|
237
236
|
|
|
238
237
|
```js
|
|
239
238
|
import { fromWebStream } from 'strtok3';
|
|
@@ -258,7 +257,7 @@ Borrow a finite range of an existing tokenizer without buffering it or taking
|
|
|
258
257
|
ownership of its input. Available from both `strtok3` and `strtok3/core`.
|
|
259
258
|
|
|
260
259
|
```ts
|
|
261
|
-
function createTokenizerView(source: ITokenizer, length: number): ITokenizerView
|
|
260
|
+
function createTokenizerView(source: ITokenizer, length: number): ITokenizerView;
|
|
262
261
|
```
|
|
263
262
|
|
|
264
263
|
The view starts at `source.position`, with local `position` zero and
|
|
@@ -275,12 +274,12 @@ await source.ignore(1);
|
|
|
275
274
|
const chunk = createTokenizerView(source, 3); // source bytes [20, 30, 40]
|
|
276
275
|
|
|
277
276
|
console.log(await chunk.readToken(UINT8)); // 20; chunk.position = 1
|
|
278
|
-
const child = chunk.createView(1);
|
|
277
|
+
const child = chunk.createView(1); // same as createTokenizerView(chunk, 1)
|
|
279
278
|
console.log(await child.readToken(UINT8)); // 30; chunk.position = 2
|
|
280
|
-
await child.close();
|
|
279
|
+
await child.close(); // source stays open, at position 3
|
|
281
280
|
|
|
282
|
-
console.log(chunk.remaining);
|
|
283
|
-
await chunk.consumeRemainder();
|
|
281
|
+
console.log(chunk.remaining); // 1
|
|
282
|
+
await chunk.consumeRemainder(); // exactly consume 40, or throw on truncation
|
|
284
283
|
await chunk.close();
|
|
285
284
|
console.log(await source.readToken(UINT8)); // 50
|
|
286
285
|
await source.close();
|
|
@@ -329,8 +328,69 @@ by a view, so callers must keep those sequential too. Explicit `abort()`
|
|
|
329
328
|
propagates to the shared source to cancel pending input and affects its other
|
|
330
329
|
users; it also closes the view.
|
|
331
330
|
|
|
331
|
+
### `TokenWriter` class
|
|
332
|
+
|
|
333
|
+
Serialize binary data synchronously into a growable in-memory buffer with the
|
|
334
|
+
same token definitions used for reading.
|
|
335
|
+
Available from both `strtok3` and `strtok3/core`.
|
|
336
|
+
|
|
337
|
+
```ts
|
|
338
|
+
const writer = new TokenWriter(); // optional initial capacity in bytes
|
|
339
|
+
const cappedWriter = new TokenWriter(undefined, { maxBufferSize: 1024 * 1024 });
|
|
340
|
+
|
|
341
|
+
writer.length; // bytes written so far
|
|
342
|
+
writer.reserve(length); // reserve zero-filled region, returns start offset
|
|
343
|
+
writer.writeToken(token, value); // encode a token, returns bytes written
|
|
344
|
+
writer.writeBuffer(value); // copy raw bytes, returns bytes written
|
|
345
|
+
writer.writeBytes(value); // alias for writeBuffer
|
|
346
|
+
writer.writeZeros(length); // append zeros, returns bytes written
|
|
347
|
+
writer.patchToken(token, offset, value); // overwrite an earlier region
|
|
348
|
+
writer.toUint8Array(); // copy of the written bytes
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
```js
|
|
352
|
+
import { TokenWriter } from 'strtok3';
|
|
353
|
+
import { UINT24_BE, UINT32_LE } from 'token-types';
|
|
354
|
+
|
|
355
|
+
const writer = new TokenWriter();
|
|
356
|
+
writer.writeToken(UINT32_LE, 0x01020304);
|
|
357
|
+
const blockLengthOffset = writer.reserve(3);
|
|
358
|
+
const bytesWritten = writer.writeBuffer(new Uint8Array([0xaa, 0xbb])); // 2
|
|
359
|
+
writer.patchToken(UINT24_BE, blockLengthOffset, 2);
|
|
360
|
+
|
|
361
|
+
console.log(writer.length); // 9
|
|
362
|
+
const data = writer.toUint8Array(); // Uint8Array(9)
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
Append methods return the number of bytes written, like `readBuffer()` returns
|
|
366
|
+
the number read; `writer.length` gives the resulting total. `writeToken()` writes
|
|
367
|
+
exactly `token.len` bytes and ignores the encoder's returned offset.
|
|
368
|
+
`reserve()` returns the starting offset of a zero-filled region for back-filling
|
|
369
|
+
with `patchToken()`. Patching outside the written region throws a `RangeError`;
|
|
370
|
+
successful patches leave `length` unchanged and return nothing.
|
|
371
|
+
|
|
372
|
+
Errors from `put()` propagate without changing the written bytes or `length`:
|
|
373
|
+
failed appends clear partial bytes before reuse, and failed patches restore the
|
|
374
|
+
original region. Encoders must write synchronously only within `token.len` bytes
|
|
375
|
+
starting at the supplied offset and must not retain the destination buffer.
|
|
376
|
+
|
|
377
|
+
`writeBuffer()` and its `writeBytes()` alias copy their input. `toUint8Array()`
|
|
378
|
+
returns a snapshot that can be modified without changing the writer. The writer
|
|
379
|
+
buffers the complete output; snapshots allocate another copy of the written bytes.
|
|
380
|
+
|
|
381
|
+
The optional `maxBufferSize` caps both total output length and backing-buffer
|
|
382
|
+
capacity. Like the tokenizer option, it defaults to `Infinity` and must be a
|
|
383
|
+
positive safe integer or `Infinity`. Requests exceeding the cap throw
|
|
384
|
+
`BufferSizeError` before allocation, encoding or changes to the output.
|
|
385
|
+
An explicit initial capacity above the cap is rejected. When omitted, the initial
|
|
386
|
+
capacity defaults to 1024 bytes or the cap, whichever is smaller; internal minimum
|
|
387
|
+
capacity and buffer growth also respect the cap. Empty appends and in-range patches
|
|
388
|
+
remain allowed at the limit. The cap does not bound total process memory, caller
|
|
389
|
+
inputs, snapshots, patch rollback copies or encoder allocations.
|
|
390
|
+
|
|
332
391
|
### `Tokenizer` object
|
|
333
|
-
|
|
392
|
+
|
|
393
|
+
The _tokenizer_ is an abstraction of a [stream](https://nodejs.org/api/stream.html), file or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array), allowing _reading_ or _peeking_ from the stream.
|
|
334
394
|
It can also be translated in chunked reads, as done in [@tokenizer/http](https://github.com/Borewit/tokenizer-http);
|
|
335
395
|
|
|
336
396
|
#### Key Features:
|
|
@@ -345,8 +405,8 @@ It can also be translated in chunked reads, as done in [@tokenizer/http](https:/
|
|
|
345
405
|
_Read_ methods advance the stream pointer, while _peek_ methods do not.
|
|
346
406
|
|
|
347
407
|
There are two kinds of functions:
|
|
348
|
-
1.
|
|
349
|
-
2.
|
|
408
|
+
1. _read_ methods: used to read a _token_ or bytes into a buffer from the [_tokenizer_](#tokenizer-object). After a successful read, `tokenizer.position` is the starting position of the read plus the number of bytes read. An explicit position can skip ahead or, for random-access tokenizers, move backwards.
|
|
409
|
+
2. _peek_ methods: same as the read, but it will _not_ advance the pointer. It allows to read (peek) ahead.
|
|
350
410
|
|
|
351
411
|
Reads and peeks require the full requested length unless `mayBeLess: true` is passed to
|
|
352
412
|
`readBuffer` or `peekBuffer`. Otherwise, insufficient input rejects with the exported
|
|
@@ -366,26 +426,26 @@ Read data from the _tokenizer_ into provided "buffer" (`Uint8Array`).
|
|
|
366
426
|
readBuffer(buffer: Uint8Array, options?: IReadChunkOptions): Promise<number>;
|
|
367
427
|
```
|
|
368
428
|
|
|
369
|
-
| Parameter
|
|
370
|
-
|
|
371
|
-
| buffer
|
|
372
|
-
| options
|
|
429
|
+
| Parameter | Type | Description |
|
|
430
|
+
| --------- | -------------------------------------------------------------- | ------------------------------------------------------------------------- |
|
|
431
|
+
| buffer | [Buffer](https://nodejs.org/api/buffer.html) | Uint8Array | Target buffer to write the data read to |
|
|
432
|
+
| options | [IReadChunkOptions](#ireadchunkoptions-interface) | Optional settings for the length, position, and handling of partial reads |
|
|
373
433
|
|
|
374
434
|
Return promise with number of bytes read.
|
|
375
435
|
The number of bytes read may be less than requested if the `mayBeLess` flag is set.
|
|
376
436
|
|
|
377
437
|
#### `peekBuffer` function
|
|
378
438
|
|
|
379
|
-
Peek (read ahead), from [
|
|
439
|
+
Peek (read ahead), from [_tokenizer_](#tokenizer-object), into the buffer without advancing the stream pointer.
|
|
380
440
|
|
|
381
441
|
```ts
|
|
382
442
|
peekBuffer(uint8Array: Uint8Array, options?: IReadChunkOptions): Promise<number>;
|
|
383
443
|
```
|
|
384
444
|
|
|
385
|
-
| Parameter
|
|
386
|
-
|
|
387
|
-
| buffer
|
|
388
|
-
| options
|
|
445
|
+
| Parameter | Type | Description |
|
|
446
|
+
| --------- | ------------------------------------------------- | ------------------------------------------------------------------------- |
|
|
447
|
+
| buffer | Buffer | Uint8Array | Target buffer to write the data read (peeked) to. |
|
|
448
|
+
| options | [IReadChunkOptions](#ireadchunkoptions-interface) | Optional settings for the length, position, and handling of partial reads |
|
|
389
449
|
|
|
390
450
|
Return value `Promise<number>` Promise with number of bytes read. The number of bytes read may be less if the `mayBeLess` flag was set.
|
|
391
451
|
|
|
@@ -397,53 +457,53 @@ when they cross EOF. Zero-length peeks remain no-ops.
|
|
|
397
457
|
|
|
398
458
|
#### `readToken` function
|
|
399
459
|
|
|
400
|
-
Read a
|
|
460
|
+
Read a _token_ from the tokenizer-stream.
|
|
401
461
|
|
|
402
462
|
```ts
|
|
403
463
|
readToken<Value>(token: IGetToken<Value>, position: number = this.position): Promise<Value>
|
|
404
|
-
```
|
|
464
|
+
```
|
|
405
465
|
|
|
406
|
-
| Parameter
|
|
407
|
-
|
|
408
|
-
| token
|
|
409
|
-
| position?
|
|
466
|
+
| Parameter | Type | Description |
|
|
467
|
+
| --------- | -------------------------- | ----------------------------------------------------------------------------- |
|
|
468
|
+
| token | [IGetToken](#token-object) | Token to read from the tokenizer-stream. |
|
|
469
|
+
| position? | number | Offset where to begin reading. Defaults to `tokenizer.position` when omitted. |
|
|
410
470
|
|
|
411
471
|
Returns a `Promise<Value>` resolving to the decoded token value. Rejects with `EndOfStreamError` if fewer than `token.len` bytes are available.
|
|
412
472
|
|
|
413
473
|
#### `peekToken` function
|
|
414
474
|
|
|
415
|
-
Peek a
|
|
475
|
+
Peek a _token_ from the [_tokenizer_](#tokenizer-object).
|
|
416
476
|
|
|
417
477
|
```ts
|
|
418
478
|
peekToken<Value>(token: IGetToken<Value>, position: number | null = this.position): Promise<Value>
|
|
419
479
|
```
|
|
420
480
|
|
|
421
|
-
| Parameter
|
|
422
|
-
|
|
423
|
-
| token
|
|
424
|
-
| position?
|
|
481
|
+
| Parameter | Type | Description |
|
|
482
|
+
| --------- | -------------------------- | --------------------------------------------------------------------------------------- |
|
|
483
|
+
| token | [IGetToken](#token-object) | Token to peek from the tokenizer-stream. |
|
|
484
|
+
| position? | number | null | Offset where to begin peeking. Defaults to `tokenizer.position` when omitted or `null`. |
|
|
425
485
|
|
|
426
|
-
Return a promise with the token value peeked from the [
|
|
486
|
+
Return a promise with the token value peeked from the [_tokenizer_](#tokenizer-object).
|
|
427
487
|
|
|
428
488
|
#### `readNumber` function
|
|
429
489
|
|
|
430
|
-
Read a numeric [
|
|
490
|
+
Read a numeric [_token_](#token-object) from the [_tokenizer_](#tokenizer-object).
|
|
431
491
|
|
|
432
492
|
```ts
|
|
433
493
|
readNumber(token: IToken<number>): Promise<number>
|
|
434
494
|
```
|
|
435
495
|
|
|
436
|
-
| Parameter
|
|
437
|
-
|
|
438
|
-
| token
|
|
496
|
+
| Parameter | Type | Description |
|
|
497
|
+
| --------- | ------------------------------- | ------------------------------------------------ |
|
|
498
|
+
| token | [IToken<number>](#token-object) | Numeric token to read from the tokenizer-stream. |
|
|
439
499
|
|
|
440
|
-
A promise resolving to a numeric value read and decoded from the
|
|
500
|
+
A promise resolving to a numeric value read and decoded from the _tokenizer-stream_.
|
|
441
501
|
Uses a reusable 8-byte buffer, so `token.len` must be at most 8 bytes.
|
|
442
502
|
For larger tokens or values such as `bigint`, use `readToken` instead.
|
|
443
503
|
|
|
444
504
|
#### `peekNumber` function
|
|
445
505
|
|
|
446
|
-
Peek a numeric [
|
|
506
|
+
Peek a numeric [_token_](#token-object) without advancing `tokenizer.position`.
|
|
447
507
|
|
|
448
508
|
```ts
|
|
449
509
|
peekNumber(token: IToken<number>): Promise<number>
|
|
@@ -461,9 +521,9 @@ Advance the offset pointer by the given number of bytes.
|
|
|
461
521
|
ignore(length: number): Promise<number>
|
|
462
522
|
```
|
|
463
523
|
|
|
464
|
-
| Parameter
|
|
465
|
-
|
|
466
|
-
| length
|
|
524
|
+
| Parameter | Type | Description |
|
|
525
|
+
| --------- | ------ | ----------------------------------------------------------------------------------- |
|
|
526
|
+
| length | number | Non-negative safe integer number of bytes to ignore. Advances `tokenizer.position`. |
|
|
467
527
|
|
|
468
528
|
Returns a promise resolving to the number of bytes ignored.
|
|
469
529
|
Buffer, blob, and file tokenizers stop at the known file size. Stream tokenizers reject with `EndOfStreamError` if the requested bytes are unavailable.
|
|
@@ -473,6 +533,7 @@ If the size is unknown, or the stream ends before its advertised size, a failed
|
|
|
473
533
|
skip may consume available bytes before rejecting; `position` reflects those bytes.
|
|
474
534
|
|
|
475
535
|
#### `close` function
|
|
536
|
+
|
|
476
537
|
Clean up resources, such as closing a file pointer if applicable.
|
|
477
538
|
|
|
478
539
|
```ts
|
|
@@ -559,8 +620,8 @@ without moving the cursor backward, subject to safe end-position arithmetic.
|
|
|
559
620
|
|
|
560
621
|
- `position`
|
|
561
622
|
|
|
562
|
-
Pointer to the current position in the [
|
|
563
|
-
For stream tokenizers, a
|
|
623
|
+
Pointer to the current position in the [_tokenizer_](#tokenizer-object) stream.
|
|
624
|
+
For stream tokenizers, a _position_ provided to a _read_ or _peek_ method must be at least this value.
|
|
564
625
|
Buffer, blob, and file tokenizers also support earlier positions.
|
|
565
626
|
|
|
566
627
|
### `ITokenizerOptions` interface
|
|
@@ -568,12 +629,12 @@ without moving the cursor backward, subject to safe end-position arithmetic.
|
|
|
568
629
|
Set these options once when creating a tokenizer. The configured limit applies to subsequent
|
|
569
630
|
reads and peeks for that tokenizer; no per-read limit is needed. Each attribute is optional:
|
|
570
631
|
|
|
571
|
-
| Attribute
|
|
572
|
-
|
|
573
|
-
| maxBufferSize | number
|
|
574
|
-
| fileInfo
|
|
575
|
-
| onClose
|
|
576
|
-
| abortSignal
|
|
632
|
+
| Attribute | Type | Description |
|
|
633
|
+
| ------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
634
|
+
| maxBufferSize | number | Maximum bytes in a dynamically sized tokenizer buffer or stream lookahead span. Disabled by default (`Infinity`). Must be a positive safe integer or `Infinity`. |
|
|
635
|
+
| fileInfo | [IFileInfo](#ifileinfo-interface) | Additional metadata about the input |
|
|
636
|
+
| onClose | `() => Promise<void>` | Asynchronous callback invoked when a buffer, blob, or file tokenizer is closed |
|
|
637
|
+
| abortSignal | [AbortSignal](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal) | Signal used to abort asynchronous stream operations |
|
|
577
638
|
|
|
578
639
|
Creating a tokenizer with an already-aborted `abortSignal` throws or rejects with `AbortError`
|
|
579
640
|
before acquiring file handles or stream readers. Signal listeners are removed when the signal
|
|
@@ -615,8 +676,8 @@ try {
|
|
|
615
676
|
|
|
616
677
|
Extends [ITokenizerOptions](#itokenizeroptions-interface) for `fromFile()`.
|
|
617
678
|
|
|
618
|
-
| Attribute
|
|
619
|
-
|
|
679
|
+
| Attribute | Type | Description |
|
|
680
|
+
| --------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
620
681
|
| regularFileOnly | `boolean` | When `true`, opens with `O_RDONLY \| O_NONBLOCK` and checks that the opened handle refers to a regular file. Defaults to `false`, preserving existing behavior. |
|
|
621
682
|
|
|
622
683
|
Use this option when non-regular inputs, such as FIFOs, should be rejected without waiting for a writer. Validation uses the opened handle, so replacing the pathname before opening does not bypass the check. Symbolic links to regular files are accepted.
|
|
@@ -645,11 +706,11 @@ try {
|
|
|
645
706
|
|
|
646
707
|
Each attribute is optional:
|
|
647
708
|
|
|
648
|
-
| Attribute | Type | Description
|
|
649
|
-
|
|
650
|
-
| length | number | Requested number of bytes to read or peek. Must be a non-negative safe integer no larger than the target buffer length. Defaults to the target buffer length.
|
|
709
|
+
| Attribute | Type | Description |
|
|
710
|
+
| --------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
711
|
+
| length | number | Requested number of bytes to read or peek. Must be a non-negative safe integer no larger than the target buffer length. Defaults to the target buffer length. |
|
|
651
712
|
| position | number | Byte offset to read or peek from. Must be a non-negative safe integer. Defaults to [tokenizer.position](#tokenizer-attributes) when omitted. Earlier positions require random access. |
|
|
652
|
-
| mayBeLess | boolean | If `true`, allows fewer than the requested number of bytes to be read or peeked instead of throwing `EndOfStreamError`. Defaults to `false`.
|
|
713
|
+
| mayBeLess | boolean | If `true`, allows fewer than the requested number of bytes to be read or peeked instead of throwing `EndOfStreamError`. Defaults to `false`. |
|
|
653
714
|
|
|
654
715
|
Invalid lengths or positions, including a `position + length` that exceeds the safe
|
|
655
716
|
integer range, reject with `RangeError`. Requesting an earlier position on a tokenizer
|
|
@@ -658,29 +719,28 @@ target buffer. To write at an offset, pass `buffer.subarray(offset)` as the targ
|
|
|
658
719
|
|
|
659
720
|
Example usage:
|
|
660
721
|
```js
|
|
661
|
-
|
|
722
|
+
const bytesPeeked = await tokenizer.peekBuffer(buffer, { mayBeLess: true });
|
|
662
723
|
```
|
|
663
724
|
|
|
664
725
|
### `IFileInfo` interface
|
|
665
726
|
|
|
666
727
|
Provides optional metadata about the file being tokenized.
|
|
667
728
|
|
|
668
|
-
| Attribute | Type
|
|
669
|
-
|
|
670
|
-
| size | number
|
|
671
|
-
| mimeType | string
|
|
672
|
-
| path | string
|
|
673
|
-
| url | string
|
|
729
|
+
| Attribute | Type | Description |
|
|
730
|
+
| --------- | ------ | ------------------------------------------------------------------------------------------------- |
|
|
731
|
+
| size | number | File size in bytes |
|
|
732
|
+
| mimeType | string | [MIME-type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) of file. |
|
|
733
|
+
| path | string | File path |
|
|
734
|
+
| url | string | File URL |
|
|
674
735
|
|
|
675
736
|
### `Token` object
|
|
676
737
|
|
|
677
|
-
The
|
|
678
|
-
A basic set of
|
|
738
|
+
The _token_ is basically a description of what to read from the [_tokenizer-stream_](#tokenizer-object).
|
|
739
|
+
A basic set of _token types_ can be found here: [_token-types_](https://github.com/Borewit/token-types).
|
|
679
740
|
|
|
680
741
|
A token is something which implements the following interface:
|
|
681
742
|
```ts
|
|
682
743
|
export interface IGetToken<T> {
|
|
683
|
-
|
|
684
744
|
/**
|
|
685
745
|
* Length in bytes of encoded value
|
|
686
746
|
*/
|
|
@@ -694,7 +754,7 @@ export interface IGetToken<T> {
|
|
|
694
754
|
get(buf: Uint8Array, off: number): T;
|
|
695
755
|
}
|
|
696
756
|
```
|
|
697
|
-
The
|
|
757
|
+
The _tokenizer_ reads `token.len` bytes from the _tokenizer-stream_ into a Buffer.
|
|
698
758
|
The `token.get` will be called with the Buffer. `token.get` is responsible for conversion from the buffer to the desired output type.
|
|
699
759
|
|
|
700
760
|
### Working with a fetch response
|
|
@@ -706,7 +766,6 @@ import { fromWebStream } from 'strtok3';
|
|
|
706
766
|
import * as Token from 'token-types';
|
|
707
767
|
|
|
708
768
|
(async () => {
|
|
709
|
-
|
|
710
769
|
const response = await fetch(url);
|
|
711
770
|
if (!response.ok || !response.body) {
|
|
712
771
|
throw new Error(`Cannot read response: ${response.status}`);
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
/** Thrown before a
|
|
1
|
+
/** Thrown before a tokenizer or writer buffer exceeds its configured limit. */
|
|
2
2
|
export class BufferSizeError extends RangeError {
|
|
3
3
|
/**
|
|
4
4
|
* Create an error describing an allocation that exceeds the configured limit.
|