@kaeawc/auto-mobile 0.0.89 → 0.0.91

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -8,24 +8,25 @@
8
8
  [![Codecov](https://codecov.io/gh/Borewit/strtok3/branch/master/graph/badge.svg)](https://codecov.io/gh/Borewit/strtok3)
9
9
  # strtok3
10
10
 
11
- A promise-based streaming [*tokenizer*](#tokenizer-object) for [Node.js](https://nodejs.org) and browsers.
11
+ A promise-based streaming [_tokenizer_](#tokenizer-object) for [Node.js](https://nodejs.org) and browsers.
12
12
 
13
- The `strtok3` module provides several methods for creating a [*tokenizer*](#tokenizer-object) from various input sources.
13
+ The `strtok3` module provides several methods for creating a [_tokenizer_](#tokenizer-object) from various input sources.
14
14
  Designed for:
15
- * Seamless support in streaming environments.
16
- * Efficiently decode binary data, strings, and numbers.
17
- * Reading [predefined](https://github.com/Borewit/token-types) or custom tokens.
18
- * Offering [*tokenizers*](#tokenizer-object) for reading from [files](#fromfile-function), [streams](#fromstream-function) or [Uint8Arrays](#frombuffer-function).
15
+ - Seamless support in streaming environments.
16
+ - Efficiently decode binary data, strings, and numbers.
17
+ - Reading [predefined](https://github.com/Borewit/token-types) or custom tokens.
18
+ - Offering [_tokenizers_](#tokenizer-object) for reading from [files](#fromfile-function), [streams](#fromstream-function) or [Uint8Arrays](#frombuffer-function).
19
19
 
20
20
  ### Features
21
+
21
22
  `strtok3` can read from:
22
- * Files, using a file path as input.
23
- * Node.js [streams](https://nodejs.org/api/stream.html).
24
- * WHATWG [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) objects containing `Uint8Array` chunks.
25
- * [Buffer](https://nodejs.org/api/buffer.html) or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array).
26
- * [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) objects.
27
- * HTTP chunked transfer provided by [@tokenizer/http](https://github.com/Borewit/tokenizer-http).
28
- * [Amazon S3](https://aws.amazon.com/s3) chunks with [@tokenizer/s3](https://github.com/Borewit/tokenizer-s3).
23
+ - Files, using a file path as input.
24
+ - Node.js [streams](https://nodejs.org/api/stream.html).
25
+ - WHATWG [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) objects containing `Uint8Array` chunks.
26
+ - [Buffer](https://nodejs.org/api/buffer.html) or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array).
27
+ - [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) objects.
28
+ - HTTP chunked transfer provided by [@tokenizer/http](https://github.com/Borewit/tokenizer-http).
29
+ - [Amazon S3](https://aws.amazon.com/s3) chunks with [@tokenizer/s3](https://github.com/Borewit/tokenizer-s3).
29
30
 
30
31
  ## Installation
31
32
 
@@ -51,6 +52,7 @@ Coverage is collected in a separate Node.js 24 job and uploaded to Codecov.
51
52
  For local development, running the TypeScript tests with `tsx` requires Node.js 18.19+ (18.x) or Node.js ≥ 20.6.0.
52
53
 
53
54
  ## Support the Project
55
+
54
56
  If you find this project useful and would like to support its development, consider sponsoring or contributing:
55
57
 
56
58
  - [Become a sponsor to Borewit](https://github.com/sponsors/Borewit)
@@ -63,7 +65,7 @@ If you find this project useful and would like to support its development, consi
63
65
 
64
66
  ### strtok3 methods
65
67
 
66
- Use one of the methods to instantiate an [*abstract tokenizer*](#tokenizer-object):
68
+ Use one of the methods to instantiate an [_abstract tokenizer_](#tokenizer-object):
67
69
  - [fromBlob](#fromblob-function)
68
70
  - [fromBuffer](#frombuffer-function)
69
71
  - [fromFile](#fromfile-function)
@@ -84,15 +86,15 @@ without Node.js file-system APIs. In browsers, `strtok3` resolves to this core e
84
86
  Create a tokenizer from a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob).
85
87
 
86
88
  ```ts
87
- function fromBlob(blob: Blob, options?: ITokenizerOptions): BlobTokenizer
89
+ function fromBlob(blob: Blob, options?: ITokenizerOptions): BlobTokenizer;
88
90
  ```
89
91
 
90
- | Parameter | Optional | Type | Description |
91
- |-----------|-----------|---------------------------------------------------|----------------------------------------------------------------------------------------|
92
- | blob | no | [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) | [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) to read from |
93
- | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
92
+ | Parameter | Optional | Type | Description |
93
+ | --------- | -------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
94
+ | blob | no | [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) | [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) to read from |
95
+ | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
94
96
 
95
- Returns a [*tokenizer*](#tokenizer-object).
97
+ Returns a [_tokenizer_](#tokenizer-object).
96
98
 
97
99
  ```js
98
100
  import { fromBlob } from 'strtok3';
@@ -104,11 +106,10 @@ async function parse() {
104
106
  const tokenizer = fromBlob(blob);
105
107
 
106
108
  const myUint8Number = await tokenizer.readToken(Token.UINT8);
107
- console.log(`My number: ${myUint8Number}`);
109
+ console.log(`My number: ${myUint8Number}`);
108
110
  }
109
111
 
110
112
  parse();
111
-
112
113
  ```
113
114
 
114
115
  #### `fromBuffer()` function
@@ -116,15 +117,15 @@ parse();
116
117
  Create a tokenizer from memory ([Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) or Node.js [Buffer](https://nodejs.org/api/buffer.html)).
117
118
 
118
119
  ```ts
119
- function fromBuffer(uint8Array: Uint8Array, options?: ITokenizerOptions): BufferTokenizer
120
+ function fromBuffer(uint8Array: Uint8Array, options?: ITokenizerOptions): BufferTokenizer;
120
121
  ```
121
122
 
122
- | Parameter | Optional | Type | Description |
123
- |------------|----------|--------------------------------------------------|-----------------------------------|
123
+ | Parameter | Optional | Type | Description |
124
+ | ---------- | -------- | --------------------------------------------------------------------------------------------------------- | --------------------------------- |
124
125
  | uint8Array | no | [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array) | Buffer or Uint8Array to read from |
125
- | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
126
+ | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
126
127
 
127
- Returns a [*tokenizer*](#tokenizer-object).
128
+ Returns a [_tokenizer_](#tokenizer-object).
128
129
 
129
130
  ```js
130
131
  import { fromBuffer } from 'strtok3';
@@ -142,22 +143,22 @@ parse();
142
143
 
143
144
  #### `fromFile` function
144
145
 
145
- Creates a [*tokenizer*](#tokenizer-object) from a local file.
146
+ Creates a [_tokenizer_](#tokenizer-object) from a local file.
146
147
 
147
148
  ```ts
148
- function fromFile(sourceFilePath: string, options?: IFromFileOptions): Promise<FileTokenizer>
149
- ```
149
+ function fromFile(sourceFilePath: string, options?: IFromFileOptions): Promise<FileTokenizer>;
150
+ ```
150
151
 
151
- | Parameter | Type | Description |
152
- |----------------|----------|----------------------------|
153
- | sourceFilePath | `string` | Path to file to read from |
154
- | options | [IFromFileOptions](#ifromfileoptions-interface) | Optional file tokenizer options |
152
+ | Parameter | Type | Description |
153
+ | -------------- | ----------------------------------------------- | ------------------------------- |
154
+ | sourceFilePath | `string` | Path to file to read from |
155
+ | options | [IFromFileOptions](#ifromfileoptions-interface) | Optional file tokenizer options |
155
156
 
156
157
  > [!NOTE]
157
158
  > - Only available for Node.js engines
158
159
  > - `fromFile` automatically embeds [file information](#ifileinfo-interface)
159
160
 
160
- A Promise resolving to a [*tokenizer*](#tokenizer-object) which can be used to parse a file.
161
+ A Promise resolving to a [_tokenizer_](#tokenizer-object) which can be used to parse a file.
161
162
 
162
163
  ```js
163
164
  import { fromFile } from 'strtok3';
@@ -174,8 +175,6 @@ async function parse() {
174
175
  }
175
176
 
176
177
  parse();
177
-
178
-
179
178
  ```
180
179
 
181
180
  #### `fromStream()` function
@@ -183,15 +182,15 @@ parse();
183
182
  Create a tokenizer from a Node.js [readable stream](https://nodejs.org/api/stream.html#readable-streams) containing binary data.
184
183
 
185
184
  ```ts
186
- function fromStream(stream: Readable, options?: ITokenizerOptions): Promise<ReadStreamTokenizer>
185
+ function fromStream(stream: Readable, options?: ITokenizerOptions): Promise<ReadStreamTokenizer>;
187
186
  ```
188
187
 
189
- | Parameter | Optional | Type | Description |
190
- |-----------|----------|------|-------------|
191
- | stream | no | Readable | Node.js readable stream to read from |
192
- | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
188
+ | Parameter | Optional | Type | Description |
189
+ | --------- | -------- | ------------------------------------------------- | ------------------------------------ |
190
+ | stream | no | Readable | Node.js readable stream to read from |
191
+ | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
193
192
 
194
- When importing from `strtok3` in Node.js, returns a promise resolving to a [*tokenizer*](#tokenizer-object).
193
+ When importing from `strtok3` in Node.js, returns a promise resolving to a [_tokenizer_](#tokenizer-object).
195
194
  For streams created by `createReadStream`, the file path and size are added to `tokenizer.fileInfo`.
196
195
  The `strtok3/core` entry point also exports `fromStream`, but returns the tokenizer synchronously without looking up file information.
197
196
 
@@ -225,15 +224,15 @@ Create a tokenizer from a [WHATWG ReadableStream](https://developer.mozilla.org/
225
224
  Uses a BYOB reader when supported by the stream, otherwise a default reader. Source cleanup differs between these readers; see [`close()`](#close-function) and [`abort()`](#abort-function).
226
225
 
227
226
  ```ts
228
- function fromWebStream(webStream: AnyWebByteStream, options?: ITokenizerOptions): ReadStreamTokenizer
227
+ function fromWebStream(webStream: AnyWebByteStream, options?: ITokenizerOptions): ReadStreamTokenizer;
229
228
  ```
230
229
 
231
- | Parameter | Optional | Type | Description |
232
- |----------------|----------|--------------------------------------------------------------------------|------------------------------------|
233
- | webStream | no | [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) | WHATWG ReadableStream to read from |
234
- | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
230
+ | Parameter | Optional | Type | Description |
231
+ | --------- | -------- | --------------------------------------------------------------------------------- | ---------------------------------- |
232
+ | webStream | no | [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) | WHATWG ReadableStream to read from |
233
+ | options | yes | [ITokenizerOptions](#itokenizeroptions-interface) | Tokenizer options |
235
234
 
236
- Returns a [*tokenizer*](#tokenizer-object).
235
+ Returns a [_tokenizer_](#tokenizer-object).
237
236
 
238
237
  ```js
239
238
  import { fromWebStream } from 'strtok3';
@@ -258,7 +257,7 @@ Borrow a finite range of an existing tokenizer without buffering it or taking
258
257
  ownership of its input. Available from both `strtok3` and `strtok3/core`.
259
258
 
260
259
  ```ts
261
- function createTokenizerView(source: ITokenizer, length: number): ITokenizerView
260
+ function createTokenizerView(source: ITokenizer, length: number): ITokenizerView;
262
261
  ```
263
262
 
264
263
  The view starts at `source.position`, with local `position` zero and
@@ -275,12 +274,12 @@ await source.ignore(1);
275
274
  const chunk = createTokenizerView(source, 3); // source bytes [20, 30, 40]
276
275
 
277
276
  console.log(await chunk.readToken(UINT8)); // 20; chunk.position = 1
278
- const child = chunk.createView(1); // same as createTokenizerView(chunk, 1)
277
+ const child = chunk.createView(1); // same as createTokenizerView(chunk, 1)
279
278
  console.log(await child.readToken(UINT8)); // 30; chunk.position = 2
280
- await child.close(); // source stays open, at position 3
279
+ await child.close(); // source stays open, at position 3
281
280
 
282
- console.log(chunk.remaining); // 1
283
- await chunk.consumeRemainder(); // exactly consume 40, or throw on truncation
281
+ console.log(chunk.remaining); // 1
282
+ await chunk.consumeRemainder(); // exactly consume 40, or throw on truncation
284
283
  await chunk.close();
285
284
  console.log(await source.readToken(UINT8)); // 50
286
285
  await source.close();
@@ -329,8 +328,69 @@ by a view, so callers must keep those sequential too. Explicit `abort()`
329
328
  propagates to the shared source to cancel pending input and affects its other
330
329
  users; it also closes the view.
331
330
 
331
+ ### `TokenWriter` class
332
+
333
+ Serialize binary data synchronously into a growable in-memory buffer with the
334
+ same token definitions used for reading.
335
+ Available from both `strtok3` and `strtok3/core`.
336
+
337
+ ```ts
338
+ const writer = new TokenWriter(); // optional initial capacity in bytes
339
+ const cappedWriter = new TokenWriter(undefined, { maxBufferSize: 1024 * 1024 });
340
+
341
+ writer.length; // bytes written so far
342
+ writer.reserve(length); // reserve zero-filled region, returns start offset
343
+ writer.writeToken(token, value); // encode a token, returns bytes written
344
+ writer.writeBuffer(value); // copy raw bytes, returns bytes written
345
+ writer.writeBytes(value); // alias for writeBuffer
346
+ writer.writeZeros(length); // append zeros, returns bytes written
347
+ writer.patchToken(token, offset, value); // overwrite an earlier region
348
+ writer.toUint8Array(); // copy of the written bytes
349
+ ```
350
+
351
+ ```js
352
+ import { TokenWriter } from 'strtok3';
353
+ import { UINT24_BE, UINT32_LE } from 'token-types';
354
+
355
+ const writer = new TokenWriter();
356
+ writer.writeToken(UINT32_LE, 0x01020304);
357
+ const blockLengthOffset = writer.reserve(3);
358
+ const bytesWritten = writer.writeBuffer(new Uint8Array([0xaa, 0xbb])); // 2
359
+ writer.patchToken(UINT24_BE, blockLengthOffset, 2);
360
+
361
+ console.log(writer.length); // 9
362
+ const data = writer.toUint8Array(); // Uint8Array(9)
363
+ ```
364
+
365
+ Append methods return the number of bytes written, like `readBuffer()` returns
366
+ the number read; `writer.length` gives the resulting total. `writeToken()` writes
367
+ exactly `token.len` bytes and ignores the encoder's returned offset.
368
+ `reserve()` returns the starting offset of a zero-filled region for back-filling
369
+ with `patchToken()`. Patching outside the written region throws a `RangeError`;
370
+ successful patches leave `length` unchanged and return nothing.
371
+
372
+ Errors from `put()` propagate without changing the written bytes or `length`:
373
+ failed appends clear partial bytes before reuse, and failed patches restore the
374
+ original region. Encoders must write synchronously only within `token.len` bytes
375
+ starting at the supplied offset and must not retain the destination buffer.
376
+
377
+ `writeBuffer()` and its `writeBytes()` alias copy their input. `toUint8Array()`
378
+ returns a snapshot that can be modified without changing the writer. The writer
379
+ buffers the complete output; snapshots allocate another copy of the written bytes.
380
+
381
+ The optional `maxBufferSize` caps both total output length and backing-buffer
382
+ capacity. Like the tokenizer option, it defaults to `Infinity` and must be a
383
+ positive safe integer or `Infinity`. Requests exceeding the cap throw
384
+ `BufferSizeError` before allocation, encoding or changes to the output.
385
+ An explicit initial capacity above the cap is rejected. When omitted, the initial
386
+ capacity defaults to 1024 bytes or the cap, whichever is smaller; internal minimum
387
+ capacity and buffer growth also respect the cap. Empty appends and in-range patches
388
+ remain allowed at the limit. The cap does not bound total process memory, caller
389
+ inputs, snapshots, patch rollback copies or encoder allocations.
390
+
332
391
  ### `Tokenizer` object
333
- The *tokenizer* is an abstraction of a [stream](https://nodejs.org/api/stream.html), file or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array), allowing _reading_ or _peeking_ from the stream.
392
+
393
+ The _tokenizer_ is an abstraction of a [stream](https://nodejs.org/api/stream.html), file or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array), allowing _reading_ or _peeking_ from the stream.
334
394
  It can also be translated in chunked reads, as done in [@tokenizer/http](https://github.com/Borewit/tokenizer-http);
335
395
 
336
396
  #### Key Features:
@@ -345,8 +405,8 @@ It can also be translated in chunked reads, as done in [@tokenizer/http](https:/
345
405
  _Read_ methods advance the stream pointer, while _peek_ methods do not.
346
406
 
347
407
  There are two kinds of functions:
348
- 1. *read* methods: used to read a *token* or bytes into a buffer from the [*tokenizer*](#tokenizer-object). After a successful read, `tokenizer.position` is the starting position of the read plus the number of bytes read. An explicit position can skip ahead or, for random-access tokenizers, move backwards.
349
- 2. *peek* methods: same as the read, but it will *not* advance the pointer. It allows to read (peek) ahead.
408
+ 1. _read_ methods: used to read a _token_ or bytes into a buffer from the [_tokenizer_](#tokenizer-object). After a successful read, `tokenizer.position` is the starting position of the read plus the number of bytes read. An explicit position can skip ahead or, for random-access tokenizers, move backwards.
409
+ 2. _peek_ methods: same as the read, but it will _not_ advance the pointer. It allows to read (peek) ahead.
350
410
 
351
411
  Reads and peeks require the full requested length unless `mayBeLess: true` is passed to
352
412
  `readBuffer` or `peekBuffer`. Otherwise, insufficient input rejects with the exported
@@ -366,26 +426,26 @@ Read data from the _tokenizer_ into provided "buffer" (`Uint8Array`).
366
426
  readBuffer(buffer: Uint8Array, options?: IReadChunkOptions): Promise<number>;
367
427
  ```
368
428
 
369
- | Parameter | Type | Description |
370
- |------------|----------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
371
- | buffer | [Buffer](https://nodejs.org/api/buffer.html) &#124; Uint8Array | Target buffer to write the data read to |
372
- | options | [IReadChunkOptions](#ireadchunkoptions-interface) | Optional settings for the length, position, and handling of partial reads |
429
+ | Parameter | Type | Description |
430
+ | --------- | -------------------------------------------------------------- | ------------------------------------------------------------------------- |
431
+ | buffer | [Buffer](https://nodejs.org/api/buffer.html) &#124; Uint8Array | Target buffer to write the data read to |
432
+ | options | [IReadChunkOptions](#ireadchunkoptions-interface) | Optional settings for the length, position, and handling of partial reads |
373
433
 
374
434
  Return promise with number of bytes read.
375
435
  The number of bytes read may be less than requested if the `mayBeLess` flag is set.
376
436
 
377
437
  #### `peekBuffer` function
378
438
 
379
- Peek (read ahead), from [*tokenizer*](#tokenizer-object), into the buffer without advancing the stream pointer.
439
+ Peek (read ahead), from [_tokenizer_](#tokenizer-object), into the buffer without advancing the stream pointer.
380
440
 
381
441
  ```ts
382
442
  peekBuffer(uint8Array: Uint8Array, options?: IReadChunkOptions): Promise<number>;
383
443
  ```
384
444
 
385
- | Parameter | Type | Description |
386
- |------------|-----------------------------------------|-----------------------------------------------------|
387
- | buffer | Buffer &#124; Uint8Array | Target buffer to write the data read (peeked) to. |
388
- | options | [IReadChunkOptions](#ireadchunkoptions-interface) | Optional settings for the length, position, and handling of partial reads |
445
+ | Parameter | Type | Description |
446
+ | --------- | ------------------------------------------------- | ------------------------------------------------------------------------- |
447
+ | buffer | Buffer &#124; Uint8Array | Target buffer to write the data read (peeked) to. |
448
+ | options | [IReadChunkOptions](#ireadchunkoptions-interface) | Optional settings for the length, position, and handling of partial reads |
389
449
 
390
450
  Return value `Promise<number>` Promise with number of bytes read. The number of bytes read may be less if the `mayBeLess` flag was set.
391
451
 
@@ -397,53 +457,53 @@ when they cross EOF. Zero-length peeks remain no-ops.
397
457
 
398
458
  #### `readToken` function
399
459
 
400
- Read a *token* from the tokenizer-stream.
460
+ Read a _token_ from the tokenizer-stream.
401
461
 
402
462
  ```ts
403
463
  readToken<Value>(token: IGetToken<Value>, position: number = this.position): Promise<Value>
404
- ```
464
+ ```
405
465
 
406
- | Parameter | Type | Description |
407
- |------------|-------------------------|---------------------------------------------------------------------------------------------------------------------- |
408
- | token | [IGetToken](#token-object) | Token to read from the tokenizer-stream. |
409
- | position? | number | Offset where to begin reading. Defaults to `tokenizer.position` when omitted. |
466
+ | Parameter | Type | Description |
467
+ | --------- | -------------------------- | ----------------------------------------------------------------------------- |
468
+ | token | [IGetToken](#token-object) | Token to read from the tokenizer-stream. |
469
+ | position? | number | Offset where to begin reading. Defaults to `tokenizer.position` when omitted. |
410
470
 
411
471
  Returns a `Promise<Value>` resolving to the decoded token value. Rejects with `EndOfStreamError` if fewer than `token.len` bytes are available.
412
472
 
413
473
  #### `peekToken` function
414
474
 
415
- Peek a *token* from the [*tokenizer*](#tokenizer-object).
475
+ Peek a _token_ from the [_tokenizer_](#tokenizer-object).
416
476
 
417
477
  ```ts
418
478
  peekToken<Value>(token: IGetToken<Value>, position: number | null = this.position): Promise<Value>
419
479
  ```
420
480
 
421
- | Parameter | Type | Description |
422
- |------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------|
423
- | token | [IGetToken](#token-object) | Token to peek from the tokenizer-stream. |
424
- | position? | number &#124; null | Offset where to begin peeking. Defaults to `tokenizer.position` when omitted or `null`. |
481
+ | Parameter | Type | Description |
482
+ | --------- | -------------------------- | --------------------------------------------------------------------------------------- |
483
+ | token | [IGetToken](#token-object) | Token to peek from the tokenizer-stream. |
484
+ | position? | number &#124; null | Offset where to begin peeking. Defaults to `tokenizer.position` when omitted or `null`. |
425
485
 
426
- Return a promise with the token value peeked from the [*tokenizer*](#tokenizer-object).
486
+ Return a promise with the token value peeked from the [_tokenizer_](#tokenizer-object).
427
487
 
428
488
  #### `readNumber` function
429
489
 
430
- Read a numeric [*token*](#token-object) from the [*tokenizer*](#tokenizer-object).
490
+ Read a numeric [_token_](#token-object) from the [_tokenizer_](#tokenizer-object).
431
491
 
432
492
  ```ts
433
493
  readNumber(token: IToken<number>): Promise<number>
434
494
  ```
435
495
 
436
- | Parameter | Type | Description |
437
- |------------|---------------------------------|----------------------------------------------------|
438
- | token | [IToken<number>](#token-object) | Numeric token to read from the tokenizer-stream. |
496
+ | Parameter | Type | Description |
497
+ | --------- | ------------------------------- | ------------------------------------------------ |
498
+ | token | [IToken<number>](#token-object) | Numeric token to read from the tokenizer-stream. |
439
499
 
440
- A promise resolving to a numeric value read and decoded from the *tokenizer-stream*.
500
+ A promise resolving to a numeric value read and decoded from the _tokenizer-stream_.
441
501
  Uses a reusable 8-byte buffer, so `token.len` must be at most 8 bytes.
442
502
  For larger tokens or values such as `bigint`, use `readToken` instead.
443
503
 
444
504
  #### `peekNumber` function
445
505
 
446
- Peek a numeric [*token*](#token-object) without advancing `tokenizer.position`.
506
+ Peek a numeric [_token_](#token-object) without advancing `tokenizer.position`.
447
507
 
448
508
  ```ts
449
509
  peekNumber(token: IToken<number>): Promise<number>
@@ -461,9 +521,9 @@ Advance the offset pointer by the given number of bytes.
461
521
  ignore(length: number): Promise<number>
462
522
  ```
463
523
 
464
- | Parameter | Type | Description |
465
- |------------|--------|------------------------------------------------------------------|
466
- | length | number | Non-negative safe integer number of bytes to ignore. Advances `tokenizer.position`. |
524
+ | Parameter | Type | Description |
525
+ | --------- | ------ | ----------------------------------------------------------------------------------- |
526
+ | length | number | Non-negative safe integer number of bytes to ignore. Advances `tokenizer.position`. |
467
527
 
468
528
  Returns a promise resolving to the number of bytes ignored.
469
529
  Buffer, blob, and file tokenizers stop at the known file size. Stream tokenizers reject with `EndOfStreamError` if the requested bytes are unavailable.
@@ -473,6 +533,7 @@ If the size is unknown, or the stream ends before its advertised size, a failed
473
533
  skip may consume available bytes before rejecting; `position` reflects those bytes.
474
534
 
475
535
  #### `close` function
536
+
476
537
  Clean up resources, such as closing a file pointer if applicable.
477
538
 
478
539
  ```ts
@@ -559,8 +620,8 @@ without moving the cursor backward, subject to safe end-position arithmetic.
559
620
 
560
621
  - `position`
561
622
 
562
- Pointer to the current position in the [*tokenizer*](#tokenizer-object) stream.
563
- For stream tokenizers, a *position* provided to a _read_ or _peek_ method must be at least this value.
623
+ Pointer to the current position in the [_tokenizer_](#tokenizer-object) stream.
624
+ For stream tokenizers, a _position_ provided to a _read_ or _peek_ method must be at least this value.
564
625
  Buffer, blob, and file tokenizers also support earlier positions.
565
626
 
566
627
  ### `ITokenizerOptions` interface
@@ -568,12 +629,12 @@ without moving the cursor backward, subject to safe end-position arithmetic.
568
629
  Set these options once when creating a tokenizer. The configured limit applies to subsequent
569
630
  reads and peeks for that tokenizer; no per-read limit is needed. Each attribute is optional:
570
631
 
571
- | Attribute | Type | Description |
572
- |-----------|------|-------------|
573
- | maxBufferSize | number | Maximum bytes in a dynamically sized tokenizer buffer or stream lookahead span. Disabled by default (`Infinity`). Must be a positive safe integer or `Infinity`. |
574
- | fileInfo | [IFileInfo](#ifileinfo-interface) | Additional metadata about the input |
575
- | onClose | `() => Promise<void>` | Asynchronous callback invoked when a buffer, blob, or file tokenizer is closed |
576
- | abortSignal | [AbortSignal](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal) | Signal used to abort asynchronous stream operations |
632
+ | Attribute | Type | Description |
633
+ | ------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
634
+ | maxBufferSize | number | Maximum bytes in a dynamically sized tokenizer buffer or stream lookahead span. Disabled by default (`Infinity`). Must be a positive safe integer or `Infinity`. |
635
+ | fileInfo | [IFileInfo](#ifileinfo-interface) | Additional metadata about the input |
636
+ | onClose | `() => Promise<void>` | Asynchronous callback invoked when a buffer, blob, or file tokenizer is closed |
637
+ | abortSignal | [AbortSignal](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal) | Signal used to abort asynchronous stream operations |
577
638
 
578
639
  Creating a tokenizer with an already-aborted `abortSignal` throws or rejects with `AbortError`
579
640
  before acquiring file handles or stream readers. Signal listeners are removed when the signal
@@ -615,8 +676,8 @@ try {
615
676
 
616
677
  Extends [ITokenizerOptions](#itokenizeroptions-interface) for `fromFile()`.
617
678
 
618
- | Attribute | Type | Description |
619
- |-----------|------|-------------|
679
+ | Attribute | Type | Description |
680
+ | --------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
620
681
  | regularFileOnly | `boolean` | When `true`, opens with `O_RDONLY \| O_NONBLOCK` and checks that the opened handle refers to a regular file. Defaults to `false`, preserving existing behavior. |
621
682
 
622
683
  Use this option when non-regular inputs, such as FIFOs, should be rejected without waiting for a writer. Validation uses the opened handle, so replacing the pathname before opening does not bypass the check. Symbolic links to regular files are accepted.
@@ -645,11 +706,11 @@ try {
645
706
 
646
707
  Each attribute is optional:
647
708
 
648
- | Attribute | Type | Description |
649
- |-----------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
650
- | length | number | Requested number of bytes to read or peek. Must be a non-negative safe integer no larger than the target buffer length. Defaults to the target buffer length. |
709
+ | Attribute | Type | Description |
710
+ | --------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
711
+ | length | number | Requested number of bytes to read or peek. Must be a non-negative safe integer no larger than the target buffer length. Defaults to the target buffer length. |
651
712
  | position | number | Byte offset to read or peek from. Must be a non-negative safe integer. Defaults to [tokenizer.position](#tokenizer-attributes) when omitted. Earlier positions require random access. |
652
- | mayBeLess | boolean | If `true`, allows fewer than the requested number of bytes to be read or peeked instead of throwing `EndOfStreamError`. Defaults to `false`. |
713
+ | mayBeLess | boolean | If `true`, allows fewer than the requested number of bytes to be read or peeked instead of throwing `EndOfStreamError`. Defaults to `false`. |
653
714
 
654
715
  Invalid lengths or positions, including a `position + length` that exceeds the safe
655
716
  integer range, reject with `RangeError`. Requesting an earlier position on a tokenizer
@@ -658,29 +719,28 @@ target buffer. To write at an offset, pass `buffer.subarray(offset)` as the targ
658
719
 
659
720
  Example usage:
660
721
  ```js
661
- const bytesPeeked = await tokenizer.peekBuffer(buffer, {mayBeLess: true});
722
+ const bytesPeeked = await tokenizer.peekBuffer(buffer, { mayBeLess: true });
662
723
  ```
663
724
 
664
725
  ### `IFileInfo` interface
665
726
 
666
727
  Provides optional metadata about the file being tokenized.
667
728
 
668
- | Attribute | Type | Description |
669
- |-----------|---------|---------------------------------------------------------------------------------------------------|
670
- | size | number | File size in bytes |
671
- | mimeType | string | [MIME-type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) of file. |
672
- | path | string | File path |
673
- | url | string | File URL |
729
+ | Attribute | Type | Description |
730
+ | --------- | ------ | ------------------------------------------------------------------------------------------------- |
731
+ | size | number | File size in bytes |
732
+ | mimeType | string | [MIME-type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) of file. |
733
+ | path | string | File path |
734
+ | url | string | File URL |
674
735
 
675
736
  ### `Token` object
676
737
 
677
- The *token* is basically a description of what to read from the [*tokenizer-stream*](#tokenizer-object).
678
- A basic set of *token types* can be found here: [*token-types*](https://github.com/Borewit/token-types).
738
+ The _token_ is basically a description of what to read from the [_tokenizer-stream_](#tokenizer-object).
739
+ A basic set of _token types_ can be found here: [_token-types_](https://github.com/Borewit/token-types).
679
740
 
680
741
  A token is something which implements the following interface:
681
742
  ```ts
682
743
  export interface IGetToken<T> {
683
-
684
744
  /**
685
745
  * Length in bytes of encoded value
686
746
  */
@@ -694,7 +754,7 @@ export interface IGetToken<T> {
694
754
  get(buf: Uint8Array, off: number): T;
695
755
  }
696
756
  ```
697
- The *tokenizer* reads `token.len` bytes from the *tokenizer-stream* into a Buffer.
757
+ The _tokenizer_ reads `token.len` bytes from the _tokenizer-stream_ into a Buffer.
698
758
  The `token.get` will be called with the Buffer. `token.get` is responsible for conversion from the buffer to the desired output type.
699
759
 
700
760
  ### Working with a fetch response
@@ -706,7 +766,6 @@ import { fromWebStream } from 'strtok3';
706
766
  import * as Token from 'token-types';
707
767
 
708
768
  (async () => {
709
-
710
769
  const response = await fetch(url);
711
770
  if (!response.ok || !response.body) {
712
771
  throw new Error(`Cannot read response: ${response.status}`);
@@ -1,4 +1,4 @@
1
- /** Thrown before a dynamically sized tokenizer buffer exceeds its configured limit. */
1
+ /** Thrown before a tokenizer or writer buffer exceeds its configured limit. */
2
2
  export class BufferSizeError extends RangeError {
3
3
  /**
4
4
  * Create an error describing an allocation that exceeds the configured limit.