@t4h.framework/fs 0.9.5 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.ai/skills/framework-fs-split/SKILL.md +2 -1
- package/CHANGELOG.md +18 -0
- package/README.md +11 -1
- package/dist/activities/FileSplitActivity.d.ts +1 -0
- package/dist/activities/FileSplitActivity.d.ts.map +1 -1
- package/dist/activities/FileSplitActivity.js +4 -1
- package/dist/activities/FileSplitActivity.js.map +1 -1
- package/dist/helpers/byte-delimiter-scanner.d.ts +9 -2
- package/dist/helpers/byte-delimiter-scanner.d.ts.map +1 -1
- package/dist/helpers/byte-delimiter-scanner.js +29 -11
- package/dist/helpers/byte-delimiter-scanner.js.map +1 -1
- package/package.json +3 -3
|
@@ -137,7 +137,8 @@ multi-byte encodings cannot skew the ranges a ranged download will use.
|
|
|
137
137
|
| `every` | `1` | Cut only every N delimiter occurrences — groups N records per child. One million lines with `every: 10_000` start 100 children, each receiving a raw block (inner delimiters untouched) to iterate locally. `keep` arithmetic applies only at actual cut points. |
|
|
138
138
|
| `trim` | none | `{ start?, end? }` **bytes** cut off each end of the file before scanning — a positional header/trailer. Byte-based on purpose: a record-start delimiter (`'\n10'`) fuses the trailer into the last record's slice, so a record-count skip would drop a valid record along with it. Trimmed content never becomes a slice nor counts toward `every`; fetch it with `read`. Positional layouts make the byte counts constants — getting them right is the caller's business. |
|
|
139
139
|
| `skipEmpty` | `true` | Drops zero-length slices (consecutive delimiters, file starting/ending with one) so they never become child workflows. Turn off when slice index must align 1:1 with record position. "Empty" means zero bytes — a whitespace-only line is a regular slice. |
|
|
140
|
-
| `encoding` | `'utf8'` | Converts the delimiter to bytes and
|
|
140
|
+
| `encoding` | `'utf8'` | Converts the delimiter to bytes and decodes each slice into the string the child receives. |
|
|
141
|
+
| `inlineMaxBytes` | `16_384` | A slice up to this many bytes is handed to the child as a string straight from the scan; a larger one as a reference to its byte range in the durable copy, read back where the child runs. Reading a few hundred bytes back costs a row lookup and a ranged object read per child, which is what a million-line split spends its CPU on. `0` hands every slice by range. Keep it well under the runtime's fan-out message budget (1 MiB by default). |
|
|
141
142
|
| `timeoutInMs` | none | Max milliseconds to await the batch; on expiry the activity **rejects** (reason `timeout`) instead of suspending forever — the safety net for a child that never reaches a terminal state. |
|
|
142
143
|
|
|
143
144
|
---
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,23 @@
|
|
|
1
1
|
# @t4h.framework/fs
|
|
2
2
|
|
|
3
|
+
## 0.10.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- [#181](https://github.com/tech4humans-brasil/framework/pull/181) [`9449e4e`](https://github.com/tech4humans-brasil/framework/commit/9449e4e9889f42dea1d52418140f47d25dff0535) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - `split` hands a slice to its child as a string when it fits `inlineMaxBytes`
|
|
8
|
+
(default 16 KiB) instead of a range read back from the durable copy.
|
|
9
|
+
|
|
10
|
+
Every child paid a lookup and a ranged read to fetch a few hundred bytes, so a
|
|
11
|
+
large file cost as many object reads as it has lines. The scanner keeps the
|
|
12
|
+
bytes of the open slice while it is under the limit, so memory stays bounded
|
|
13
|
+
by `inlineMaxBytes` whatever the file size. A larger slice still travels by
|
|
14
|
+
range; `inlineMaxBytes: 0` restores the previous behavior.
|
|
15
|
+
|
|
16
|
+
### Patch Changes
|
|
17
|
+
|
|
18
|
+
- Updated dependencies [[`9449e4e`](https://github.com/tech4humans-brasil/framework/commit/9449e4e9889f42dea1d52418140f47d25dff0535), [`9449e4e`](https://github.com/tech4humans-brasil/framework/commit/9449e4e9889f42dea1d52418140f47d25dff0535)]:
|
|
19
|
+
- @t4h.framework/core@0.15.0
|
|
20
|
+
|
|
3
21
|
## 0.9.5
|
|
4
22
|
|
|
5
23
|
### Patch Changes
|
package/README.md
CHANGED
|
@@ -223,6 +223,7 @@ await split(file, workflow, {
|
|
|
223
223
|
trim: { start: 241, end: 241 }, // bytes cut off each end before scanning
|
|
224
224
|
skipEmpty: false, // keep zero-length slices (default: skipped)
|
|
225
225
|
encoding: 'latin1', // delimiter bytes + slice hydration (default: utf8)
|
|
226
|
+
inlineMaxBytes: 16_384, // slices up to this size travel as strings (default: 16 KiB)
|
|
226
227
|
timeoutInMs: 3_600_000, // max ms to await the batch; rejects on expiry
|
|
227
228
|
})
|
|
228
229
|
```
|
|
@@ -256,6 +257,15 @@ await split(file, workflow, {
|
|
|
256
257
|
- **`encoding`** converts the delimiter to bytes and tells the runtime how to
|
|
257
258
|
hydrate each slice for the child. Offsets are always computed over raw
|
|
258
259
|
bytes, so multi-byte encodings never skew the ranges.
|
|
260
|
+
- **`inlineMaxBytes`** (default 16 KiB) is the size up to which a slice is
|
|
261
|
+
handed to the child as a string, straight from the scan. A slice above it is
|
|
262
|
+
handed as a reference to its byte range in the durable copy, read back
|
|
263
|
+
where the child runs. A line of a CSV is a few hundred bytes: reading it
|
|
264
|
+
back would cost a row lookup and a ranged object read per child, and that,
|
|
265
|
+
not the scan, is what a million-line split spends its CPU on. Set it to `0`
|
|
266
|
+
to hand every slice by range. Keep it well under the fan-out message budget
|
|
267
|
+
of the runtime (`WORKFLOW_BATCH_FANOUT_CHUNK_MAX_BYTES`, 1 MiB by default):
|
|
268
|
+
an inline slice that does not fit a message is refused.
|
|
259
269
|
- **`timeoutInMs`** is the maximum time in milliseconds to await the batch. On
|
|
260
270
|
expiry the activity rejects (reason `timeout`) instead of suspending
|
|
261
271
|
forever — the safety net for a child that never reaches a terminal state.
|
|
@@ -380,5 +390,5 @@ const scanner = new ByteDelimiterScanner({ delimiter: { value: '\n' } })
|
|
|
380
390
|
|
|
381
391
|
await pipeline(source, scanner, destination)
|
|
382
392
|
|
|
383
|
-
scanner.
|
|
393
|
+
scanner.slices // [{ start: 0, end: 3, content: <Buffer> }, ...]
|
|
384
394
|
```
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"FileSplitActivity.d.ts","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,EACb,KAAK,MAAM,EAGX,KAAK,QAAQ,EACb,gBAAgB,EAEhB,mBAAmB,EACpB,MAAM,qBAAqB,CAAA;AAG5B,OAAO,EAEL,KAAK,QAAQ,EACb,KAAK,kBAAkB,EACxB,MAAM,sCAAsC,CAAA;AAE7C,MAAM,MAAM,gBAAgB,GAAG;IAC7B,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,WAAW,CAAC,EAAE,MAAM,CAAA;CACrB,CAAA;AAED,MAAM,MAAM,sBAAsB,GAAG;IACnC,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,MAAM,CAAA;IAChB,OAAO,EAAE,gBAAgB,CAAA;CAC1B,CAAA;AAED,MAAM,MAAM,cAAc,GAAG;IAC3B,KAAK,EAAE,MAAM,CAAA;IACb,KAAK,EAAE,MAAM,CAAA;CACd,CAAA;AAED,qBAAa,iBAAkB,SAAQ,aAAa,CAClD,SAAS;IACP,MAAM,EAAE,MAAM;IACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC;IAC5B,OAAO,EAAE,gBAAgB;CAC1B,EACD,sBAAsB,EACtB,cAAc,EACd,gBAAgB,EAChB,sBAAsB,EACtB,OAAO,mBAAmB,CAC3B;IACC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAGrB;IAEK,OAAO,CACZ,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,sBAAsB;IASZ,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACR,EAAE,sBAAsB;;;;
|
|
1
|
+
{"version":3,"file":"FileSplitActivity.d.ts","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,EACb,KAAK,MAAM,EAGX,KAAK,QAAQ,EACb,gBAAgB,EAEhB,mBAAmB,EACpB,MAAM,qBAAqB,CAAA;AAG5B,OAAO,EAEL,KAAK,QAAQ,EACb,KAAK,kBAAkB,EACxB,MAAM,sCAAsC,CAAA;AAE7C,MAAM,MAAM,gBAAgB,GAAG;IAC7B,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,cAAc,CAAC,EAAE,MAAM,CAAA;IACvB,WAAW,CAAC,EAAE,MAAM,CAAA;CACrB,CAAA;AAED,MAAM,MAAM,sBAAsB,GAAG;IACnC,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,MAAM,CAAA;IAChB,OAAO,EAAE,gBAAgB,CAAA;CAC1B,CAAA;AAED,MAAM,MAAM,cAAc,GAAG;IAC3B,KAAK,EAAE,MAAM,CAAA;IACb,KAAK,EAAE,MAAM,CAAA;CACd,CAAA;AAED,qBAAa,iBAAkB,SAAQ,aAAa,CAClD,SAAS;IACP,MAAM,EAAE,MAAM;IACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC;IAC5B,OAAO,EAAE,gBAAgB;CAC1B,EACD,sBAAsB,EACtB,cAAc,EACd,gBAAgB,EAChB,sBAAsB,EACtB,OAAO,mBAAmB,CAC3B;IACC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAGrB;IAEK,OAAO,CACZ,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,sBAAsB;IASZ,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACR,EAAE,sBAAsB;;;;IAuClB,OAAO,CAAC,OAAO,EAAE,cAAc;IAI/B,QAAQ,CAAC,MAAM,EAAE,cAAc,GAAG,gBAAgB;CAG1D;AAED,wBAAsB,KAAK,CACzB,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,OAAO,CAAC,gBAAgB,CAAC,CAS3B;AAED,yBAAiB,KAAK,CAAC;IACd,MAAM,IAAI,GAAI,cAAY,KAAG,kBAAiC,CAAA;IAE9D,MAAM,IAAI,GACf,OAAO,MAAM,EACb,OAAO,kBAAkB,CAAC,MAAM,CAAC,KAChC,kBAAuC,CAAA;CAC3C"}
|
|
@@ -19,10 +19,13 @@ export class FileSplitActivity extends AsyncActivity {
|
|
|
19
19
|
trim: options.trim,
|
|
20
20
|
skipEmpty: options.skipEmpty,
|
|
21
21
|
encoding,
|
|
22
|
+
inlineMaxBytes: options.inlineMaxBytes,
|
|
22
23
|
});
|
|
23
24
|
const source = await binary.stream();
|
|
24
25
|
const durable = await this.claims.fs.write(`split/${this.id}`, source.pipe(scanner));
|
|
25
|
-
const slices = await Promise.all(scanner.
|
|
26
|
+
const slices = await Promise.all(scanner.slices.map(({ start, end, content }) => content
|
|
27
|
+
? content.toString(encoding)
|
|
28
|
+
: this.claims.fs.slice(durable, { start, end }, encoding)));
|
|
26
29
|
const batch = await this.claims.workflow.startMany(workflow, slices);
|
|
27
30
|
if (batch.count === 0)
|
|
28
31
|
return this.done({ batch: batch.id, count: 0 });
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"FileSplitActivity.js","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,EAEb,KAAK,EACL,OAAO,EAEP,gBAAgB,EAChB,aAAa,EACb,mBAAmB,GACpB,MAAM,qBAAqB,CAAA;AAE5B,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACvD,OAAO,EACL,oBAAoB,GAGrB,MAAM,sCAAsC,CAAA;
|
|
1
|
+
{"version":3,"file":"FileSplitActivity.js","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,EAEb,KAAK,EACL,OAAO,EAEP,gBAAgB,EAChB,aAAa,EACb,mBAAmB,GACpB,MAAM,qBAAqB,CAAA;AAE5B,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACvD,OAAO,EACL,oBAAoB,GAGrB,MAAM,sCAAsC,CAAA;AAuB7C,MAAM,OAAO,iBAAkB,SAAQ,aAWtC;IACkB,MAAM,GAAG,IAAI,KAAK,CAAC;QAClC,EAAE,EAAE,eAAe;QACnB,QAAQ,EAAE,aAAa;KACxB,CAAC,CAAA;IAEK,OAAO,CACZ,MAAc,EACd,QAA4B,EAC5B,OAAyB;QAEzB,IAAI,CAAC,QAAQ,CAAC,EAAE;YACd,MAAM,IAAI,SAAS,CACjB,sDAAsD,CACvD,CAAA;QAEH,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,CAAC,EAAE,EAAE,OAAO,EAAE,CAAA;IACnD,CAAC;IAEM,KAAK,CAAC,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACgB;QACvB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,MAAM,CAAA;QAE3C,MAAM,OAAO,GAAG,IAAI,oBAAoB,CAAC;YACvC,SAAS,EAAE,OAAO,CAAC,SAAS;YAC5B,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,IAAI,EAAE,OAAO,CAAC,IAAI;YAClB,SAAS,EAAE,OAAO,CAAC,SAAS;YAC5B,QAAQ;YACR,cAAc,EAAE,OAAO,CAAC,cAAc;SACvC,CAAC,CAAA;QAEF,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,MAAM,EAAE,CAAA;QAEpC,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,KAAK,CACxC,SAAS,IAAI,CAAC,EAAE,EAAE,EAClB,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CACrB,CAAA;QAED,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,GAAG,CAC9B,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE,EAAE,CAC7C,OAAO;YACL,CAAC,CAAC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC;YAC5B,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,KAAK,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,GAAG,EAAE,EAAE,QAAQ,CAAC,CAC5D,CACF,CAAA;QAED,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,SAAS,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAA;QAEpE,IAAI,KAAK,CAAC,KAAK,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,KAAK,CAAC,EAAE,EAAE,KAAK,EAAE,CAAC,EAAE,CAAC,CAAA;QAEtE,OAAO,IAAI,CAAC,OAAO,CAAC;YAClB,MAAM,EAAE,mBAAmB;YAC3B,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI;gBACzB,SAAS,EAAE,IAAI,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,CAAC,WAAW,CAAC;aACtD,CAAC;SACH,CAAC,CAAA;IACJ,CAAC;IAEM,OAAO,CAAC,OAAuB;QACpC,OAAO,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,CAAA;IAC3B,CAAC;IAEM,QAAQ,CAAC,MAAsB;QACpC,OAAO,IAAI,gBAAgB,CAAC,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,KAAK,CAAC,CAAA;IACzD,CAAC;CACF;AAED,MAAM,CAAC,KAAK,UAAU,KAAK,CACzB,MAAc,EACd,QAA4B,EAC5B,OAAyB;IAEzB,MAAM,SAAS,GAAG,MAAM,OAAO,CAAC,UAAU,CACxC,iBAAiB,EACjB,MAAM,EACN,QAAQ,EACR,OAAO,CACR,CAAA;IAED,OAAO,MAAM,SAAS,CAAC,IAAI,EAAE,CAAA;AAC/B,CAAC;AAED,MAAM,KAAW,KAAK,CAOrB;AAPD,WAAiB,KAAK;IACP,UAAI,GAAG,CAAC,KAAK,GAAG,IAAI,EAAsB,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAA;IAExD,UAAI,GAAG,CAClB,KAAa,EACb,IAAiC,EACb,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAA;AAC5C,CAAC,EAPgB,KAAK,KAAL,KAAK,QAOrB"}
|
|
@@ -3,6 +3,9 @@ export type ByteRange = {
|
|
|
3
3
|
start: number;
|
|
4
4
|
end: number;
|
|
5
5
|
};
|
|
6
|
+
export type ByteSlice = ByteRange & {
|
|
7
|
+
content?: Buffer;
|
|
8
|
+
};
|
|
6
9
|
export type FileSplitDelimiter = {
|
|
7
10
|
value: string;
|
|
8
11
|
keep?: {
|
|
@@ -20,9 +23,11 @@ export type ByteDelimiterScannerOptions = {
|
|
|
20
23
|
trim?: ByteTrim;
|
|
21
24
|
skipEmpty?: boolean;
|
|
22
25
|
encoding?: BufferEncoding;
|
|
26
|
+
inlineMaxBytes?: number;
|
|
23
27
|
};
|
|
28
|
+
export declare const DEFAULT_INLINE_MAX_BYTES: number;
|
|
24
29
|
export declare class ByteDelimiterScanner extends Transform {
|
|
25
|
-
readonly
|
|
30
|
+
readonly slices: ByteSlice[];
|
|
26
31
|
private readonly delimiter;
|
|
27
32
|
private readonly keepPrevious;
|
|
28
33
|
private readonly keepNext;
|
|
@@ -30,11 +35,13 @@ export declare class ByteDelimiterScanner extends Transform {
|
|
|
30
35
|
private readonly trimStart;
|
|
31
36
|
private readonly trimEnd;
|
|
32
37
|
private readonly skipEmpty;
|
|
38
|
+
private readonly inlineMaxBytes;
|
|
33
39
|
private pending;
|
|
34
40
|
private pendingAt;
|
|
41
|
+
private searchAt;
|
|
35
42
|
private sliceStart;
|
|
36
43
|
private matches;
|
|
37
|
-
constructor({ delimiter, every, trim, skipEmpty, encoding, }: ByteDelimiterScannerOptions);
|
|
44
|
+
constructor({ delimiter, every, trim, skipEmpty, encoding, inlineMaxBytes, }: ByteDelimiterScannerOptions);
|
|
38
45
|
_transform(chunk: Buffer, _encoding: BufferEncoding, callback: TransformCallback): void;
|
|
39
46
|
_flush(callback: TransformCallback): void;
|
|
40
47
|
private scan;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"byte-delimiter-scanner.d.ts","sourceRoot":"","sources":["../../src/helpers/byte-delimiter-scanner.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,KAAK,iBAAiB,EAAE,MAAM,aAAa,CAAA;AAE/D,MAAM,MAAM,SAAS,GAAG;IACtB,KAAK,EAAE,MAAM,CAAA;IACb,GAAG,EAAE,MAAM,CAAA;CACZ,CAAA;AAED,MAAM,MAAM,kBAAkB,GAAG;IAC/B,KAAK,EAAE,MAAM,CAAA;IACb,IAAI,CAAC,EAAE;QACL,QAAQ,CAAC,EAAE,MAAM,CAAA;QACjB,IAAI,CAAC,EAAE,MAAM,CAAA;KACd,CAAA;CACF,CAAA;AAED,MAAM,MAAM,QAAQ,GAAG;IACrB,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,GAAG,CAAC,EAAE,MAAM,CAAA;CACb,CAAA;AAED,MAAM,MAAM,2BAA2B,GAAG;IACxC,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;
|
|
1
|
+
{"version":3,"file":"byte-delimiter-scanner.d.ts","sourceRoot":"","sources":["../../src/helpers/byte-delimiter-scanner.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,KAAK,iBAAiB,EAAE,MAAM,aAAa,CAAA;AAE/D,MAAM,MAAM,SAAS,GAAG;IACtB,KAAK,EAAE,MAAM,CAAA;IACb,GAAG,EAAE,MAAM,CAAA;CACZ,CAAA;AAED,MAAM,MAAM,SAAS,GAAG,SAAS,GAAG;IAClC,OAAO,CAAC,EAAE,MAAM,CAAA;CACjB,CAAA;AAED,MAAM,MAAM,kBAAkB,GAAG;IAC/B,KAAK,EAAE,MAAM,CAAA;IACb,IAAI,CAAC,EAAE;QACL,QAAQ,CAAC,EAAE,MAAM,CAAA;QACjB,IAAI,CAAC,EAAE,MAAM,CAAA;KACd,CAAA;CACF,CAAA;AAED,MAAM,MAAM,QAAQ,GAAG;IACrB,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,GAAG,CAAC,EAAE,MAAM,CAAA;CACb,CAAA;AAED,MAAM,MAAM,2BAA2B,GAAG;IACxC,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,cAAc,CAAC,EAAE,MAAM,CAAA;CACxB,CAAA;AAED,eAAO,MAAM,wBAAwB,QAAY,CAAA;AAEjD,qBAAa,oBAAqB,SAAQ,SAAS;IACjD,SAAgB,MAAM,EAAE,SAAS,EAAE,CAAK;IAExC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAQ;IAClC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAQ;IACrC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAQ;IACjC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAQ;IACjC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAQ;IAClC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAQ;IAChC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAQ;IAEvC,OAAO,CAAC,OAAO,CAA0B;IACzC,OAAO,CAAC,SAAS,CAAI;IACrB,OAAO,CAAC,QAAQ,CAAI;IACpB,OAAO,CAAC,UAAU,CAAI;IACtB,OAAO,CAAC,OAAO,CAAI;gBAEP,EACV,SAAS,EACT,KAAS,EACT,IAAI,EACJ,SAAgB,EAChB,QAAiB,EACjB,cAAyC,GAC1C,EAAE,2BAA2B;IAuCvB,UAAU,CACf,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,cAAc,EACzB,QAAQ,EAAE,iBAAiB;IAStB,MAAM,CAAC,QAAQ,EAAE,iBAAiB;IAyBzC,OAAO,CAAC,IAAI;IAyCZ,OAAO,CAAC,GAAG;CAkBZ"}
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { Transform } from 'node:stream';
|
|
2
|
+
export const DEFAULT_INLINE_MAX_BYTES = 16 * 1024;
|
|
2
3
|
export class ByteDelimiterScanner extends Transform {
|
|
3
|
-
|
|
4
|
+
slices = [];
|
|
4
5
|
delimiter;
|
|
5
6
|
keepPrevious;
|
|
6
7
|
keepNext;
|
|
@@ -8,11 +9,13 @@ export class ByteDelimiterScanner extends Transform {
|
|
|
8
9
|
trimStart;
|
|
9
10
|
trimEnd;
|
|
10
11
|
skipEmpty;
|
|
12
|
+
inlineMaxBytes;
|
|
11
13
|
pending = Buffer.alloc(0);
|
|
12
14
|
pendingAt = 0;
|
|
15
|
+
searchAt = 0;
|
|
13
16
|
sliceStart = 0;
|
|
14
17
|
matches = 0;
|
|
15
|
-
constructor({ delimiter, every = 1, trim, skipEmpty = true, encoding = 'utf8', }) {
|
|
18
|
+
constructor({ delimiter, every = 1, trim, skipEmpty = true, encoding = 'utf8', inlineMaxBytes = DEFAULT_INLINE_MAX_BYTES, }) {
|
|
16
19
|
super();
|
|
17
20
|
this.delimiter = Buffer.from(delimiter.value, encoding);
|
|
18
21
|
this.keepPrevious = delimiter.keep?.previous ?? 0;
|
|
@@ -21,6 +24,7 @@ export class ByteDelimiterScanner extends Transform {
|
|
|
21
24
|
this.trimStart = trim?.start ?? 0;
|
|
22
25
|
this.trimEnd = trim?.end ?? 0;
|
|
23
26
|
this.skipEmpty = skipEmpty;
|
|
27
|
+
this.inlineMaxBytes = inlineMaxBytes;
|
|
24
28
|
this.sliceStart = this.trimStart;
|
|
25
29
|
if (this.delimiter.length === 0)
|
|
26
30
|
throw new TypeError('Split delimiter must not be empty');
|
|
@@ -35,6 +39,8 @@ export class ByteDelimiterScanner extends Transform {
|
|
|
35
39
|
this.trimStart < 0 ||
|
|
36
40
|
this.trimEnd < 0)
|
|
37
41
|
throw new TypeError('Split trim sizes must be non-negative integers');
|
|
42
|
+
if (!Number.isInteger(inlineMaxBytes) || inlineMaxBytes < 0)
|
|
43
|
+
throw new TypeError('Split inlineMaxBytes must be a non-negative integer');
|
|
38
44
|
}
|
|
39
45
|
_transform(chunk, _encoding, callback) {
|
|
40
46
|
this.scan(this.pending.length > 0 ? Buffer.concat([this.pending, chunk]) : chunk);
|
|
@@ -43,21 +49,23 @@ export class ByteDelimiterScanner extends Transform {
|
|
|
43
49
|
_flush(callback) {
|
|
44
50
|
const limit = this.pendingAt + this.pending.length - this.trimEnd;
|
|
45
51
|
while (true) {
|
|
46
|
-
const last = this.
|
|
52
|
+
const last = this.slices.at(-1);
|
|
47
53
|
if (!last || last.end <= limit)
|
|
48
54
|
break;
|
|
49
55
|
if (last.start >= limit) {
|
|
50
|
-
this.
|
|
56
|
+
this.slices.pop();
|
|
51
57
|
continue;
|
|
52
58
|
}
|
|
53
59
|
last.end = limit;
|
|
60
|
+
if (last.content)
|
|
61
|
+
last.content = last.content.subarray(0, limit - last.start);
|
|
54
62
|
}
|
|
55
63
|
if (this.sliceStart <= limit)
|
|
56
|
-
this.cut(this.sliceStart, limit);
|
|
64
|
+
this.cut(this.sliceStart, limit, this.pending);
|
|
57
65
|
callback();
|
|
58
66
|
}
|
|
59
67
|
scan(buffer) {
|
|
60
|
-
let searchFrom = 0;
|
|
68
|
+
let searchFrom = Math.max(0, this.searchAt - this.pendingAt);
|
|
61
69
|
while (true) {
|
|
62
70
|
const index = buffer.indexOf(this.delimiter, searchFrom);
|
|
63
71
|
if (index === -1)
|
|
@@ -69,17 +77,27 @@ export class ByteDelimiterScanner extends Transform {
|
|
|
69
77
|
this.matches += 1;
|
|
70
78
|
if (this.matches % this.cutEvery !== 0)
|
|
71
79
|
continue;
|
|
72
|
-
this.cut(this.sliceStart, at + this.keepPrevious);
|
|
80
|
+
this.cut(this.sliceStart, at + this.keepPrevious, buffer);
|
|
73
81
|
this.sliceStart = at + this.delimiter.length - this.keepNext;
|
|
74
82
|
}
|
|
75
83
|
const tail = Math.max(searchFrom, buffer.length - (this.delimiter.length - 1));
|
|
76
|
-
this.
|
|
77
|
-
this.
|
|
84
|
+
this.searchAt = this.pendingAt + tail;
|
|
85
|
+
const openAt = this.sliceStart - this.pendingAt;
|
|
86
|
+
const isOpenSliceInline = openAt >= 0 && buffer.length - openAt <= this.inlineMaxBytes;
|
|
87
|
+
const keepFrom = isOpenSliceInline ? Math.min(tail, openAt) : tail;
|
|
88
|
+
this.pending = buffer.subarray(keepFrom);
|
|
89
|
+
this.pendingAt += keepFrom;
|
|
78
90
|
}
|
|
79
|
-
cut(start, end) {
|
|
91
|
+
cut(start, end, buffer) {
|
|
80
92
|
if (this.skipEmpty && end <= start)
|
|
81
93
|
return;
|
|
82
|
-
this.
|
|
94
|
+
const isInline = start >= this.pendingAt && end - start <= this.inlineMaxBytes;
|
|
95
|
+
if (!isInline) {
|
|
96
|
+
this.slices.push({ start, end });
|
|
97
|
+
return;
|
|
98
|
+
}
|
|
99
|
+
const content = Buffer.from(buffer.subarray(start - this.pendingAt, end - this.pendingAt));
|
|
100
|
+
this.slices.push({ start, end, content });
|
|
83
101
|
}
|
|
84
102
|
}
|
|
85
103
|
//# sourceMappingURL=byte-delimiter-scanner.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"byte-delimiter-scanner.js","sourceRoot":"","sources":["../../src/helpers/byte-delimiter-scanner.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAA0B,MAAM,aAAa,CAAA;
|
|
1
|
+
{"version":3,"file":"byte-delimiter-scanner.js","sourceRoot":"","sources":["../../src/helpers/byte-delimiter-scanner.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAA0B,MAAM,aAAa,CAAA;AAiC/D,MAAM,CAAC,MAAM,wBAAwB,GAAG,EAAE,GAAG,IAAI,CAAA;AAEjD,MAAM,OAAO,oBAAqB,SAAQ,SAAS;IACjC,MAAM,GAAgB,EAAE,CAAA;IAEvB,SAAS,CAAQ;IACjB,YAAY,CAAQ;IACpB,QAAQ,CAAQ;IAChB,QAAQ,CAAQ;IAChB,SAAS,CAAQ;IACjB,OAAO,CAAQ;IACf,SAAS,CAAS;IAClB,cAAc,CAAQ;IAE/B,OAAO,GAAW,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;IACjC,SAAS,GAAG,CAAC,CAAA;IACb,QAAQ,GAAG,CAAC,CAAA;IACZ,UAAU,GAAG,CAAC,CAAA;IACd,OAAO,GAAG,CAAC,CAAA;IAEnB,YAAY,EACV,SAAS,EACT,KAAK,GAAG,CAAC,EACT,IAAI,EACJ,SAAS,GAAG,IAAI,EAChB,QAAQ,GAAG,MAAM,EACjB,cAAc,GAAG,wBAAwB,GACb;QAC5B,KAAK,EAAE,CAAA;QAEP,IAAI,CAAC,SAAS,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAA;QACvD,IAAI,CAAC,YAAY,GAAG,SAAS,CAAC,IAAI,EAAE,QAAQ,IAAI,CAAC,CAAA;QACjD,IAAI,CAAC,QAAQ,GAAG,SAAS,CAAC,IAAI,EAAE,IAAI,IAAI,CAAC,CAAA;QACzC,IAAI,CAAC,QAAQ,GAAG,KAAK,CAAA;QACrB,IAAI,CAAC,SAAS,GAAG,IAAI,EAAE,KAAK,IAAI,CAAC,CAAA;QACjC,IAAI,CAAC,OAAO,GAAG,IAAI,EAAE,GAAG,IAAI,CAAC,CAAA;QAC7B,IAAI,CAAC,SAAS,GAAG,SAAS,CAAA;QAC1B,IAAI,CAAC,cAAc,GAAG,cAAc,CAAA;QACpC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,CAAA;QAEhC,IAAI,IAAI,CAAC,SAAS,CAAC,MAAM,KAAK,CAAC;YAC7B,MAAM,IAAI,SAAS,CAAC,mCAAmC,CAAC,CAAA;QAE1D,IAAI,IAAI,CAAC,YAAY,GAAG,CAAC,IAAI,IAAI,CAAC,QAAQ,GAAG,CAAC;YAC5C,MAAM,IAAI,SAAS,CAAC,iDAAiD,CAAC,CAAA;QAExE,IAAI,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM;YAC3D,MAAM,IAAI,SAAS,CACjB,iEAAiE,CAClE,CAAA;QAEH,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,GAAG,CAAC;YACvC,MAAM,IAAI,SAAS,CAAC,wCAAwC,CAAC,CAAA;QAE/D,IACE,CAAC,MAAM,CAAC,SAAS,CAAC,IAAI,CAAC,SAAS,CAAC;YACjC,CAAC,MAAM,CAAC,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC;YAC/B,IAAI,CAAC,SAAS,GAAG,CAAC;YAClB,IAAI,CAAC,OAAO,GAAG,CAAC;YAEhB,MAAM,IAAI,SAAS,CAAC,gDAAgD,CAAC,CAAA;QAEvE,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,cAAc,CAAC,IAAI,cAAc,GAAG,CAAC;YACzD,MAAM,IAAI,SAAS,CAAC,qDAAqD,CAAC,CAAA;IAC9E,CAAC;IAEM,UAAU,CACf,KAAa,EACb,SAAyB,EACzB,QAA2B;QAE3B,IAAI,CAAC,IAAI,CACP,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CACvE,CAAA;QAED,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACvB,CAAC;IAEM,MAAM,CAAC,QAA2B;QACvC,MAAM,KAAK,GAAG,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,OAAO,CAAA;QAEjE,OAAO,IAAI,EAAE,CAAC;YACZ,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAA;YAE/B,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,GAAG,IAAI,KAAK;gBAAE,MAAK;YAErC,IAAI,IAAI,CAAC,KAAK,IAAI,KAAK,EAAE,CAAC;gBACxB,IAAI,CAAC,MAAM,CAAC,GAAG,EAAE,CAAA;gBAEjB,SAAQ;YACV,CAAC;YAED,IAAI,CAAC,GAAG,GAAG,KAAK,CAAA;YAEhB,IAAI,IAAI,CAAC,OAAO;gBACd,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAA;QAC/D,CAAC;QAED,IAAI,IAAI,CAAC,UAAU,IAAI,KAAK;YAAE,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,UAAU,EAAE,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,CAAA;QAE5E,QAAQ,EAAE,CAAA;IACZ,CAAC;IAEO,IAAI,CAAC,MAAc;QACzB,IAAI,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,CAAA;QAE5D,OAAO,IAAI,EAAE,CAAC;YACZ,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,SAAS,EAAE,UAAU,CAAC,CAAA;YAExD,IAAI,KAAK,KAAK,CAAC,CAAC;gBAAE,MAAK;YAEvB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,GAAG,KAAK,CAAA;YAEjC,UAAU,GAAG,KAAK,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAA;YAE1C,IAAI,EAAE,GAAG,IAAI,CAAC,SAAS;gBAAE,SAAQ;YAEjC,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;YAEjB,IAAI,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,QAAQ,KAAK,CAAC;gBAAE,SAAQ;YAEhD,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,UAAU,EAAE,EAAE,GAAG,IAAI,CAAC,YAAY,EAAE,MAAM,CAAC,CAAA;YAEzD,IAAI,CAAC,UAAU,GAAG,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAA;QAC9D,CAAC;QAED,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CACnB,UAAU,EACV,MAAM,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAC5C,CAAA;QAED,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,GAAG,IAAI,CAAA;QAErC,MAAM,MAAM,GAAG,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,CAAA;QAE/C,MAAM,iBAAiB,GACrB,MAAM,IAAI,CAAC,IAAI,MAAM,CAAC,MAAM,GAAG,MAAM,IAAI,IAAI,CAAC,cAAc,CAAA;QAE9D,MAAM,QAAQ,GAAG,iBAAiB,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,CAAA;QAElE,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAA;QACxC,IAAI,CAAC,SAAS,IAAI,QAAQ,CAAA;IAC5B,CAAC;IAEO,GAAG,CAAC,KAAa,EAAE,GAAW,EAAE,MAAc;QACpD,IAAI,IAAI,CAAC,SAAS,IAAI,GAAG,IAAI,KAAK;YAAE,OAAM;QAE1C,MAAM,QAAQ,GACZ,KAAK,IAAI,IAAI,CAAC,SAAS,IAAI,GAAG,GAAG,KAAK,IAAI,IAAI,CAAC,cAAc,CAAA;QAE/D,IAAI,CAAC,QAAQ,EAAE,CAAC;YACd,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC,CAAA;YAEhC,OAAM;QACR,CAAC;QAED,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CACzB,MAAM,CAAC,QAAQ,CAAC,KAAK,GAAG,IAAI,CAAC,SAAS,EAAE,GAAG,GAAG,IAAI,CAAC,SAAS,CAAC,CAC9D,CAAA;QAED,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC,CAAA;IAC3C,CAAC;CACF"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@t4h.framework/fs",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.10.0",
|
|
4
4
|
"description": "File system module for the T4H Framework",
|
|
5
5
|
"homepage": "https://github.com/tech4humans-brasil/framework/tree/main/packages/fs",
|
|
6
6
|
"bugs": "https://github.com/tech4humans-brasil/framework/issues",
|
|
@@ -32,12 +32,12 @@
|
|
|
32
32
|
"test:watch": "vitest"
|
|
33
33
|
},
|
|
34
34
|
"devDependencies": {
|
|
35
|
-
"@t4h.framework/core": "^0.
|
|
35
|
+
"@t4h.framework/core": "^0.15.0",
|
|
36
36
|
"typescript": "^5.9.3",
|
|
37
37
|
"vitest": "^4.0.18"
|
|
38
38
|
},
|
|
39
39
|
"peerDependencies": {
|
|
40
|
-
"@t4h.framework/core": "^0.
|
|
40
|
+
"@t4h.framework/core": "^0.15.0"
|
|
41
41
|
},
|
|
42
42
|
"packageManager": "yarn@4.12.0",
|
|
43
43
|
"engines": {
|