@t4h.framework/fs 0.12.1 → 0.13.0

package/.ai/skills/framework-fs/SKILL.md

@@ -31,7 +31,6 @@ import {
   read, FileReadActivity, type FileReadOptions,
   // turn workflow content into a Binary — see the framework-fs-write skill
   write, FileWriteActivity, type FileWriteOptions,
-  ByteDelimiterScanner, type FileSplitDelimiter, type ByteRange, type ByteTrim,
 } from '@t4h.framework/fs'
 ```
 
@@ -66,7 +65,7 @@ export abstract class FileSystemClaim {
 |--------|---------|
 | `write(path, readable, contentType?)` | Persist stream or buffer at `path`; return a `Binary` handle (metadata + `stream()`). When `contentType` is omitted, infer it from the path. |
 | `read(path)` | Load an existing file at `path`; return a `Binary` handle. |
-| `slice(binary, range, encoding?)` | A view over a byte range of an existing binary, **without copying data**. When `encoding` is set, the runtime hydrates the slice as a decoded string instead of a binary — this is how `split` children receive plain strings. Implementations must **compose ranges**: slicing a slice is relative to it, and the stored range is absolute against the underlying file. |
+| `slice(binary, range, encoding?)` | A view over a byte range of an existing binary, **without copying data**. When `encoding` is set, the runtime hydrates the slice as a decoded string instead of a binary — this is how `read` returns a window as a string. Implementations must **compose ranges**: slicing a slice is relative to it, and the stored range is absolute against the underlying file. |
 
 All methods may return synchronously or as a `Promise`. Implementations choose storage layout; callers should only rely on the returned `Binary` API.
 
@@ -132,7 +131,7 @@ These packages depend on `@t4h.framework/fs` and declare `FileSystemClaim`; read
 |---------|--------|
 | `@t4h.framework/http` | The HTTP activity persists response bodies to `fs` and returns them wrapped in a `Body` — you read them with `res.body.json()` / `text()`. |
 | `@t4h.framework/jsx` | Render activities write their output to `fs` and hand back a `Binary` you consume in the workflow. |
-| `@t4h.framework/fs` itself | `FileSplitActivity` persists the durable copy and slices it by ranges; `FileJoinActivity` streams batch results into a new file; `FileReadActivity` slices (or streams, when the binary is not file-backed) a small window into a string. See the `framework-fs-split` skill. |
+| `@t4h.framework/fs` itself | `FileSplitActivity` streams the source and hands its slices to `startMany` as strings; `FileJoinActivity` streams batch results into a new file; `FileReadActivity` slices (or streams, when the binary is not file-backed) a small window into a string. See the `framework-fs-split` skill. |
 
 Orchestrate these package primitives from your workflow; the `fs` write/read happens inside them.
 

package/.ai/skills/framework-fs-split/SKILL.md

@@ -2,7 +2,7 @@
 name: framework-fs-split
 description: >-
   Guides correct use of `split`, `join` and `read` from @t4h.framework/fs:
-  fanning one large file out into a batch of workflows (one per byte-range
+  fanning one large file out into a batch of workflows (one per
   slice), folding the batch results back into a single file, and reading small
   byte windows (headers, trailers) into the workflow — with O(1) parent
   history and bounded memory regardless of file size. Use whenever a workflow
@@ -20,8 +20,7 @@ description: >-
 # `split` / `join` / `read` — file fan-out, fold and windows for workflows
 
 `split` turns **one file into a batch of workflows**: it streams the file once,
-cuts it into **byte-range slices** (never copying slice content), starts one
-child workflow per slice, and **suspends the parent until every child is
+cuts it into slices, starts one child workflow per slice, and **suspends the parent until every child is
 terminal**. `join` is the inverse: it folds the batch results back into a
 single file, streaming one result at a time.
 
@@ -62,13 +61,13 @@ Package path: `framework/packages/fs`. Import only from the package root.
 Everything here exists to keep the parent history **O(1)** whatever the file
 size:
 
-1. `split` reads the source stream **exactly once, worker side**: the same
-   pass persists a durable copy through `FileSystemClaim.write` while a byte
-   scanner records slice offsets. No slice content is materialized.
-2. Children receive **range views** over the durable copy
-   (`FileSystemClaim.slice`); the runtime hydrates each slice only inside that
-   child's own run — as a plain decoded string when the child's schema is
-   `Type.String()`.
+1. `split` reads the source once, worker side: it cuts records, drops the ones
+   `where` rejects, groups the rest by `every` and hands each slice to the
+   batch as a decoded string as soon as it is complete. Nothing collects the
+   slices, no copy of the file is written, and a slice past 3 MiB
+   fails the split while it reads.
+2. Each child receives its slice as a plain string — what its
+   `Type.String()` schema declares.
 3. The parent resumes with only `WorkflowBatchRef { id, count }`. Per-child
    results are **never aggregated onto the ref**: they live with each child
    run.
@@ -128,17 +127,16 @@ the next slice. Forgetting `{ next: 2 }` silently strips the record type code
 from every slice.
 
 Offsets are always computed over **raw bytes** (never decoded characters), so
-multi-byte encodings cannot skew the ranges a ranged download will use.
+multi-byte encodings cannot skew the cuts.
 
 ### Other options
 
 | Option | Default | Behavior |
 |--------|---------|----------|
-| `every` | `1` | Cut only every N delimiter occurrences — groups N records per child. One million lines with `every: 10_000` start 100 children, each receiving a raw block (inner delimiters untouched) to iterate locally. `keep` arithmetic applies only at actual cut points. |
+| `every` | `1` | Cut only every N delimiter occurrences — groups N records per child. One million lines with `every: 10_000` start 100 children, each receiving a raw block (inner delimiters untouched) to iterate locally. `keep` arithmetic applies only at actual cut points. A slice may take up to 3 MiB; past that the split fails while it reads — lower `every`. |
 | `trim` | none | `{ start?, end? }` **bytes** cut off each end of the file before scanning — a positional header/trailer. Byte-based on purpose: a record-start delimiter (`'\n10'`) fuses the trailer into the last record's slice, so a record-count skip would drop a valid record along with it. Trimmed content never becomes a slice nor counts toward `every`; fetch it with `read`. Positional layouts make the byte counts constants — getting them right is the caller's business. |
 | `skipEmpty` | `true` | Drops zero-length slices (consecutive delimiters, file starting/ending with one) so they never become child workflows. Turn off when slice index must align 1:1 with record position. "Empty" means zero bytes — a whitespace-only line is a regular slice. |
 | `encoding` | `'utf8'` | Converts the delimiter to bytes and decodes each slice into the string the child receives. |
-| `inlineMaxBytes` | `16_384` | A slice up to this many bytes is handed to the child as a string straight from the scan; a larger one as a reference to its byte range in the durable copy, read back where the child runs. Reading a few hundred bytes back costs a row lookup and a ranged object read per child, which is what a million-line split spends its CPU on. `0` hands every slice by range. Keep it well under the runtime's fan-out message budget (1 MiB by default). |
 | `timeoutInMs` | none | Max milliseconds to await the batch; on expiry the activity **rejects** (reason `timeout`) instead of suspending forever — the safety net for a child that never reaches a terminal state. |
 
 ---
@@ -248,8 +246,9 @@ const batch = await split(file, processInvoice, {
 
 ## Runtime requirements
 
-- Claim providers: `FileSystemClaim` (with `slice`), `WorkflowClaim` (with
-  `startMany`), and `WorkflowBatchClaim` (from `@t4h.framework/core`).
+- Claim providers: `FileSystemClaim` (with `slice`, for `read` and `join`),
+  `WorkflowClaim` (with `startMany` reading an `AsyncIterable`), and
+  `WorkflowBatchClaim` (from `@t4h.framework/core`).
 - Production whitelist: `FileSplitActivity`, `FileJoinActivity`,
   `FileReadActivity` and core's `StartManyWorkflowsActivity` must be
   registered via `runtime.addActivity`.
@@ -271,26 +270,22 @@ from `src/activities/__tests__/`):
 
 ```typescript
 import { AsyncActivityResultPending, Claim, WorkflowBatchClaim, WorkflowBatchRef, WorkflowClaim } from '@t4h.framework/core'
-import { FileSplitActivity, FileSystemClaim, Split } from '@t4h.framework/fs'
+import { FileSplitActivity, Split } from '@t4h.framework/fs'
 
-const write = vi.fn(async (_path, readable) => durableBinary) // must consume the stream
-const slice = vi.fn((binary, range, encoding) => ({ binary, range, encoding }))
-const startMany = vi.fn(async (_wf, inputs) => new WorkflowBatchRef('batch-1', inputs.length))
+const inputs: unknown[] = []
+const startMany = vi.fn(async (_wf, slices: AsyncIterable<unknown>) => {
+  for await (const slice of slices) inputs.push(slice) // must consume the stream
+  return new WorkflowBatchRef('batch-1', inputs.length)
+})
 
 const result = await Claim.run(
-  [
-    { provide: FileSystemClaim, value: { write, slice } as never },
-    { provide: WorkflowClaim, value: { startMany } as never },
-  ],
-  () => {
-    const activity = new FileSplitActivity()
-    activity.id = 'activity-1'
-    return activity.bootstrap({ binary, workflow: 'child', options: { delimiter: Split.line() } })
-  },
+  [{ provide: WorkflowClaim, value: { startMany } as never }],
+  () =>
+    new FileSplitActivity().bootstrap({ binary, workflow: 'child', options: { delimiter: Split.line() } }),
 )
 
 expect(result).toBeInstanceOf(AsyncActivityResultPending)
-expect(slice).toHaveBeenCalledWith(durableBinary, { start: 0, end: 3 }, 'utf8')
+expect(inputs).toEqual(['foo', 'bar', 'baz'])
 ```
 
 For `join`, provide `WorkflowBatchClaim` as an async generator:
@@ -301,9 +296,8 @@ const outputs = vi.fn(async function* () {
 })
 ```
 
-Behaviors worth asserting because they regress easily: the durable copy keeps
-the original bytes; slices are views over the durable copy (not the source);
-`keep` arithmetic on positional cuts; the empty batch resolving immediately;
+Behaviors worth asserting because they regress easily: the source is streamed
+once; `keep` arithmetic on positional cuts; the empty batch resolving immediately;
 CSV escaping (quotes doubled, delimiter/newline wrapped); the status filter.
 
 ---

package/CHANGELOG.md

@@ -1,5 +1,21 @@
 # @t4h.framework/fs
 
+## 0.13.0
+
+### Minor Changes
+
+- [#214](https://github.com/tech4humans-brasil/framework/pull/214) [`fc4db5f`](https://github.com/tech4humans-brasil/framework/commit/fc4db5fa01454a45a3727b76dbacc215aa6b3ab7) Thanks [@gusteycamargo](https://github.com/gusteycamargo)! - `split` reads the file once, as a stream: it cuts the records, drops the ones
+  `where` rejects, groups the rest by `every` and hands each slice to
+  `startMany` as a string as soon as it is complete. No durable copy of the file
+  is written and no slice is read back by range. A slice may take up to 3 MiB
+  (the same as the runtime's default limit for the input of a workflow
+  run); past that the split fails while it reads, naming the byte the slice
+  starts at, where `where` used to cap a record at 1 MiB and nothing capped a
+  slice. `inlineMaxBytes` is removed: every slice reaches its child as a
+  string. `ByteDelimiterScanner` is removed from the public API.
+
+## 0.12.2
+
 ## 0.12.1
 
 ## 0.12.0

package/README.md

@@ -145,7 +145,7 @@ which stream through storage.
 
 ### split
 
-Splits a file into byte-range slices, starts one child workflow per slice as
+Splits a file into slices, starts one child workflow per slice as
 a single batch, and suspends the parent until every child is terminal —
 regardless of each child succeeding or failing (a rejected child never
 rejects the batch):
@@ -170,14 +170,15 @@ const main = new Workflow(fileSchema, async (file: Binary) => {
 })
 ```
 
-The design goal is constant memory regardless of file size:
+The source is read once, as a stream:
 
-1. The source stream is read **exactly once**, worker side. The same pass
-   persists a durable copy through `FileSystemClaim.write` while a byte
-   scanner records the slice offsets — no slice content is ever materialized
-   in the activity.
-2. Children receive **range views** over the durable copy
-   (`FileSystemClaim.slice`).
+1. The source is read once, worker side: cut into records, filtered by
+   `where` when given, grouped by `every`, and each slice goes into the batch
+   as a string as soon as it is complete. Nothing collects the slices, and a
+   slice may take up to 3 MiB: past that the split fails
+   while it reads, at the byte the slice starts.
+2. The runtime reads that stream once and stores the batch as it goes; no copy
+   of the file is written and no slice is read back by range.
 3. The activity output is just the batch reference (`{ id, count }`), so the
    parent history stays O(1) whatever the slice count. Per-child results are
    never aggregated: they live with each child run, to be inspected
@@ -222,8 +223,7 @@ await split(file, workflow, {
   every: 10_000, // cut only every N occurrences (groups records)
   trim: { start: 241, end: 241 }, // bytes cut off each end before scanning
   skipEmpty: false, // keep zero-length slices (default: skipped)
-  encoding: 'latin1', // delimiter bytes + slice hydration (default: utf8)
-  inlineMaxBytes: 16_384, // slices up to this size travel as strings (default: 16 KiB)
+  encoding: 'latin1', // delimiter bytes + slice decoding (default: utf8)
   timeoutInMs: 3_600_000, // max ms to await the batch; rejects on expiry
 })
 ```
@@ -254,18 +254,9 @@ await split(file, workflow, {
   workflows. Turn it off when the slice index must align 1:1 with the record
   position in the source file. "Empty" means zero bytes: a whitespace-only
   line is a regular slice.
-- **`encoding`** converts the delimiter to bytes and tells the runtime how to
-  hydrate each slice for the child. Offsets are always computed over raw
-  bytes, so multi-byte encodings never skew the ranges.
-- **`inlineMaxBytes`** (default 16 KiB) is the size up to which a slice is
-  handed to the child as a string, straight from the scan. A slice above it is
-  handed as a reference to its byte range in the durable copy, read back
-  where the child runs. A line of a CSV is a few hundred bytes: reading it
-  back would cost a row lookup and a ranged object read per child, and that,
-  not the scan, is what a million-line split spends its CPU on. Set it to `0`
-  to hand every slice by range. Keep it well under the fan-out message budget
-  of the runtime (`WORKFLOW_BATCH_FANOUT_CHUNK_MAX_BYTES`, 1 MiB by default):
-  an inline slice that does not fit a message is refused.
+- **`encoding`** converts the delimiter to bytes and decodes each slice into
+  the string its child receives. Offsets are always computed over raw bytes,
+  so multi-byte encodings never skew the cuts.
 - **`timeoutInMs`** is the maximum time in milliseconds to await the batch. On
   expiry the activity rejects (reason `timeout`) instead of suspending
   forever — the safety net for a child that never reaches a terminal state.
@@ -374,21 +365,3 @@ already hydrated one at a time — the query compiles once and adds one
 predicate call per entry, no extra serialization and no change to the memory
 profile. It is applied app-side: every child output is still fetched and
 parsed; `where` decides what enters the file, not what leaves the database.
-
-### ByteDelimiterScanner
-
-The scanner behind `split` is exported for direct use. It is a pass-through
-`Transform`: chunks flow through unchanged (so the same pass can persist the
-file) while it records slice offsets in absolute byte positions, handling
-delimiters split across chunk boundaries. `ranges` is complete once the stream
-has flushed.
-
-```typescript
-import { ByteDelimiterScanner } from '@t4h.framework/fs'
-
-const scanner = new ByteDelimiterScanner({ delimiter: { value: '\n' } })
-
-await pipeline(source, scanner, destination)
-
-scanner.slices // [{ start: 0, end: 3, content: <Buffer> }, ...]
-```

package/dist/activities/FileSplitActivity.d.ts

@@ -1,5 +1,5 @@
 import { AsyncActivity, type Binary, type Workflow, WorkflowBatchCollectionRef, workflowBatchSchema } from '@t4h.framework/core';
-import { type ByteTrim, type FileSplitDelimiter } from '../helpers/ByteDelimiterScanner.js';
+import { type ByteTrim, type FileSplitDelimiter } from '../helpers/split-records.js';
 type FileSplitWhereOperators = {
     $eq?: string;
     $ne?: string;
@@ -25,7 +25,6 @@ export type FileSplitOptions = {
     trim?: ByteTrim;
     skipEmpty?: boolean;
     encoding?: BufferEncoding;
-    inlineMaxBytes?: number;
     timeoutInMs?: number;
 };
 export type FileSplitActivityInput = {
@@ -48,7 +47,8 @@ export declare class FileSplitActivity extends AsyncActivity<readonly [
         batch: import("typebox").TString;
         count: import("typebox").TNumber;
     }>> | import("@t4h.framework/core").AsyncActivityResultFulfilled<FileSplitBatch | undefined>>;
-    private toFilter;
+    private slices;
+    private filter;
     advance(payload: FileSplitBatch): import("@t4h.framework/core").AsyncActivityResultFulfilled<FileSplitBatch | undefined>;
     toOutput(output: FileSplitBatch): WorkflowBatchCollectionRef;
 }

package/dist/activities/FileSplitActivity.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"FileSplitActivity.d.ts","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAEA,OAAO,EACL,aAAa,EACb,KAAK,MAAM,EAGX,KAAK,QAAQ,EACb,0BAA0B,EAG1B,mBAAmB,EACpB,MAAM,qBAAqB,CAAA;AAM5B,OAAO,EAEL,KAAK,QAAQ,EACb,KAAK,kBAAkB,EACxB,MAAM,oCAAoC,CAAA;AAK3C,KAAK,uBAAuB,GAAG;IAC7B,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,GAAG,CAAC,EAAE,MAAM,EAAE,CAAA;IACd,IAAI,CAAC,EAAE,MAAM,EAAE,CAAA;IACf,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,IAAI,CAAC,EAAE,cAAc,CAAA;CACtB,CAAA;AAED,MAAM,MAAM,cAAc,GACtB,MAAM,GACN,CAAC,uBAAuB,GAAG;IACzB,IAAI,CAAC,EAAE,cAAc,EAAE,CAAA;IACvB,GAAG,CAAC,EAAE,cAAc,EAAE,CAAA;IACtB,IAAI,CAAC,EAAE,cAAc,EAAE,CAAA;CACxB,CAAC,CAAA;AAEN,MAAM,MAAM,gBAAgB,GAAG;IAC7B,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,KAAK,CAAC,EAAE,cAAc,CAAA;IACtB,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,cAAc,CAAC,EAAE,MAAM,CAAA;IACvB,WAAW,CAAC,EAAE,MAAM,CAAA;CACrB,CAAA;AAED,MAAM,MAAM,sBAAsB,GAAG;IACnC,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,MAAM,CAAA;IAChB,OAAO,EAAE,gBAAgB,CAAA;CAC1B,CAAA;AAED,MAAM,MAAM,cAAc,GAAG;IAC3B,KAAK,EAAE,MAAM,CAAA;IACb,KAAK,EAAE,MAAM,CAAA;CACd,CAAA;AAED,qBAAa,iBAAkB,SAAQ,aAAa,CAClD,SAAS;IACP,MAAM,EAAE,MAAM;IACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC;IAC5B,OAAO,EAAE,gBAAgB;CAC1B,EACD,sBAAsB,EACtB,cAAc,EACd,0BAA0B,EAC1B,sBAAsB,EACtB,OAAO,mBAAmB,CAC3B;IACC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAGrB;IAEK,OAAO,CACZ,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,sBAAsB;IAsBZ,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACR,EAAE,sBAAsB;;;;IA6CzB,OAAO,CAAC,QAAQ;IAkBT,OAAO,CAAC,OAAO,EAAE,cAAc;IAI/B,QAAQ,CAAC,MAAM,EAAE,cAAc,GAAG,0BAA0B;CAGpE;AAED,wBAAsB,KAAK,CACzB,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,OAAO,CAAC,0BAA0B,CAAC,CASrC;AAED,yBAAiB,KAAK,CAAC;IACd,MAAM,IAAI,GAAI,cAAY,KAAG,kBAAiC,CAAA;IAE9D,MAAM,IAAI,GACf,OAAO,MAAM,EACb,OAAO,kBAAkB,CAAC,MAAM,CAAC,KAChC,kBAAuC,CAAA;CAC3C"}
+{"version":3,"file":"FileSplitActivity.d.ts","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,EACb,KAAK,MAAM,EAGX,KAAK,QAAQ,EACb,0BAA0B,EAG1B,mBAAmB,EACpB,MAAM,qBAAqB,CAAA;AAM5B,OAAO,EAEL,KAAK,QAAQ,EACb,KAAK,kBAAkB,EAExB,MAAM,6BAA6B,CAAA;AAIpC,KAAK,uBAAuB,GAAG;IAC7B,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,GAAG,CAAC,EAAE,MAAM,CAAA;IACZ,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,GAAG,CAAC,EAAE,MAAM,EAAE,CAAA;IACd,IAAI,CAAC,EAAE,MAAM,EAAE,CAAA;IACf,MAAM,CAAC,EAAE,MAAM,CAAA;IACf,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,IAAI,CAAC,EAAE,cAAc,CAAA;CACtB,CAAA;AAED,MAAM,MAAM,cAAc,GACtB,MAAM,GACN,CAAC,uBAAuB,GAAG;IACzB,IAAI,CAAC,EAAE,cAAc,EAAE,CAAA;IACvB,GAAG,CAAC,EAAE,cAAc,EAAE,CAAA;IACtB,IAAI,CAAC,EAAE,cAAc,EAAE,CAAA;CACxB,CAAC,CAAA;AAEN,MAAM,MAAM,gBAAgB,GAAG;IAC7B,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,KAAK,CAAC,EAAE,cAAc,CAAA;IACtB,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,WAAW,CAAC,EAAE,MAAM,CAAA;CACrB,CAAA;AAED,MAAM,MAAM,sBAAsB,GAAG;IACnC,MAAM,EAAE,MAAM,CAAA;IACd,QAAQ,EAAE,MAAM,CAAA;IAChB,OAAO,EAAE,gBAAgB,CAAA;CAC1B,CAAA;AAED,MAAM,MAAM,cAAc,GAAG;IAC3B,KAAK,EAAE,MAAM,CAAA;IACb,KAAK,EAAE,MAAM,CAAA;CACd,CAAA;AAED,qBAAa,iBAAkB,SAAQ,aAAa,CAClD,SAAS;IACP,MAAM,EAAE,MAAM;IACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC;IAC5B,OAAO,EAAE,gBAAgB;CAC1B,EACD,sBAAsB,EACtB,cAAc,EACd,0BAA0B,EAC1B,sBAAsB,EACtB,OAAO,mBAAmB,CAC3B;IACC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAErB;IAEK,OAAO,CACZ,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,sBAAsB;IAsBZ,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACR,EAAE,sBAAsB;;;;YAgBV,MAAM;YA4BN,MAAM;IAad,OAAO,CAAC,OAAO,EAAE,cAAc;IAI/B,QAAQ,CAAC,MAAM,EAAE,cAAc,GAAG,0BAA0B;CAGpE;AAED,wBAAsB,KAAK,CACzB,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,QAAQ,CAAC,GAAG,EAAE,GAAG,CAAC,EAC5B,OAAO,EAAE,gBAAgB,GACxB,OAAO,CAAC,0BAA0B,CAAC,CASrC;AAED,yBAAiB,KAAK,CAAC;IACd,MAAM,IAAI,GAAI,cAAY,KAAG,kBAAiC,CAAA;IAE9D,MAAM,IAAI,GACf,OAAO,MAAM,EACb,OAAO,kBAAkB,CAAC,MAAM,CAAC,KAChC,kBAAuC,CAAA;CAC3C"}

package/dist/activities/FileSplitActivity.js

@@ -1,14 +1,11 @@
-import { pipeline } from 'node:stream';
 import { AsyncActivity, Claim, History, WorkflowBatchCollectionRef, WorkflowBatchQuery, WorkflowClaim, workflowBatchSchema, } from '@t4h.framework/core';
 import sift from 'sift';
 import { STRING_OPERATIONS } from '../constants/operations.js';
-import { FileSystemClaim } from '../FileSystemClaim.js';
-import { ByteDelimiterScanner, } from '../helpers/ByteDelimiterScanner.js';
-import { ByteRecordFilter } from '../helpers/ByteRecordFilter.js';
+import { groupRecords } from '../helpers/group-records.js';
+import { splitRecords, } from '../helpers/split-records.js';
 const { createQueryTester } = sift;
 export class FileSplitActivity extends AsyncActivity {
     claims = new Claim({
-        fs: FileSystemClaim,
         workflow: WorkflowClaim,
     });
     toInput(binary, workflow, options) {
@@ -23,25 +20,7 @@ export class FileSplitActivity extends AsyncActivity {
         return { binary, workflow: workflow.id, options };
     }
     async bootstrap({ binary, workflow, options, }) {
-        const encoding = options.encoding ?? 'utf8';
-        const filter = this.toFilter(options.where, options);
-        const scanner = new ByteDelimiterScanner({
-            delimiter: options.delimiter,
-            every: options.every,
-            trim: filter ? undefined : options.trim,
-            skipEmpty: options.skipEmpty,
-            encoding,
-            inlineMaxBytes: options.inlineMaxBytes,
-        });
-        const source = await binary.stream();
-        const durable = await this.claims.fs.write(`split/${this.id}`, filter
-            ? pipeline(source, filter, scanner, () => { })
-            : source.pipe(scanner));
-        const kept = filter && filter.accepted === 0 ? [] : scanner.slices;
-        const slices = await Promise.all(kept.map(({ start, end, content }) => content
-            ? content.toString(encoding)
-            : this.claims.fs.slice(durable, { start, end }, encoding)));
-        const batch = await this.claims.workflow.startMany(workflow, slices);
+        const batch = await this.claims.workflow.startMany(workflow, this.slices(binary, options));
         if (batch.count === 0)
             return this.done({ batch: batch.id, count: 0 });
         return this.pending({
@@ -51,18 +30,25 @@ export class FileSplitActivity extends AsyncActivity {
             }),
         });
     }
-    toFilter(where, { delimiter, trim, encoding = 'utf8' }) {
-        if (!where)
-            return;
-        const matches = createQueryTester(where, {
-            operations: STRING_OPERATIONS,
-        });
-        return new ByteRecordFilter({
+    async *slices(binary, { delimiter, every, where, trim, skipEmpty = true, ...options }) {
+        const encoding = options.encoding ?? 'utf8';
+        const records = splitRecords(await binary.stream(), {
             delimiter,
             trim,
             encoding,
-            accept: record => matches(record.toString(encoding)),
         });
+        const slices = groupRecords(where === undefined ? records : this.filter(records, where, encoding), { delimiter, every, encoding });
+        for await (const { content } of slices)
+            if (!skipEmpty || content.length > 0)
+                yield content.toString(encoding);
+    }
+    async *filter(records, where, encoding) {
+        const matches = createQueryTester(where, {
+            operations: STRING_OPERATIONS,
+        });
+        for await (const record of records)
+            if (matches(Buffer.concat(record.pieces).toString(encoding)))
+                yield record;
     }
     advance(payload) {
         return this.done(payload);

package/dist/activities/FileSplitActivity.js.map

@@ -1 +1 @@
-{"version":3,"file":"FileSplitActivity.js","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAA;AAEtC,OAAO,EACL,aAAa,EAEb,KAAK,EACL,OAAO,EAEP,0BAA0B,EAC1B,kBAAkB,EAClB,aAAa,EACb,mBAAmB,GACpB,MAAM,qBAAqB,CAAA;AAC5B,OAAO,IAAI,MAAM,MAAM,CAAA;AAGvB,OAAO,EAAE,iBAAiB,EAAE,MAAM,4BAA4B,CAAA;AAC9D,OAAO,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAA;AACvD,OAAO,EACL,oBAAoB,GAGrB,MAAM,oCAAoC,CAAA;AAC3C,OAAO,EAAE,gBAAgB,EAAE,MAAM,gCAAgC,CAAA;AAEjE,MAAM,EAAE,iBAAiB,EAAE,GAAG,IAAI,CAAA;AA8ClC,MAAM,OAAO,iBAAkB,SAAQ,aAWtC;IACkB,MAAM,GAAG,IAAI,KAAK,CAAC;QAClC,EAAE,EAAE,eAAe;QACnB,QAAQ,EAAE,aAAa;KACxB,CAAC,CAAA;IAEK,OAAO,CACZ,MAAc,EACd,QAA4B,EAC5B,OAAyB;QAEzB,IAAI,CAAC,QAAQ,CAAC,EAAE;YACd,MAAM,IAAI,SAAS,CACjB,sDAAsD,CACvD,CAAA;QAEH,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS,EAAE,CAAC;YAChC,kBAAkB,CAAC,kBAAkB,CACnC,OAAO,CAAC,KAAK,EACb,OAAO,EACP,aAAa,EACb,iBAAiB,CAClB,CAAA;YAED,iBAAiB,CAAC,OAAO,CAAC,KAAsB,EAAE;gBAChD,UAAU,EAAE,iBAAiB;aAC9B,CAAC,CAAA;QACJ,CAAC;QAED,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,CAAC,EAAE,EAAE,OAAO,EAAE,CAAA;IACnD,CAAC;IAEM,KAAK,CAAC,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACgB;QACvB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,MAAM,CAAA;QAE3C,MAAM,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAC,OAAO,CAAC,KAAK,EAAE,OAAO,CAAC,CAAA;QAEpD,MAAM,OAAO,GAAG,IAAI,oBAAoB,CAAC;YACvC,SAAS,EAAE,OAAO,CAAC,SAAS;YAC5B,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,IAAI;YACvC,SAAS,EAAE,OAAO,CAAC,SAAS;YAC5B,QAAQ;YACR,cAAc,EAAE,OAAO,CAAC,cAAc;SACvC,CAAC,CAAA;QAEF,MAAM,MAAM,GAAG,MAAM,MAAM,CAAC,MAAM,EAAE,CAAA;QAEpC,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,KAAK,CACxC,SAAS,IAAI,CAAC,EAAE,EAAE,EAClB,MAAM;YACJ,CAAC,CAAC,QAAQ,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,GAAG,EAAE,GAAE,CAAC,CAAC;YAC7C,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CACzB,CAAA;QAED,MAAM,IAAI,GAAG,MAAM,IAAI,MAAM,CAAC,QAAQ,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,MAAM,CAAA;QAElE,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,GAAG,CAC9B,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE,EAAE,CACnC,OAAO;YACL,CAAC,CAAC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC;YAC5B,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,KAAK,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,GAAG,EAAE,EAAE,QAAQ,CAAC,CAC5D,CACF,CAAA;QAED,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,SAAS,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAA;QAEpE,IAAI,KAAK,CAAC,KAAK,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,KAAK,CAAC,EAAE,EAAE,KAAK,EAAE,CAAC,EAAE,CAAC,CAAA;QAEtE,OAAO,IAAI,CAAC,OAAO,CAAC;YAClB,MAAM,EAAE,mBAAmB;YAC3B,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI;gBACzB,SAAS,EAAE,IAAI,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,CAAC,WAAW,CAAC;aACtD,CAAC;SACH,CAAC,CAAA;IACJ,CAAC;IAEO,QAAQ,CACd,KAAiC,EACjC,EAAE,SAAS,EAAE,IAAI,EAAE,QAAQ,GAAG,MAAM,EAAoB;QAExD,IAAI,CAAC,KAAK;YAAE,OAAM;QAElB,MAAM,OAAO,GAAG,iBAAiB,CAAC,KAAsB,EAAE;YACxD,UAAU,EAAE,iBAAiB;SAC9B,CAAC,CAAA;QAEF,OAAO,IAAI,gBAAgB,CAAC;YAC1B,SAAS;YACT,IAAI;YACJ,QAAQ;YACR,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC,OAAO,CAAC,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;SACrD,CAAC,CAAA;IACJ,CAAC;IAEM,OAAO,CAAC,OAAuB;QACpC,OAAO,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,CAAA;IAC3B,CAAC;IAEM,QAAQ,CAAC,MAAsB;QACpC,OAAO,IAAI,0BAA0B,CAAC,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,KAAK,CAAC,CAAA;IACnE,CAAC;CACF;AAED,MAAM,CAAC,KAAK,UAAU,KAAK,CACzB,MAAc,EACd,QAA4B,EAC5B,OAAyB;IAEzB,MAAM,SAAS,GAAG,MAAM,OAAO,CAAC,UAAU,CACxC,iBAAiB,EACjB,MAAM,EACN,QAAQ,EACR,OAAO,CACR,CAAA;IAED,OAAO,MAAM,SAAS,CAAC,IAAI,EAAE,CAAA;AAC/B,CAAC;AAED,MAAM,KAAW,KAAK,CAOrB;AAPD,WAAiB,KAAK;IACP,UAAI,GAAG,CAAC,KAAK,GAAG,IAAI,EAAsB,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAA;IAExD,UAAI,GAAG,CAClB,KAAa,EACb,IAAiC,EACb,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAA;AAC5C,CAAC,EAPgB,KAAK,KAAL,KAAK,QAOrB"}
+{"version":3,"file":"FileSplitActivity.js","sourceRoot":"","sources":["../../src/activities/FileSplitActivity.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,EAEb,KAAK,EACL,OAAO,EAEP,0BAA0B,EAC1B,kBAAkB,EAClB,aAAa,EACb,mBAAmB,GACpB,MAAM,qBAAqB,CAAA;AAC5B,OAAO,IAAI,MAAM,MAAM,CAAA;AAGvB,OAAO,EAAE,iBAAiB,EAAE,MAAM,4BAA4B,CAAA;AAC9D,OAAO,EAAE,YAAY,EAAE,MAAM,6BAA6B,CAAA;AAC1D,OAAO,EAIL,YAAY,GACb,MAAM,6BAA6B,CAAA;AAEpC,MAAM,EAAE,iBAAiB,EAAE,GAAG,IAAI,CAAA;AA6ClC,MAAM,OAAO,iBAAkB,SAAQ,aAWtC;IACkB,MAAM,GAAG,IAAI,KAAK,CAAC;QAClC,QAAQ,EAAE,aAAa;KACxB,CAAC,CAAA;IAEK,OAAO,CACZ,MAAc,EACd,QAA4B,EAC5B,OAAyB;QAEzB,IAAI,CAAC,QAAQ,CAAC,EAAE;YACd,MAAM,IAAI,SAAS,CACjB,sDAAsD,CACvD,CAAA;QAEH,IAAI,OAAO,CAAC,KAAK,KAAK,SAAS,EAAE,CAAC;YAChC,kBAAkB,CAAC,kBAAkB,CACnC,OAAO,CAAC,KAAK,EACb,OAAO,EACP,aAAa,EACb,iBAAiB,CAClB,CAAA;YAED,iBAAiB,CAAC,OAAO,CAAC,KAAsB,EAAE;gBAChD,UAAU,EAAE,iBAAiB;aAC9B,CAAC,CAAA;QACJ,CAAC;QAED,OAAO,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,CAAC,EAAE,EAAE,OAAO,EAAE,CAAA;IACnD,CAAC;IAEM,KAAK,CAAC,SAAS,CAAC,EACrB,MAAM,EACN,QAAQ,EACR,OAAO,GACgB;QACvB,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,SAAS,CAChD,QAAQ,EACR,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAC7B,CAAA;QAED,IAAI,KAAK,CAAC,KAAK,KAAK,CAAC;YAAE,OAAO,IAAI,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,KAAK,CAAC,EAAE,EAAE,KAAK,EAAE,CAAC,EAAE,CAAC,CAAA;QAEtE,OAAO,IAAI,CAAC,OAAO,CAAC;YAClB,MAAM,EAAE,mBAAmB;YAC3B,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI;gBACzB,SAAS,EAAE,IAAI,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,CAAC,WAAW,CAAC;aACtD,CAAC;SACH,CAAC,CAAA;IACJ,CAAC;IAEO,KAAK,CAAC,CAAC,MAAM,CACnB,MAAc,EACd,EACE,SAAS,EACT,KAAK,EACL,KAAK,EACL,IAAI,EACJ,SAAS,GAAG,IAAI,EAChB,GAAG,OAAO,EACO;QAEnB,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,MAAM,CAAA;QAE3C,MAAM,OAAO,GAAG,YAAY,CAAC,MAAM,MAAM,CAAC,MAAM,EAAE,EAAE;YAClD,SAAS;YACT,IAAI;YACJ,QAAQ;SACT,CAAC,CAAA;QAEF,MAAM,MAAM,GAAG,YAAY,CACzB,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,MAAM,CAAC,OAAO,EAAE,KAAK,EAAE,QAAQ,CAAC,EACrE,EAAE,SAAS,EAAE,KAAK,EAAE,QAAQ,EAAE,CAC/B,CAAA;QAED,IAAI,KAAK,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,MAAM;YACpC,IAAI,CAAC,SAAS,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC;gBAAE,MAAM,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAA;IAC1E,CAAC;IAEO,KAAK,CAAC,CAAC,MAAM,CACnB,OAAkC,EAClC,KAAqB,EACrB,QAAwB;QAExB,MAAM,OAAO,GAAG,iBAAiB,CAAC,KAAsB,EAAE;YACxD,UAAU,EAAE,iBAAiB;SAC9B,CAAC,CAAA;QAEF,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI,OAAO;YAChC,IAAI,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;gBAAE,MAAM,MAAM,CAAA;IAC9E,CAAC;IAEM,OAAO,CAAC,OAAuB;QACpC,OAAO,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,CAAA;IAC3B,CAAC;IAEM,QAAQ,CAAC,MAAsB;QACpC,OAAO,IAAI,0BAA0B,CAAC,MAAM,CAAC,KAAK,EAAE,MAAM,CAAC,KAAK,CAAC,CAAA;IACnE,CAAC;CACF;AAED,MAAM,CAAC,KAAK,UAAU,KAAK,CACzB,MAAc,EACd,QAA4B,EAC5B,OAAyB;IAEzB,MAAM,SAAS,GAAG,MAAM,OAAO,CAAC,UAAU,CACxC,iBAAiB,EACjB,MAAM,EACN,QAAQ,EACR,OAAO,CACR,CAAA;IAED,OAAO,MAAM,SAAS,CAAC,IAAI,EAAE,CAAA;AAC/B,CAAC;AAED,MAAM,KAAW,KAAK,CAOrB;AAPD,WAAiB,KAAK;IACP,UAAI,GAAG,CAAC,KAAK,GAAG,IAAI,EAAsB,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAA;IAExD,UAAI,GAAG,CAClB,KAAa,EACb,IAAiC,EACb,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAA;AAC5C,CAAC,EAPgB,KAAK,KAAL,KAAK,QAOrB"}

package/dist/fs.d.ts

@@ -4,5 +4,5 @@ export * from './activities/FileJoinActivity.js';
 export * from './activities/FileReadActivity.js';
 export * from './activities/FileSplitActivity.js';
 export * from './activities/FileWriteActivity.js';
-export * from './helpers/ByteDelimiterScanner.js';
+export type { ByteTrim, FileSplitDelimiter } from './helpers/split-records.js';
 //# sourceMappingURL=fs.d.ts.map

package/dist/fs.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"fs.d.ts","sourceRoot":"","sources":["../src/fs.ts"],"names":[],"mappings":"AAAA,cAAc,iBAAiB,CAAA;AAC/B,cAAc,sBAAsB,CAAA;AACpC,cAAc,kCAAkC,CAAA;AAChD,cAAc,kCAAkC,CAAA;AAChD,cAAc,mCAAmC,CAAA;AACjD,cAAc,mCAAmC,CAAA;AACjD,cAAc,mCAAmC,CAAA"}
+{"version":3,"file":"fs.d.ts","sourceRoot":"","sources":["../src/fs.ts"],"names":[],"mappings":"AAAA,cAAc,iBAAiB,CAAA;AAC/B,cAAc,sBAAsB,CAAA;AACpC,cAAc,kCAAkC,CAAA;AAChD,cAAc,kCAAkC,CAAA;AAChD,cAAc,mCAAmC,CAAA;AACjD,cAAc,mCAAmC,CAAA;AACjD,YAAY,EAAE,QAAQ,EAAE,kBAAkB,EAAE,MAAM,4BAA4B,CAAA"}

package/dist/fs.js

@@ -4,5 +4,4 @@ export * from './activities/FileJoinActivity.js';
 export * from './activities/FileReadActivity.js';
 export * from './activities/FileSplitActivity.js';
 export * from './activities/FileWriteActivity.js';
-export * from './helpers/ByteDelimiterScanner.js';
 //# sourceMappingURL=fs.js.map

package/dist/fs.js.map

@@ -1 +1 @@
-{"version":3,"file":"fs.js","sourceRoot":"","sources":["../src/fs.ts"],"names":[],"mappings":"AAAA,cAAc,iBAAiB,CAAA;AAC/B,cAAc,sBAAsB,CAAA;AACpC,cAAc,kCAAkC,CAAA;AAChD,cAAc,kCAAkC,CAAA;AAChD,cAAc,mCAAmC,CAAA;AACjD,cAAc,mCAAmC,CAAA;AACjD,cAAc,mCAAmC,CAAA"}
+{"version":3,"file":"fs.js","sourceRoot":"","sources":["../src/fs.ts"],"names":[],"mappings":"AAAA,cAAc,iBAAiB,CAAA;AAC/B,cAAc,sBAAsB,CAAA;AACpC,cAAc,kCAAkC,CAAA;AAChD,cAAc,kCAAkC,CAAA;AAChD,cAAc,mCAAmC,CAAA;AACjD,cAAc,mCAAmC,CAAA"}

package/dist/helpers/group-records.d.ts

@@ -0,0 +1,12 @@
+import { type ByteRecord, type FileSplitDelimiter } from './split-records.js';
+export type ByteSlice = {
+    start: number;
+    content: Buffer;
+};
+export type GroupRecordsOptions = {
+    delimiter: FileSplitDelimiter;
+    every?: number;
+    encoding?: BufferEncoding;
+};
+export declare function groupRecords(source: AsyncIterable<ByteRecord>, { delimiter, every, encoding }: GroupRecordsOptions): AsyncGenerator<ByteSlice>;
+//# sourceMappingURL=group-records.d.ts.map

package/dist/helpers/group-records.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"group-records.d.ts","sourceRoot":"","sources":["../../src/helpers/group-records.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,UAAU,EACf,KAAK,kBAAkB,EAIxB,MAAM,oBAAoB,CAAA;AAE3B,MAAM,MAAM,SAAS,GAAG;IACtB,KAAK,EAAE,MAAM,CAAA;IACb,OAAO,EAAE,MAAM,CAAA;CAChB,CAAA;AAED,MAAM,MAAM,mBAAmB,GAAG;IAChC,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,QAAQ,CAAC,EAAE,cAAc,CAAA;CAC1B,CAAA;AAED,wBAAgB,YAAY,CAC1B,MAAM,EAAE,aAAa,CAAC,UAAU,CAAC,EACjC,EAAE,SAAS,EAAE,KAAS,EAAE,QAAiB,EAAE,EAAE,mBAAmB,GAC/D,cAAc,CAAC,SAAS,CAAC,CAW3B"}

package/dist/helpers/group-records.js

@@ -0,0 +1,36 @@
+import { MAX_SLICE_BYTES, sliceTooLarge, toDelimiter, } from './split-records.js';
+export function groupRecords(source, { delimiter, every = 1, encoding = 'utf8' }) {
+    const { bytes, keepPrevious, keepNext } = toDelimiter(delimiter, encoding);
+    if (!Number.isInteger(every) || every < 1)
+        throw new TypeError('Split every must be a positive integer');
+    return groups(source, bytes.subarray(keepPrevious, bytes.length - keepNext), every);
+}
+async function* groups(source, joint, every) {
+    let parts = [];
+    let sizeInBytes = 0;
+    let count = 0;
+    let start = 0;
+    for await (const record of source) {
+        if (count === 0)
+            start = record.start;
+        else {
+            parts.push(joint);
+            sizeInBytes += joint.length;
+        }
+        for (const piece of record.pieces) {
+            parts.push(piece);
+            sizeInBytes += piece.length;
+        }
+        if (sizeInBytes > MAX_SLICE_BYTES)
+            throw sliceTooLarge(start, sizeInBytes);
+        if (++count < every)
+            continue;
+        yield { start, content: Buffer.concat(parts, sizeInBytes) };
+        parts = [];
+        sizeInBytes = 0;
+        count = 0;
+    }
+    if (count > 0)
+        yield { start, content: Buffer.concat(parts, sizeInBytes) };
+}
+//# sourceMappingURL=group-records.js.map

package/dist/helpers/group-records.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"group-records.js","sourceRoot":"","sources":["../../src/helpers/group-records.ts"],"names":[],"mappings":"AAAA,OAAO,EAGL,eAAe,EACf,aAAa,EACb,WAAW,GACZ,MAAM,oBAAoB,CAAA;AAa3B,MAAM,UAAU,YAAY,CAC1B,MAAiC,EACjC,EAAE,SAAS,EAAE,KAAK,GAAG,CAAC,EAAE,QAAQ,GAAG,MAAM,EAAuB;IAEhE,MAAM,EAAE,KAAK,EAAE,YAAY,EAAE,QAAQ,EAAE,GAAG,WAAW,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAA;IAE1E,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,GAAG,CAAC;QACvC,MAAM,IAAI,SAAS,CAAC,wCAAwC,CAAC,CAAA;IAE/D,OAAO,MAAM,CACX,MAAM,EACN,KAAK,CAAC,QAAQ,CAAC,YAAY,EAAE,KAAK,CAAC,MAAM,GAAG,QAAQ,CAAC,EACrD,KAAK,CACN,CAAA;AACH,CAAC;AAED,KAAK,SAAS,CAAC,CAAC,MAAM,CACpB,MAAiC,EACjC,KAAa,EACb,KAAa;IAEb,IAAI,KAAK,GAAa,EAAE,CAAA;IACxB,IAAI,WAAW,GAAG,CAAC,CAAA;IACnB,IAAI,KAAK,GAAG,CAAC,CAAA;IACb,IAAI,KAAK,GAAG,CAAC,CAAA;IAEb,IAAI,KAAK,EAAE,MAAM,MAAM,IAAI,MAAM,EAAE,CAAC;QAClC,IAAI,KAAK,KAAK,CAAC;YAAE,KAAK,GAAG,MAAM,CAAC,KAAK,CAAA;aAChC,CAAC;YACJ,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;YAEjB,WAAW,IAAI,KAAK,CAAC,MAAM,CAAA;QAC7B,CAAC;QAED,KAAK,MAAM,KAAK,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;YAClC,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;YAEjB,WAAW,IAAI,KAAK,CAAC,MAAM,CAAA;QAC7B,CAAC;QAED,IAAI,WAAW,GAAG,eAAe;YAAE,MAAM,aAAa,CAAC,KAAK,EAAE,WAAW,CAAC,CAAA;QAE1E,IAAI,EAAE,KAAK,GAAG,KAAK;YAAE,SAAQ;QAE7B,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,EAAE,WAAW,CAAC,EAAE,CAAA;QAE3D,KAAK,GAAG,EAAE,CAAA;QACV,WAAW,GAAG,CAAC,CAAA;QACf,KAAK,GAAG,CAAC,CAAA;IACX,CAAC;IAED,IAAI,KAAK,GAAG,CAAC;QAAE,MAAM,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,CAAC,MAAM,CAAC,KAAK,EAAE,WAAW,CAAC,EAAE,CAAA;AAC5E,CAAC"}

package/dist/helpers/split-records.d.ts

@@ -0,0 +1,30 @@
+export type FileSplitDelimiter = {
+    value: string;
+    keep?: {
+        previous?: number;
+        next?: number;
+    };
+};
+export type ByteTrim = {
+    start?: number;
+    end?: number;
+};
+export type ByteRecord = {
+    start: number;
+    pieces: Buffer[];
+};
+export type SplitRecordsOptions = {
+    delimiter: FileSplitDelimiter;
+    trim?: ByteTrim;
+    encoding?: BufferEncoding;
+};
+export declare const MAX_SLICE_BYTES: number;
+export type ByteDelimiter = {
+    bytes: Buffer;
+    keepPrevious: number;
+    keepNext: number;
+};
+export declare function toDelimiter({ value, keep }: FileSplitDelimiter, encoding: BufferEncoding): ByteDelimiter;
+export declare function sliceTooLarge(start: number, sizeInBytes: number): RangeError;
+export declare function splitRecords(source: AsyncIterable<Buffer>, { delimiter, trim, encoding }: SplitRecordsOptions): AsyncGenerator<ByteRecord>;
+//# sourceMappingURL=split-records.d.ts.map

package/dist/helpers/split-records.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"split-records.d.ts","sourceRoot":"","sources":["../../src/helpers/split-records.ts"],"names":[],"mappings":"AAAA,MAAM,MAAM,kBAAkB,GAAG;IAC/B,KAAK,EAAE,MAAM,CAAA;IACb,IAAI,CAAC,EAAE;QACL,QAAQ,CAAC,EAAE,MAAM,CAAA;QACjB,IAAI,CAAC,EAAE,MAAM,CAAA;KACd,CAAA;CACF,CAAA;AAED,MAAM,MAAM,QAAQ,GAAG;IACrB,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,GAAG,CAAC,EAAE,MAAM,CAAA;CACb,CAAA;AAED,MAAM,MAAM,UAAU,GAAG;IACvB,KAAK,EAAE,MAAM,CAAA;IACb,MAAM,EAAE,MAAM,EAAE,CAAA;CACjB,CAAA;AAED,MAAM,MAAM,mBAAmB,GAAG;IAChC,SAAS,EAAE,kBAAkB,CAAA;IAC7B,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,QAAQ,CAAC,EAAE,cAAc,CAAA;CAC1B,CAAA;AAED,eAAO,MAAM,eAAe,QAAkB,CAAA;AAE9C,MAAM,MAAM,aAAa,GAAG;IAC1B,KAAK,EAAE,MAAM,CAAA;IACb,YAAY,EAAE,MAAM,CAAA;IACpB,QAAQ,EAAE,MAAM,CAAA;CACjB,CAAA;AAED,wBAAgB,WAAW,CACzB,EAAE,KAAK,EAAE,IAAI,EAAE,EAAE,kBAAkB,EACnC,QAAQ,EAAE,cAAc,GACvB,aAAa,CAiBf;AAED,wBAAgB,aAAa,CAAC,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,cAI/D;AAED,wBAAgB,YAAY,CAC1B,MAAM,EAAE,aAAa,CAAC,MAAM,CAAC,EAC7B,EAAE,SAAS,EAAE,IAAI,EAAE,QAAiB,EAAE,EAAE,mBAAmB,GAC1D,cAAc,CAAC,UAAU,CAAC,CAkB5B"}

package/dist/helpers/split-records.js

@@ -0,0 +1,112 @@
+export const MAX_SLICE_BYTES = 3 * 1024 * 1024;
+export function toDelimiter({ value, keep }, encoding) {
+    const bytes = Buffer.from(value, encoding);
+    const keepPrevious = keep?.previous ?? 0;
+    const keepNext = keep?.next ?? 0;
+    if (bytes.length === 0)
+        throw new TypeError('Split delimiter must not be empty');
+    if (keepPrevious < 0 || keepNext < 0)
+        throw new TypeError('Split delimiter keep sizes must not be negative');
+    if (keepPrevious + keepNext > bytes.length)
+        throw new TypeError('Split delimiter keep sizes must not exceed the delimiter length');
+    return { bytes, keepPrevious, keepNext };
+}
+export function sliceTooLarge(start, sizeInBytes) {
+    return new RangeError(`Split slice starting at byte ${start} is larger than ${MAX_SLICE_BYTES} bytes (${sizeInBytes} so far): lower every or check the delimiter`);
+}
+export function splitRecords(source, { delimiter, trim, encoding = 'utf8' }) {
+    const cut = toDelimiter(delimiter, encoding);
+    const trimStart = trim?.start ?? 0;
+    const trimEnd = trim?.end ?? 0;
+    if (!Number.isInteger(trimStart) ||
+        !Number.isInteger(trimEnd) ||
+        trimStart < 0 ||
+        trimEnd < 0)
+        throw new TypeError('Split trim sizes must be non-negative integers');
+    return records(skipLast(skipFirst(source, trimStart), trimEnd), cut, trimStart);
+}
+async function* records(source, { bytes, keepPrevious, keepNext }, offset) {
+    let pieces = [];
+    let start = offset;
+    let consumed = offset;
+    let searchAt = offset;
+    let tail = Buffer.alloc(0);
+    const take = (end) => {
+        const taken = [];
+        let remaining = end - start;
+        while (remaining > 0) {
+            const piece = pieces[0];
+            if (piece.length <= remaining) {
+                taken.push(piece);
+                pieces.shift();
+                remaining -= piece.length;
+                continue;
+            }
+            taken.push(piece.subarray(0, remaining));
+            pieces[0] = piece.subarray(remaining);
+            remaining = 0;
+        }
+        start = end;
+        return taken;
+    };
+    for await (const chunk of source) {
+        if (chunk.length === 0)
+            continue;
+        const chunkAt = consumed;
+        pieces.push(chunk);
+        consumed += chunk.length;
+        const matches = [];
+        if (tail.length > 0) {
+            const probe = Buffer.concat([tail, chunk.subarray(0, bytes.length - 1)]);
+            const probeAt = chunkAt - tail.length;
+            const index = probe.indexOf(bytes, Math.max(0, searchAt - probeAt));
+            if (index !== -1 && probeAt + index < chunkAt) {
+                matches.push(probeAt + index);
+                searchAt = probeAt + index + bytes.length;
+            }
+        }
+        let index;
+        while ((index = chunk.indexOf(bytes, Math.max(0, searchAt - chunkAt))) !== -1) {
+            matches.push(chunkAt + index);
+            searchAt = chunkAt + index + bytes.length;
+        }
+        for (const at of matches) {
+            const record = { start, pieces: take(at + keepPrevious) };
+            take(at + bytes.length - keepNext);
+            yield record;
+        }
+        if (consumed - start > MAX_SLICE_BYTES)
+            throw sliceTooLarge(start, consumed - start);
+        tail =
+            chunk.length >= bytes.length - 1
+                ? Buffer.from(chunk.subarray(chunk.length - (bytes.length - 1)))
+                : Buffer.concat([tail, chunk]).subarray(-(bytes.length - 1));
+    }
+    const last = { start, pieces };
+    pieces = [];
+    yield last;
+}
+async function* skipFirst(source, count) {
+    let remaining = count;
+    for await (const chunk of source) {
+        if (remaining >= chunk.length) {
+            remaining -= chunk.length;
+            continue;
+        }
+        yield chunk.subarray(remaining);
+        remaining = 0;
+    }
+}
+async function* skipLast(source, count) {
+    if (count === 0)
+        return yield* source;
+    let tail = Buffer.alloc(0);
+    for await (const chunk of source) {
+        const buffer = Buffer.concat([tail, chunk]);
+        const end = Math.max(0, buffer.length - count);
+        if (end > 0)
+            yield buffer.subarray(0, end);
+        tail = buffer.subarray(end);
+    }
+}
+//# sourceMappingURL=split-records.js.map

package/dist/helpers/split-records.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"split-records.js","sourceRoot":"","sources":["../../src/helpers/split-records.ts"],"names":[],"mappings":"AAwBA,MAAM,CAAC,MAAM,eAAe,GAAG,CAAC,GAAG,IAAI,GAAG,IAAI,CAAA;AAQ9C,MAAM,UAAU,WAAW,CACzB,EAAE,KAAK,EAAE,IAAI,EAAsB,EACnC,QAAwB;IAExB,MAAM,KAAK,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAA;IAC1C,MAAM,YAAY,GAAG,IAAI,EAAE,QAAQ,IAAI,CAAC,CAAA;IACxC,MAAM,QAAQ,GAAG,IAAI,EAAE,IAAI,IAAI,CAAC,CAAA;IAEhC,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QACpB,MAAM,IAAI,SAAS,CAAC,mCAAmC,CAAC,CAAA;IAE1D,IAAI,YAAY,GAAG,CAAC,IAAI,QAAQ,GAAG,CAAC;QAClC,MAAM,IAAI,SAAS,CAAC,iDAAiD,CAAC,CAAA;IAExE,IAAI,YAAY,GAAG,QAAQ,GAAG,KAAK,CAAC,MAAM;QACxC,MAAM,IAAI,SAAS,CACjB,iEAAiE,CAClE,CAAA;IAEH,OAAO,EAAE,KAAK,EAAE,YAAY,EAAE,QAAQ,EAAE,CAAA;AAC1C,CAAC;AAED,MAAM,UAAU,aAAa,CAAC,KAAa,EAAE,WAAmB;IAC9D,OAAO,IAAI,UAAU,CACnB,gCAAgC,KAAK,mBAAmB,eAAe,WAAW,WAAW,8CAA8C,CAC5I,CAAA;AACH,CAAC;AAED,MAAM,UAAU,YAAY,CAC1B,MAA6B,EAC7B,EAAE,SAAS,EAAE,IAAI,EAAE,QAAQ,GAAG,MAAM,EAAuB;IAE3D,MAAM,GAAG,GAAG,WAAW,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAA;IAC5C,MAAM,SAAS,GAAG,IAAI,EAAE,KAAK,IAAI,CAAC,CAAA;IAClC,MAAM,OAAO,GAAG,IAAI,EAAE,GAAG,IAAI,CAAC,CAAA;IAE9B,IACE,CAAC,MAAM,CAAC,SAAS,CAAC,SAAS,CAAC;QAC5B,CAAC,MAAM,CAAC,SAAS,CAAC,OAAO,CAAC;QAC1B,SAAS,GAAG,CAAC;QACb,OAAO,GAAG,CAAC;QAEX,MAAM,IAAI,SAAS,CAAC,gDAAgD,CAAC,CAAA;IAEvE,OAAO,OAAO,CACZ,QAAQ,CAAC,SAAS,CAAC,MAAM,EAAE,SAAS,CAAC,EAAE,OAAO,CAAC,EAC/C,GAAG,EACH,SAAS,CACV,CAAA;AACH,CAAC;AAED,KAAK,SAAS,CAAC,CAAC,OAAO,CACrB,MAA6B,EAC7B,EAAE,KAAK,EAAE,YAAY,EAAE,QAAQ,EAAiB,EAChD,MAAc;IAEd,IAAI,MAAM,GAAa,EAAE,CAAA;IACzB,IAAI,KAAK,GAAG,MAAM,CAAA;IAClB,IAAI,QAAQ,GAAG,MAAM,CAAA;IACrB,IAAI,QAAQ,GAAG,MAAM,CAAA;IACrB,IAAI,IAAI,GAAW,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;IAElC,MAAM,IAAI,GAAG,CAAC,GAAW,EAAE,EAAE;QAC3B,MAAM,KAAK,GAAa,EAAE,CAAA;QAE1B,IAAI,SAAS,GAAG,GAAG,GAAG,KAAK,CAAA;QAE3B,OAAO,SAAS,GAAG,CAAC,EAAE,CAAC;YACrB,MAAM,KAAK,GAAG,MAAM,CAAC,CAAC,CAAE,CAAA;YAExB,IAAI,KAAK,CAAC,MAAM,IAAI,SAAS,EAAE,CAAC;gBAC9B,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;gBAEjB,MAAM,CAAC,KAAK,EAAE,CAAA;gBAEd,SAAS,IAAI,KAAK,CAAC,MAAM,CAAA;gBAEzB,SAAQ;YACV,CAAC;YAED,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC,EAAE,SAAS,CAAC,CAAC,CAAA;YAExC,MAAM,CAAC,CAAC,CAAC,GAAG,KAAK,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAA;YAErC,SAAS,GAAG,CAAC,CAAA;QACf,CAAC;QAED,KAAK,GAAG,GAAG,CAAA;QAEX,OAAO,KAAK,CAAA;IACd,CAAC,CAAA;IAED,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QACjC,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;YAAE,SAAQ;QAEhC,MAAM,OAAO,GAAG,QAAQ,CAAA;QAExB,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;QAElB,QAAQ,IAAI,KAAK,CAAC,MAAM,CAAA;QAExB,MAAM,OAAO,GAAa,EAAE,CAAA;QAE5B,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACpB,MAAM,KAAK,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,KAAK,CAAC,QAAQ,CAAC,CAAC,EAAE,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,CAAA;YACxE,MAAM,OAAO,GAAG,OAAO,GAAG,IAAI,CAAC,MAAM,CAAA;YACrC,MAAM,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,GAAG,OAAO,CAAC,CAAC,CAAA;YAEnE,IAAI,KAAK,KAAK,CAAC,CAAC,IAAI,OAAO,GAAG,KAAK,GAAG,OAAO,EAAE,CAAC;gBAC9C,OAAO,CAAC,IAAI,CAAC,OAAO,GAAG,KAAK,CAAC,CAAA;gBAE7B,QAAQ,GAAG,OAAO,GAAG,KAAK,GAAG,KAAK,CAAC,MAAM,CAAA;YAC3C,CAAC;QACH,CAAC;QAED,IAAI,KAAa,CAAA;QAEjB,OACE,CAAC,KAAK,GAAG,KAAK,CAAC,OAAO,CAAC,KAAK,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,QAAQ,GAAG,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,EACtE,CAAC;YACD,OAAO,CAAC,IAAI,CAAC,OAAO,GAAG,KAAK,CAAC,CAAA;YAE7B,QAAQ,GAAG,OAAO,GAAG,KAAK,GAAG,KAAK,CAAC,MAAM,CAAA;QAC3C,CAAC;QAED,KAAK,MAAM,EAAE,IAAI,OAAO,EAAE,CAAC;YACzB,MAAM,MAAM,GAAG,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,GAAG,YAAY,CAAC,EAAE,CAAA;YAEzD,IAAI,CAAC,EAAE,GAAG,KAAK,CAAC,MAAM,GAAG,QAAQ,CAAC,CAAA;YAElC,MAAM,MAAM,CAAA;QACd,CAAC;QAED,IAAI,QAAQ,GAAG,KAAK,GAAG,eAAe;YACpC,MAAM,aAAa,CAAC,KAAK,EAAE,QAAQ,GAAG,KAAK,CAAC,CAAA;QAE9C,IAAI;YACF,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC;gBAC9B,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC;gBAChE,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAA;IAClE,CAAC;IAED,MAAM,IAAI,GAAG,EAAE,KAAK,EAAE,MAAM,EAAE,CAAA;IAE9B,MAAM,GAAG,EAAE,CAAA;IAEX,MAAM,IAAI,CAAA;AACZ,CAAC;AAED,KAAK,SAAS,CAAC,CAAC,SAAS,CAAC,MAA6B,EAAE,KAAa;IACpE,IAAI,SAAS,GAAG,KAAK,CAAA;IAErB,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QACjC,IAAI,SAAS,IAAI,KAAK,CAAC,MAAM,EAAE,CAAC;YAC9B,SAAS,IAAI,KAAK,CAAC,MAAM,CAAA;YAEzB,SAAQ;QACV,CAAC;QAED,MAAM,KAAK,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAA;QAE/B,SAAS,GAAG,CAAC,CAAA;IACf,CAAC;AACH,CAAC;AAED,KAAK,SAAS,CAAC,CAAC,QAAQ,CAAC,MAA6B,EAAE,KAAa;IACnE,IAAI,KAAK,KAAK,CAAC;QAAE,OAAO,KAAK,CAAC,CAAC,MAAM,CAAA;IAErC,IAAI,IAAI,GAAW,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;IAElC,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;QACjC,MAAM,MAAM,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,CAAA;QAC3C,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,MAAM,CAAC,MAAM,GAAG,KAAK,CAAC,CAAA;QAE9C,IAAI,GAAG,GAAG,CAAC;YAAE,MAAM,MAAM,CAAC,QAAQ,CAAC,CAAC,EAAE,GAAG,CAAC,CAAA;QAE1C,IAAI,GAAG,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAA;IAC7B,CAAC;AACH,CAAC"}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t4h.framework/fs",
-  "version": "0.12.1",
+  "version": "0.13.0",
   "description": "File system module for the T4H Framework",
   "homepage": "https://github.com/tech4humans-brasil/framework/tree/main/packages/fs",
   "bugs": "https://github.com/tech4humans-brasil/framework/issues",
@@ -32,7 +32,7 @@
     "test:watch": "vitest"
   },
   "devDependencies": {
-    "@t4h.framework/core": "^0.17.1",
+    "@t4h.framework/core": "^0.19.0",
     "typescript": "^5.9.3",
     "vitest": "^4.0.18"
   },

package/dist/helpers/ByteDelimiterScanner.d.ts

@@ -1,50 +0,0 @@
-import { Transform, type TransformCallback } from 'node:stream';
-export type ByteRange = {
-    start: number;
-    end: number;
-};
-export type ByteSlice = ByteRange & {
-    content?: Buffer;
-};
-export type FileSplitDelimiter = {
-    value: string;
-    keep?: {
-        previous?: number;
-        next?: number;
-    };
-};
-export type ByteTrim = {
-    start?: number;
-    end?: number;
-};
-export type ByteDelimiterScannerOptions = {
-    delimiter: FileSplitDelimiter;
-    every?: number;
-    trim?: ByteTrim;
-    skipEmpty?: boolean;
-    encoding?: BufferEncoding;
-    inlineMaxBytes?: number;
-};
-export declare const DEFAULT_INLINE_MAX_BYTES: number;
-export declare class ByteDelimiterScanner extends Transform {
-    readonly slices: ByteSlice[];
-    private readonly delimiter;
-    private readonly keepPrevious;
-    private readonly keepNext;
-    private readonly cutEvery;
-    private readonly trimStart;
-    private readonly trimEnd;
-    private readonly skipEmpty;
-    private readonly inlineMaxBytes;
-    private pending;
-    private pendingAt;
-    private searchAt;
-    private sliceStart;
-    private matches;
-    constructor({ delimiter, every, trim, skipEmpty, encoding, inlineMaxBytes, }: ByteDelimiterScannerOptions);
-    _transform(chunk: Buffer, _encoding: BufferEncoding, callback: TransformCallback): void;
-    _flush(callback: TransformCallback): void;
-    private scan;
-    private cut;
-}
-//# sourceMappingURL=ByteDelimiterScanner.d.ts.map

package/dist/helpers/ByteDelimiterScanner.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"ByteDelimiterScanner.d.ts","sourceRoot":"","sources":["../../src/helpers/ByteDelimiterScanner.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,KAAK,iBAAiB,EAAE,MAAM,aAAa,CAAA;AAE/D,MAAM,MAAM,SAAS,GAAG;IACtB,KAAK,EAAE,MAAM,CAAA;IACb,GAAG,EAAE,MAAM,CAAA;CACZ,CAAA;AAED,MAAM,MAAM,SAAS,GAAG,SAAS,GAAG;IAClC,OAAO,CAAC,EAAE,MAAM,CAAA;CACjB,CAAA;AAED,MAAM,MAAM,kBAAkB,GAAG;IAC/B,KAAK,EAAE,MAAM,CAAA;IACb,IAAI,CAAC,EAAE;QACL,QAAQ,CAAC,EAAE,MAAM,CAAA;QACjB,IAAI,CAAC,EAAE,MAAM,CAAA;KACd,CAAA;CACF,CAAA;AAED,MAAM,MAAM,QAAQ,GAAG;IACrB,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,GAAG,CAAC,EAAE,MAAM,CAAA;CACb,CAAA;AAED,MAAM,MAAM,2BAA2B,GAAG;IACxC,SAAS,EAAE,kBAAkB,CAAA;IAC7B,KAAK,CAAC,EAAE,MAAM,CAAA;IACd,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,SAAS,CAAC,EAAE,OAAO,CAAA;IACnB,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,cAAc,CAAC,EAAE,MAAM,CAAA;CACxB,CAAA;AAED,eAAO,MAAM,wBAAwB,QAAY,CAAA;AAEjD,qBAAa,oBAAqB,SAAQ,SAAS;IACjD,SAAgB,MAAM,EAAE,SAAS,EAAE,CAAK;IAExC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAQ;IAClC,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAQ;IACrC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAQ;IACjC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAQ;IACjC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAQ;IAClC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAQ;IAChC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAQ;IAEvC,OAAO,CAAC,OAAO,CAA0B;IACzC,OAAO,CAAC,SAAS,CAAI;IACrB,OAAO,CAAC,QAAQ,CAAI;IACpB,OAAO,CAAC,UAAU,CAAI;IACtB,OAAO,CAAC,OAAO,CAAI;gBAEP,EACV,SAAS,EACT,KAAS,EACT,IAAI,EACJ,SAAgB,EAChB,QAAiB,EACjB,cAAyC,GAC1C,EAAE,2BAA2B;IAuCvB,UAAU,CACf,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,cAAc,EACzB,QAAQ,EAAE,iBAAiB;IAStB,MAAM,CAAC,QAAQ,EAAE,iBAAiB;IAyBzC,OAAO,CAAC,IAAI;IAyCZ,OAAO,CAAC,GAAG;CAkBZ"}

package/dist/helpers/ByteDelimiterScanner.js

@@ -1,103 +0,0 @@
-import { Transform } from 'node:stream';
-export const DEFAULT_INLINE_MAX_BYTES = 16 * 1024;
-export class ByteDelimiterScanner extends Transform {
-    slices = [];
-    delimiter;
-    keepPrevious;
-    keepNext;
-    cutEvery;
-    trimStart;
-    trimEnd;
-    skipEmpty;
-    inlineMaxBytes;
-    pending = Buffer.alloc(0);
-    pendingAt = 0;
-    searchAt = 0;
-    sliceStart = 0;
-    matches = 0;
-    constructor({ delimiter, every = 1, trim, skipEmpty = true, encoding = 'utf8', inlineMaxBytes = DEFAULT_INLINE_MAX_BYTES, }) {
-        super();
-        this.delimiter = Buffer.from(delimiter.value, encoding);
-        this.keepPrevious = delimiter.keep?.previous ?? 0;
-        this.keepNext = delimiter.keep?.next ?? 0;
-        this.cutEvery = every;
-        this.trimStart = trim?.start ?? 0;
-        this.trimEnd = trim?.end ?? 0;
-        this.skipEmpty = skipEmpty;
-        this.inlineMaxBytes = inlineMaxBytes;
-        this.sliceStart = this.trimStart;
-        if (this.delimiter.length === 0)
-            throw new TypeError('Split delimiter must not be empty');
-        if (this.keepPrevious < 0 || this.keepNext < 0)
-            throw new TypeError('Split delimiter keep sizes must not be negative');
-        if (this.keepPrevious + this.keepNext > this.delimiter.length)
-            throw new TypeError('Split delimiter keep sizes must not exceed the delimiter length');
-        if (!Number.isInteger(every) || every < 1)
-            throw new TypeError('Split every must be a positive integer');
-        if (!Number.isInteger(this.trimStart) ||
-            !Number.isInteger(this.trimEnd) ||
-            this.trimStart < 0 ||
-            this.trimEnd < 0)
-            throw new TypeError('Split trim sizes must be non-negative integers');
-        if (!Number.isInteger(inlineMaxBytes) || inlineMaxBytes < 0)
-            throw new TypeError('Split inlineMaxBytes must be a non-negative integer');
-    }
-    _transform(chunk, _encoding, callback) {
-        this.scan(this.pending.length > 0 ? Buffer.concat([this.pending, chunk]) : chunk);
-        callback(null, chunk);
-    }
-    _flush(callback) {
-        const limit = this.pendingAt + this.pending.length - this.trimEnd;
-        while (true) {
-            const last = this.slices.at(-1);
-            if (!last || last.end <= limit)
-                break;
-            if (last.start >= limit) {
-                this.slices.pop();
-                continue;
-            }
-            last.end = limit;
-            if (last.content)
-                last.content = last.content.subarray(0, limit - last.start);
-        }
-        if (this.sliceStart <= limit)
-            this.cut(this.sliceStart, limit, this.pending);
-        callback();
-    }
-    scan(buffer) {
-        let searchFrom = Math.max(0, this.searchAt - this.pendingAt);
-        while (true) {
-            const index = buffer.indexOf(this.delimiter, searchFrom);
-            if (index === -1)
-                break;
-            const at = this.pendingAt + index;
-            searchFrom = index + this.delimiter.length;
-            if (at < this.trimStart)
-                continue;
-            this.matches += 1;
-            if (this.matches % this.cutEvery !== 0)
-                continue;
-            this.cut(this.sliceStart, at + this.keepPrevious, buffer);
-            this.sliceStart = at + this.delimiter.length - this.keepNext;
-        }
-        const tail = Math.max(searchFrom, buffer.length - (this.delimiter.length - 1));
-        this.searchAt = this.pendingAt + tail;
-        const openAt = this.sliceStart - this.pendingAt;
-        const isOpenSliceInline = openAt >= 0 && buffer.length - openAt <= this.inlineMaxBytes;
-        const keepFrom = isOpenSliceInline ? Math.min(tail, openAt) : tail;
-        this.pending = buffer.subarray(keepFrom);
-        this.pendingAt += keepFrom;
-    }
-    cut(start, end, buffer) {
-        if (this.skipEmpty && end <= start)
-            return;
-        const isInline = start >= this.pendingAt && end - start <= this.inlineMaxBytes;
-        if (!isInline) {
-            this.slices.push({ start, end });
-            return;
-        }
-        const content = Buffer.from(buffer.subarray(start - this.pendingAt, end - this.pendingAt));
-        this.slices.push({ start, end, content });
-    }
-}
-//# sourceMappingURL=ByteDelimiterScanner.js.map

package/dist/helpers/ByteDelimiterScanner.js.map

@@ -1 +0,0 @@
-{"version":3,"file":"ByteDelimiterScanner.js","sourceRoot":"","sources":["../../src/helpers/ByteDelimiterScanner.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAA0B,MAAM,aAAa,CAAA;AAiC/D,MAAM,CAAC,MAAM,wBAAwB,GAAG,EAAE,GAAG,IAAI,CAAA;AAEjD,MAAM,OAAO,oBAAqB,SAAQ,SAAS;IACjC,MAAM,GAAgB,EAAE,CAAA;IAEvB,SAAS,CAAQ;IACjB,YAAY,CAAQ;IACpB,QAAQ,CAAQ;IAChB,QAAQ,CAAQ;IAChB,SAAS,CAAQ;IACjB,OAAO,CAAQ;IACf,SAAS,CAAS;IAClB,cAAc,CAAQ;IAE/B,OAAO,GAAW,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;IACjC,SAAS,GAAG,CAAC,CAAA;IACb,QAAQ,GAAG,CAAC,CAAA;IACZ,UAAU,GAAG,CAAC,CAAA;IACd,OAAO,GAAG,CAAC,CAAA;IAEnB,YAAY,EACV,SAAS,EACT,KAAK,GAAG,CAAC,EACT,IAAI,EACJ,SAAS,GAAG,IAAI,EAChB,QAAQ,GAAG,MAAM,EACjB,cAAc,GAAG,wBAAwB,GACb;QAC5B,KAAK,EAAE,CAAA;QAEP,IAAI,CAAC,SAAS,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAA;QACvD,IAAI,CAAC,YAAY,GAAG,SAAS,CAAC,IAAI,EAAE,QAAQ,IAAI,CAAC,CAAA;QACjD,IAAI,CAAC,QAAQ,GAAG,SAAS,CAAC,IAAI,EAAE,IAAI,IAAI,CAAC,CAAA;QACzC,IAAI,CAAC,QAAQ,GAAG,KAAK,CAAA;QACrB,IAAI,CAAC,SAAS,GAAG,IAAI,EAAE,KAAK,IAAI,CAAC,CAAA;QACjC,IAAI,CAAC,OAAO,GAAG,IAAI,EAAE,GAAG,IAAI,CAAC,CAAA;QAC7B,IAAI,CAAC,SAAS,GAAG,SAAS,CAAA;QAC1B,IAAI,CAAC,cAAc,GAAG,cAAc,CAAA;QACpC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,CAAA;QAEhC,IAAI,IAAI,CAAC,SAAS,CAAC,MAAM,KAAK,CAAC;YAC7B,MAAM,IAAI,SAAS,CAAC,mCAAmC,CAAC,CAAA;QAE1D,IAAI,IAAI,CAAC,YAAY,GAAG,CAAC,IAAI,IAAI,CAAC,QAAQ,GAAG,CAAC;YAC5C,MAAM,IAAI,SAAS,CAAC,iDAAiD,CAAC,CAAA;QAExE,IAAI,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM;YAC3D,MAAM,IAAI,SAAS,CACjB,iEAAiE,CAClE,CAAA;QAEH,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,KAAK,CAAC,IAAI,KAAK,GAAG,CAAC;YACvC,MAAM,IAAI,SAAS,CAAC,wCAAwC,CAAC,CAAA;QAE/D,IACE,CAAC,MAAM,CAAC,SAAS,CAAC,IAAI,CAAC,SAAS,CAAC;YACjC,CAAC,MAAM,CAAC,SAAS,CAAC,IAAI,CAAC,OAAO,CAAC;YAC/B,IAAI,CAAC,SAAS,GAAG,CAAC;YAClB,IAAI,CAAC,OAAO,GAAG,CAAC;YAEhB,MAAM,IAAI,SAAS,CAAC,gDAAgD,CAAC,CAAA;QAEvE,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,cAAc,CAAC,IAAI,cAAc,GAAG,CAAC;YACzD,MAAM,IAAI,SAAS,CAAC,qDAAqD,CAAC,CAAA;IAC9E,CAAC;IAEM,UAAU,CACf,KAAa,EACb,SAAyB,EACzB,QAA2B;QAE3B,IAAI,CAAC,IAAI,CACP,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CACvE,CAAA;QAED,QAAQ,CAAC,IAAI,EAAE,KAAK,CAAC,CAAA;IACvB,CAAC;IAEM,MAAM,CAAC,QAA2B;QACvC,MAAM,KAAK,GAAG,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,OAAO,CAAA;QAEjE,OAAO,IAAI,EAAE,CAAC;YACZ,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAA;YAE/B,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,GAAG,IAAI,KAAK;gBAAE,MAAK;YAErC,IAAI,IAAI,CAAC,KAAK,IAAI,KAAK,EAAE,CAAC;gBACxB,IAAI,CAAC,MAAM,CAAC,GAAG,EAAE,CAAA;gBAEjB,SAAQ;YACV,CAAC;YAED,IAAI,CAAC,GAAG,GAAG,KAAK,CAAA;YAEhB,IAAI,IAAI,CAAC,OAAO;gBACd,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC,EAAE,KAAK,GAAG,IAAI,CAAC,KAAK,CAAC,CAAA;QAC/D,CAAC;QAED,IAAI,IAAI,CAAC,UAAU,IAAI,KAAK;YAAE,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,UAAU,EAAE,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,CAAA;QAE5E,QAAQ,EAAE,CAAA;IACZ,CAAC;IAEO,IAAI,CAAC,MAAc;QACzB,IAAI,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,CAAA;QAE5D,OAAO,IAAI,EAAE,CAAC;YACZ,MAAM,KAAK,GAAG,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,SAAS,EAAE,UAAU,CAAC,CAAA;YAExD,IAAI,KAAK,KAAK,CAAC,CAAC;gBAAE,MAAK;YAEvB,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,GAAG,KAAK,CAAA;YAEjC,UAAU,GAAG,KAAK,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAA;YAE1C,IAAI,EAAE,GAAG,IAAI,CAAC,SAAS;gBAAE,SAAQ;YAEjC,IAAI,CAAC,OAAO,IAAI,CAAC,CAAA;YAEjB,IAAI,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,QAAQ,KAAK,CAAC;gBAAE,SAAQ;YAEhD,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,UAAU,EAAE,EAAE,GAAG,IAAI,CAAC,YAAY,EAAE,MAAM,CAAC,CAAA;YAEzD,IAAI,CAAC,UAAU,GAAG,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAA;QAC9D,CAAC;QAED,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,CACnB,UAAU,EACV,MAAM,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAC5C,CAAA;QAED,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,GAAG,IAAI,CAAA;QAErC,MAAM,MAAM,GAAG,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,SAAS,CAAA;QAE/C,MAAM,iBAAiB,GACrB,MAAM,IAAI,CAAC,IAAI,MAAM,CAAC,MAAM,GAAG,MAAM,IAAI,IAAI,CAAC,cAAc,CAAA;QAE9D,MAAM,QAAQ,GAAG,iBAAiB,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC,IAAI,CAAA;QAElE,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAA;QACxC,IAAI,CAAC,SAAS,IAAI,QAAQ,CAAA;IAC5B,CAAC;IAEO,GAAG,CAAC,KAAa,EAAE,GAAW,EAAE,MAAc;QACpD,IAAI,IAAI,CAAC,SAAS,IAAI,GAAG,IAAI,KAAK;YAAE,OAAM;QAE1C,MAAM,QAAQ,GACZ,KAAK,IAAI,IAAI,CAAC,SAAS,IAAI,GAAG,GAAG,KAAK,IAAI,IAAI,CAAC,cAAc,CAAA;QAE/D,IAAI,CAAC,QAAQ,EAAE,CAAC;YACd,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC,CAAA;YAEhC,OAAM;QACR,CAAC;QAED,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,CACzB,MAAM,CAAC,QAAQ,CAAC,KAAK,GAAG,IAAI,CAAC,SAAS,EAAE,GAAG,GAAG,IAAI,CAAC,SAAS,CAAC,CAC9D,CAAA;QAED,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,GAAG,EAAE,OAAO,EAAE,CAAC,CAAA;IAC3C,CAAC;CACF"}

package/dist/helpers/ByteRecordFilter.d.ts

@@ -1,32 +0,0 @@
-import { Transform, type TransformCallback } from 'node:stream';
-import type { ByteTrim, FileSplitDelimiter } from './ByteDelimiterScanner.js';
-export type ByteRecordFilterOptions = {
-    delimiter: FileSplitDelimiter;
-    accept: (record: Buffer) => boolean;
-    trim?: ByteTrim;
-    encoding?: BufferEncoding;
-    maxRecordBytes?: number;
-};
-export declare const DEFAULT_FILTER_MAX_RECORD_BYTES: number;
-export declare class ByteRecordFilter extends Transform {
-    accepted: number;
-    private readonly delimiter;
-    private readonly joint;
-    private readonly keepPrevious;
-    private readonly keepNext;
-    private readonly trimStart;
-    private readonly trimEnd;
-    private readonly maxRecordBytes;
-    private readonly accept;
-    private pending;
-    private pendingAt;
-    private received;
-    private searchAt;
-    private recordStart;
-    constructor({ delimiter, accept, trim, encoding, maxRecordBytes, }: ByteRecordFilterOptions);
-    _transform(chunk: Buffer, _encoding: BufferEncoding, callback: TransformCallback): void;
-    _flush(callback: TransformCallback): void;
-    private scan;
-    private settle;
-}
-//# sourceMappingURL=ByteRecordFilter.d.ts.map

package/dist/helpers/ByteRecordFilter.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"ByteRecordFilter.d.ts","sourceRoot":"","sources":["../../src/helpers/ByteRecordFilter.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAAE,KAAK,iBAAiB,EAAE,MAAM,aAAa,CAAA;AAE/D,OAAO,KAAK,EAAE,QAAQ,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAA;AAE7E,MAAM,MAAM,uBAAuB,GAAG;IACpC,SAAS,EAAE,kBAAkB,CAAA;IAC7B,MAAM,EAAE,CAAC,MAAM,EAAE,MAAM,KAAK,OAAO,CAAA;IACnC,IAAI,CAAC,EAAE,QAAQ,CAAA;IACf,QAAQ,CAAC,EAAE,cAAc,CAAA;IACzB,cAAc,CAAC,EAAE,MAAM,CAAA;CACxB,CAAA;AAED,eAAO,MAAM,+BAA+B,QAAc,CAAA;AAE1D,qBAAa,gBAAiB,SAAQ,SAAS;IACtC,QAAQ,SAAI;IAEnB,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAQ;IAClC,OAAO,CAAC,QAAQ,CAAC,KAAK,CAAQ;IAC9B,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAQ;IACrC,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAQ;IACjC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAQ;IAClC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAQ;IAChC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAQ;IACvC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAA6B;IAEpD,OAAO,CAAC,OAAO,CAA0B;IACzC,OAAO,CAAC,SAAS,CAAI;IACrB,OAAO,CAAC,QAAQ,CAAI;IACpB,OAAO,CAAC,QAAQ,CAAQ;IACxB,OAAO,CAAC,WAAW,CAAQ;gBAEf,EACV,SAAS,EACT,MAAM,EACN,IAAI,EACJ,QAAiB,EACjB,cAAgD,GACjD,EAAE,uBAAuB;IAqBnB,UAAU,CACf,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,cAAc,EACzB,QAAQ,EAAE,iBAAiB;IAiBtB,MAAM,CAAC,QAAQ,EAAE,iBAAiB;IAYzC,OAAO,CAAC,IAAI;IA8CZ,OAAO,CAAC,MAAM;CAcf"}

package/dist/helpers/ByteRecordFilter.js

@@ -1,89 +0,0 @@
-import { Transform } from 'node:stream';
-export const DEFAULT_FILTER_MAX_RECORD_BYTES = 1024 * 1024;
-export class ByteRecordFilter extends Transform {
-    accepted = 0;
-    delimiter;
-    joint;
-    keepPrevious;
-    keepNext;
-    trimStart;
-    trimEnd;
-    maxRecordBytes;
-    accept;
-    pending = Buffer.alloc(0);
-    pendingAt = 0;
-    received = 0;
-    searchAt;
-    recordStart;
-    constructor({ delimiter, accept, trim, encoding = 'utf8', maxRecordBytes = DEFAULT_FILTER_MAX_RECORD_BYTES, }) {
-        super();
-        this.delimiter = Buffer.from(delimiter.value, encoding);
-        this.keepPrevious = delimiter.keep?.previous ?? 0;
-        this.keepNext = delimiter.keep?.next ?? 0;
-        this.joint = this.delimiter.subarray(this.keepPrevious, this.delimiter.length - this.keepNext);
-        this.trimStart = trim?.start ?? 0;
-        this.trimEnd = trim?.end ?? 0;
-        this.maxRecordBytes = maxRecordBytes;
-        this.accept = accept;
-        this.searchAt = this.trimStart;
-        this.recordStart = this.trimStart;
-        if (this.delimiter.length === 0)
-            throw new TypeError('Split delimiter must not be empty');
-    }
-    _transform(chunk, _encoding, callback) {
-        this.pending =
-            this.pending.length > 0 ? Buffer.concat([this.pending, chunk]) : chunk;
-        this.received += chunk.length;
-        try {
-            const output = this.scan(this.received - this.trimEnd);
-            if (output.length > 0)
-                this.push(output);
-            callback();
-        }
-        catch (error) {
-            callback(error);
-        }
-    }
-    _flush(callback) {
-        const limit = this.received - this.trimEnd;
-        const output = [];
-        if (this.recordStart <= limit)
-            this.settle(this.recordStart, limit, output);
-        if (output.length > 0)
-            this.push(Buffer.concat(output));
-        callback();
-    }
-    scan(limit) {
-        const output = [];
-        while (true) {
-            const index = this.pending.indexOf(this.delimiter, this.searchAt - this.pendingAt);
-            if (index === -1) {
-                this.searchAt = Math.max(this.searchAt, this.pendingAt + this.pending.length - (this.delimiter.length - 1));
-                break;
-            }
-            const at = this.pendingAt + index;
-            if (at + this.delimiter.length > limit)
-                break;
-            this.settle(this.recordStart, at + this.keepPrevious, output);
-            this.recordStart = at + this.delimiter.length - this.keepNext;
-            this.searchAt = at + this.delimiter.length;
-        }
-        const end = this.pendingAt + this.pending.length;
-        const keepFrom = Math.min(Math.max(Math.min(this.recordStart, this.searchAt), this.pendingAt), end);
-        this.pending = this.pending.subarray(keepFrom - this.pendingAt);
-        this.pendingAt = keepFrom;
-        if (end - this.recordStart > this.maxRecordBytes)
-            throw new RangeError(`Split where can only test records up to ${this.maxRecordBytes} bytes, found a record starting at byte ${this.recordStart} without a delimiter`);
-        return Buffer.concat(output);
-    }
-    settle(start, end, output) {
-        const record = this.pending.subarray(start - this.pendingAt, end - this.pendingAt);
-        if (!this.accept(record))
-            return;
-        if (this.accepted > 0)
-            output.push(this.joint);
-        output.push(record);
-        this.accepted += 1;
-    }
-}
-//# sourceMappingURL=ByteRecordFilter.js.map

package/dist/helpers/ByteRecordFilter.js.map

@@ -1 +0,0 @@
-{"version":3,"file":"ByteRecordFilter.js","sourceRoot":"","sources":["../../src/helpers/ByteRecordFilter.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,SAAS,EAA0B,MAAM,aAAa,CAAA;AAY/D,MAAM,CAAC,MAAM,+BAA+B,GAAG,IAAI,GAAG,IAAI,CAAA;AAE1D,MAAM,OAAO,gBAAiB,SAAQ,SAAS;IACtC,QAAQ,GAAG,CAAC,CAAA;IAEF,SAAS,CAAQ;IACjB,KAAK,CAAQ;IACb,YAAY,CAAQ;IACpB,QAAQ,CAAQ;IAChB,SAAS,CAAQ;IACjB,OAAO,CAAQ;IACf,cAAc,CAAQ;IACtB,MAAM,CAA6B;IAE5C,OAAO,GAAW,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAA;IACjC,SAAS,GAAG,CAAC,CAAA;IACb,QAAQ,GAAG,CAAC,CAAA;IACZ,QAAQ,CAAQ;IAChB,WAAW,CAAQ;IAE3B,YAAY,EACV,SAAS,EACT,MAAM,EACN,IAAI,EACJ,QAAQ,GAAG,MAAM,EACjB,cAAc,GAAG,+BAA+B,GACxB;QACxB,KAAK,EAAE,CAAA;QAEP,IAAI,CAAC,SAAS,GAAG,MAAM,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,QAAQ,CAAC,CAAA;QACvD,IAAI,CAAC,YAAY,GAAG,SAAS,CAAC,IAAI,EAAE,QAAQ,IAAI,CAAC,CAAA;QACjD,IAAI,CAAC,QAAQ,GAAG,SAAS,CAAC,IAAI,EAAE,IAAI,IAAI,CAAC,CAAA;QACzC,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,SAAS,CAAC,QAAQ,CAClC,IAAI,CAAC,YAAY,EACjB,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,IAAI,CAAC,QAAQ,CACtC,CAAA;QACD,IAAI,CAAC,SAAS,GAAG,IAAI,EAAE,KAAK,IAAI,CAAC,CAAA;QACjC,IAAI,CAAC,OAAO,GAAG,IAAI,EAAE,GAAG,IAAI,CAAC,CAAA;QAC7B,IAAI,CAAC,cAAc,GAAG,cAAc,CAAA;QACpC,IAAI,CAAC,MAAM,GAAG,MAAM,CAAA;QACpB,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAA;QAC9B,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC,SAAS,CAAA;QAEjC,IAAI,IAAI,CAAC,SAAS,CAAC,MAAM,KAAK,CAAC;YAC7B,MAAM,IAAI,SAAS,CAAC,mCAAmC,CAAC,CAAA;IAC5D,CAAC;IAEM,UAAU,CACf,KAAa,EACb,SAAyB,EACzB,QAA2B;QAE3B,IAAI,CAAC,OAAO;YACV,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,IAAI,CAAC,OAAO,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAA;QACxE,IAAI,CAAC,QAAQ,IAAI,KAAK,CAAC,MAAM,CAAA;QAE7B,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,OAAO,CAAC,CAAA;YAEtD,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC;gBAAE,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA;YAExC,QAAQ,EAAE,CAAA;QACZ,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,QAAQ,CAAC,KAAc,CAAC,CAAA;QAC1B,CAAC;IACH,CAAC;IAEM,MAAM,CAAC,QAA2B;QACvC,MAAM,KAAK,GAAG,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,OAAO,CAAA;QAE1C,MAAM,MAAM,GAAa,EAAE,CAAA;QAE3B,IAAI,IAAI,CAAC,WAAW,IAAI,KAAK;YAAE,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,WAAW,EAAE,KAAK,EAAE,MAAM,CAAC,CAAA;QAE3E,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC;YAAE,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAA;QAEvD,QAAQ,EAAE,CAAA;IACZ,CAAC;IAEO,IAAI,CAAC,KAAa;QACxB,MAAM,MAAM,GAAa,EAAE,CAAA;QAE3B,OAAO,IAAI,EAAE,CAAC;YACZ,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,OAAO,CAChC,IAAI,CAAC,SAAS,EACd,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAC/B,CAAA;YAED,IAAI,KAAK,KAAK,CAAC,CAAC,EAAE,CAAC;gBACjB,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,GAAG,CACtB,IAAI,CAAC,QAAQ,EACb,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CACnE,CAAA;gBAED,MAAK;YACP,CAAC;YAED,MAAM,EAAE,GAAG,IAAI,CAAC,SAAS,GAAG,KAAK,CAAA;YAEjC,IAAI,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,KAAK;gBAAE,MAAK;YAE7C,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,WAAW,EAAE,EAAE,GAAG,IAAI,CAAC,YAAY,EAAE,MAAM,CAAC,CAAA;YAE7D,IAAI,CAAC,WAAW,GAAG,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAA;YAC7D,IAAI,CAAC,QAAQ,GAAG,EAAE,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,CAAA;QAC5C,CAAC;QAED,MAAM,GAAG,GAAG,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAA;QAEhD,MAAM,QAAQ,GAAG,IAAI,CAAC,GAAG,CACvB,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,WAAW,EAAE,IAAI,CAAC,QAAQ,CAAC,EAAE,IAAI,CAAC,SAAS,CAAC,EACnE,GAAG,CACJ,CAAA;QAED,IAAI,CAAC,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,CAAA;QAC/D,IAAI,CAAC,SAAS,GAAG,QAAQ,CAAA;QAEzB,IAAI,GAAG,GAAG,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC,cAAc;YAC9C,MAAM,IAAI,UAAU,CAClB,2CAA2C,IAAI,CAAC,cAAc,2CAA2C,IAAI,CAAC,WAAW,sBAAsB,CAChJ,CAAA;QAEH,OAAO,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAA;IAC9B,CAAC;IAEO,MAAM,CAAC,KAAa,EAAE,GAAW,EAAE,MAAgB;QACzD,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,QAAQ,CAClC,KAAK,GAAG,IAAI,CAAC,SAAS,EACtB,GAAG,GAAG,IAAI,CAAC,SAAS,CACrB,CAAA;QAED,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC;YAAE,OAAM;QAEhC,IAAI,IAAI,CAAC,QAAQ,GAAG,CAAC;YAAE,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAA;QAE9C,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAA;QAEnB,IAAI,CAAC,QAAQ,IAAI,CAAC,CAAA;IACpB,CAAC;CACF"}