readline-pager 0.2.2 → 0.2.4

package/README.md

@@ -1,16 +1,21 @@
 # 📄 readline-pager
 
-Memory-efficient, paginated file reader for Node.js with async iteration, prefetching, backward reading, and optional worker support.
+<p align="center"><img src="logo.webp" alt="logo" width="349"></p>
+
+Memory-efficient, paginated file reader for Node.js with async iteration, prefetching, backward reading and optional worker support.
 
 `readline-pager` reads large text files page-by-page without loading the entire file into memory.
 
 - ✅ Zero dependencies
-- ✅ Async iterator support (`for await...of`)
+- ✅ Async iterator (`for await...of`) + manual `next()` API
 - ✅ Forward & backward reading (EOF → BOF)
 - ✅ Optional worker thread mode (forward only)
 - ✅ Up to ~3× faster than Node.js `readline`
 - ✅ ~97% test coverage & fully typed (TypeScript)
 
+> **Important:**
+> Performance is heavily dependent on the `chunkSize` option; ensure you fine-tune it for your specific I/O hardware. A setting of **64 KB** is typically a good starting point. Increasing it might gradually improve read speeds, usually reaching an optimal peak depending on your hardware's capabilities.
+
 ---
 
 ## 📦 Installation
@@ -21,7 +26,7 @@ npm install readline-pager
 
 ---
 
-## 🚀 Quick Start
+## 🚀 Quick start
 
 ```ts
 import { createPager } from "readline-pager";
@@ -35,24 +40,25 @@ for await (const page of pager) {
 
 ---
 
-## ⚡ Manual iteration (recommended for maximum throughput)
+**Recommended for highest throughput:**
 
 ```ts
 const pager = createPager("./bigfile.txt");
 
+while (true) {
+  const page = await pager.next();
+  if (!page) break;
+}
+
+// or
 let page;
 while ((page = await pager.next()) !== null) {
-  // page: string[]
   // process page
 }
 ```
 
-`pager.next()` returns:
-
-- `Promise<string[]>` — next page
-- `Promise<null>` — end of file
-
-> Use `while + next()` when raw throughput matters (see Iteration Performance Notes).
+- `while + next()` is the fastest iteration method (avoids extra async-iterator overhead).
+- `for await of` is more ergonomic and convenient.
 
 ---
 
@@ -60,19 +66,21 @@ while ((page = await pager.next()) !== null) {
 
 ```ts
 createPager(filepath, {
+  chunkSize?: number,     // default: 64 * 1024 (64 KiB)
   pageSize?: number,      // default: 1_000
-  delimiter?: string      // default: "\n"
+  delimiter?: string,      // default: "\n"
   prefetch?: number,      // default: 1
   backward?: boolean,     // default: false
   useWorker?: boolean,    // default: false (forward only)
 });
 ```
 
+- `chunkSize`: number of bytes read per I/O operation. **Tune this** — default is `64 * 1024`.
 - `pageSize` — number of lines per page.
 - `delimiter` — line separator.
-- `prefetch` — max number of pages buffered internally. Higher values increase throughput but use more memory.
+- `prefetch` — max number of pages buffered internally. Not required for typical use; tuning has little effect once the engine is optimized.
 - `backward` — read file from end → start (not supported with `useWorker`).
-- `useWorker` — offload parsing to a worker thread (forward mode only).
+- `useWorker` — offload parsing to a worker thread (forward only).
 
 ---
 
@@ -87,23 +95,15 @@ Returns the next page or `null` when finished. Empty lines are preserved.
 - A completely empty file (`0` bytes) produces `[""]` on the first read.
 - A file with multiple empty lines returns each line as an empty string (e.g., `["", ""]` for two empty lines). Node.js `readline` may emit fewer or no `line` events in these cases.
 
-✅ Key points:
-
-- A 0-byte file → `[""]`
-- Consecutive `\n\n` → `["", ""]`
-- Node.js `readline` may skip initial empty line(s) and emit nothing for empty files.
-
 ### `pager.close(): void`
 
 Stops reading and releases resources immediately. Safe to call at any time.
 
-### Properties
+### Read-only properties
 
-```ts
-pager.lineCount; // total lines emitted so far
-pager.firstLine; // first line emitted (available after first read)
-pager.lastLine; // last line emitted (updated per page)
-```
+- `pager.lineCount` — lines emitted so far
+- `pager.firstLine` — first emitted line (available after first read)
+- `pager.lastLine` — last emitted line (updated per page)
 
 ---
 
@@ -115,49 +115,26 @@ Run the included benchmark:
 # default run
 node test/_benchmark.ts
 
-# customize
-node test/_benchmark.ts --lines=20000 --page-size=500 --prefetch=4 --backward
+# or customize with args
+node test/_benchmark.ts --lines=20000 --page-size=500 --backward
 ```
 
-Benchmarks were executed on a high-end Linux machine (SSD + fast CPU) using generated files.
-
-### Summary (averages)
-
-| Lines       | File Size (MB) | Implementation | Avg Time (ms) | Avg Throughput (MB/s) | Speedup vs `readline` |
-| ----------- | -------------- | -------------- | ------------- | --------------------: | --------------------: |
-| 1,000,000   | 35.29 MB       | readline       | 100.21        |                352.31 |                     — |
-| 1,000,000   | 35.29 MB       | readline-pager | 43.31         |                815.71 |      **2.32× faster** |
-| 10,000,000  | 352.86 MB      | readline       | 802.61        |                439.80 |                     — |
-| 10,000,000  | 352.86 MB      | readline-pager | 292.33        |               1207.77 |      **2.75× faster** |
-| 100,000,000 | 3528.59 MB     | readline       | 7777.52       |                453.75 |                     — |
-| 100,000,000 | 3528.59 MB     | readline-pager | 2742.99       |               1286.50 |      **2.83× faster** |
-
-**Key takeaways**
-
-- `readline-pager` is consistently **~2.3×–2.8× faster** than Node.js `readline`.
-- Relative performance gains increase with file size.
-- Sustained throughput exceeds **1.2 GB/s** on large files (machine-dependent).
-
----
-
-## 🧠 Iteration Performance Notes
-
-- **Fastest**: manual
-  `while ((page = await pager.next()) !== null) { ... }`
-  Avoids async-iterator protocol overhead and microtask churn.
+> Test setup: generated text files with uuid, run on a fast NVMe machine with default options; values are averages from multiple runs. Results are machine-dependent.
 
-- **More ergonomic**:
-  `for await (const page of pager) { ... }`
-  Cleaner, but slightly slower in hot paths.
+|  Lines |  File MB | Node `readline` (MB/s) | Bun streaming (MB/s) | `readline-pager` (Node) (MB/s) |
+| -----: | -------: | ---------------------: | -------------------: | -----------------------------: |
+|    10M |   352.86 |                   ~423 |                 ~296 |                     **~1,327** |
+|   100M |  3528.59 |                   ~441 |                 ~298 |                     **~1,378** |
+| 1,000M | 35285.95 |                   ~426 |                 ~294 |                     **~1,168** |
 
-**Recommendation:** use the explicit `next()` loop for benchmarks and performance-critical workloads; use `for await...of` for clarity elsewhere.
+**Takeaway:** `readline-pager` delivers multi-GB/s memory-to-memory throughput on large files on typical NVMe hardware; results vary with `chunkSize`, runtime (Node vs Bun), and CPU/OS.
 
 ---
 
 ## 🛠 Development & Contributing
 
-- Minimum supported Node.js: **18.12+** (LTS).
-- Development/test environment: **Node v25.6.1**, TypeScript `~5.9.x`.
+- Minimum supported Node.js: **v18.12** (lts/hydrogen).
+- Development/test environment: **Node v25.6**, **TypeScript v5.9**.
 
 Run tests:
 

package/dist/main.cjs

@@ -1,5 +1,4 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: 'Module' });
-const require_constants = require('./constants-Cyb_UQAC.cjs');
 let node_fs_promises = require("node:fs/promises");
 let node_worker_threads = require("node:worker_threads");
 
@@ -32,7 +31,7 @@ function createPageQueue() {
 //#endregion
 //#region src/reader/backward.reader.ts
 function createBackwardReader(filepath, options) {
-	const { pageSize, delimiter, prefetch } = options;
+	const { chunkSize, pageSize, delimiter, prefetch } = options;
 	const pageQueue = createPageQueue();
 	let fd = null;
 	let pos = 0;
@@ -54,11 +53,11 @@ function createBackwardReader(filepath, options) {
 		await init();
 		if (!fd) return;
 		while (pageQueue.queue.length < prefetch && pos > 0) {
-			const readSize = Math.min(require_constants.CHUNK_SIZE, pos);
+			const readSize = Math.min(chunkSize, pos);
 			pos -= readSize;
 			const buf = Buffer.allocUnsafe(readSize);
 			await fd.read(buf, 0, readSize, pos);
-			buffer += buf.toString(require_constants.ENCODING);
+			buffer += buf.toString("utf8");
 			let idx;
 			while ((idx = buffer.lastIndexOf(delimiter)) !== -1) {
 				const line = buffer.slice(idx + delimiter.length);
@@ -126,7 +125,7 @@ function createBackwardReader(filepath, options) {
 //#endregion
 //#region src/reader/forward.reader.ts
 function createForwardReader(filepath, options) {
-	const { pageSize, delimiter, prefetch } = options;
+	const { chunkSize, pageSize, delimiter, prefetch } = options;
 	const pageQueue = createPageQueue();
 	let fd = null;
 	let pos = 0;
@@ -149,11 +148,11 @@ function createForwardReader(filepath, options) {
 		await init();
 		if (!fd) return;
 		while (pageQueue.queue.length < prefetch && pos < size) {
-			const readSize = Math.min(require_constants.CHUNK_SIZE, size - pos);
+			const readSize = Math.min(chunkSize, size - pos);
 			const buf = Buffer.allocUnsafe(readSize);
 			const { bytesRead } = await fd.read(buf, 0, readSize, pos);
 			pos += bytesRead;
-			buffer += buf.toString(require_constants.ENCODING, 0, bytesRead);
+			buffer += buf.toString("utf8", 0, bytesRead);
 			let idx;
 			while ((idx = buffer.indexOf(delimiter)) !== -1) {
 				const line = buffer.slice(0, idx);
@@ -216,9 +215,10 @@ function createForwardReader(filepath, options) {
 //#region src/reader/worker.reader.ts
 const workerFile = typeof {} !== "undefined" ? new URL("./worker.mjs", require("url").pathToFileURL(__filename).href) : require.resolve("./worker.cjs");
 function createWorkerReader(filepath, options) {
-	const { pageSize, delimiter, prefetch } = options;
+	const { chunkSize, pageSize, delimiter, prefetch } = options;
 	const worker = new node_worker_threads.Worker(new URL(workerFile, require("url").pathToFileURL(__filename).href), { workerData: {
 		filepath,
+		chunkSize,
 		pageSize,
 		delimiter
 	} });
@@ -274,20 +274,23 @@ function createWorkerReader(filepath, options) {
 //#endregion
 //#region src/main.ts
 function createPager(filepath, options = {}) {
-	const { pageSize = 1e3, delimiter = "\n", prefetch = 1, backward = false, useWorker = false } = options;
+	const { chunkSize = 64 * 1024, pageSize = 1e3, delimiter = "\n", prefetch = 1, backward = false, useWorker = false } = options;
 	if (!filepath) throw new Error("filepath required");
 	if (pageSize <= 0) throw new RangeError("pageSize must be > 0");
 	if (prefetch <= 0) throw new RangeError("prefetch must be >= 1");
 	if (backward && useWorker) throw new Error("backward not supported with useWorker");
 	return useWorker ? createWorkerReader(filepath, {
+		chunkSize,
 		pageSize,
 		prefetch,
 		delimiter
 	}) : backward ? createBackwardReader(filepath, {
+		chunkSize,
 		pageSize,
 		prefetch,
 		delimiter
 	}) : createForwardReader(filepath, {
+		chunkSize,
 		pageSize,
 		prefetch,
 		delimiter

package/dist/main.d.cts

@@ -1,5 +1,6 @@
 //#region src/types.d.ts
 interface ReaderOptions {
+  chunkSize: number;
   pageSize: number;
   delimiter: string;
   prefetch: number;

package/dist/main.d.mts

@@ -1,5 +1,6 @@
 //#region src/types.d.ts
 interface ReaderOptions {
+  chunkSize: number;
   pageSize: number;
   delimiter: string;
   prefetch: number;

package/dist/main.mjs

@@ -1,4 +1,3 @@
-import { n as ENCODING, t as CHUNK_SIZE } from "./constants-BNKQoOqH.mjs";
 import { createRequire } from "node:module";
 import { open } from "node:fs/promises";
 import { Worker } from "node:worker_threads";
@@ -36,7 +35,7 @@ function createPageQueue() {
 //#endregion
 //#region src/reader/backward.reader.ts
 function createBackwardReader(filepath, options) {
-	const { pageSize, delimiter, prefetch } = options;
+	const { chunkSize, pageSize, delimiter, prefetch } = options;
 	const pageQueue = createPageQueue();
 	let fd = null;
 	let pos = 0;
@@ -58,11 +57,11 @@ function createBackwardReader(filepath, options) {
 		await init();
 		if (!fd) return;
 		while (pageQueue.queue.length < prefetch && pos > 0) {
-			const readSize = Math.min(CHUNK_SIZE, pos);
+			const readSize = Math.min(chunkSize, pos);
 			pos -= readSize;
 			const buf = Buffer.allocUnsafe(readSize);
 			await fd.read(buf, 0, readSize, pos);
-			buffer += buf.toString(ENCODING);
+			buffer += buf.toString("utf8");
 			let idx;
 			while ((idx = buffer.lastIndexOf(delimiter)) !== -1) {
 				const line = buffer.slice(idx + delimiter.length);
@@ -130,7 +129,7 @@ function createBackwardReader(filepath, options) {
 //#endregion
 //#region src/reader/forward.reader.ts
 function createForwardReader(filepath, options) {
-	const { pageSize, delimiter, prefetch } = options;
+	const { chunkSize, pageSize, delimiter, prefetch } = options;
 	const pageQueue = createPageQueue();
 	let fd = null;
 	let pos = 0;
@@ -153,11 +152,11 @@ function createForwardReader(filepath, options) {
 		await init();
 		if (!fd) return;
 		while (pageQueue.queue.length < prefetch && pos < size) {
-			const readSize = Math.min(CHUNK_SIZE, size - pos);
+			const readSize = Math.min(chunkSize, size - pos);
 			const buf = Buffer.allocUnsafe(readSize);
 			const { bytesRead } = await fd.read(buf, 0, readSize, pos);
 			pos += bytesRead;
-			buffer += buf.toString(ENCODING, 0, bytesRead);
+			buffer += buf.toString("utf8", 0, bytesRead);
 			let idx;
 			while ((idx = buffer.indexOf(delimiter)) !== -1) {
 				const line = buffer.slice(0, idx);
@@ -220,9 +219,10 @@ function createForwardReader(filepath, options) {
 //#region src/reader/worker.reader.ts
 const workerFile = typeof import.meta !== "undefined" ? new URL("./worker.mjs", import.meta.url) : __require.resolve("./worker.cjs");
 function createWorkerReader(filepath, options) {
-	const { pageSize, delimiter, prefetch } = options;
+	const { chunkSize, pageSize, delimiter, prefetch } = options;
 	const worker = new Worker(new URL(workerFile, import.meta.url), { workerData: {
 		filepath,
+		chunkSize,
 		pageSize,
 		delimiter
 	} });
@@ -278,20 +278,23 @@ function createWorkerReader(filepath, options) {
 //#endregion
 //#region src/main.ts
 function createPager(filepath, options = {}) {
-	const { pageSize = 1e3, delimiter = "\n", prefetch = 1, backward = false, useWorker = false } = options;
+	const { chunkSize = 64 * 1024, pageSize = 1e3, delimiter = "\n", prefetch = 1, backward = false, useWorker = false } = options;
 	if (!filepath) throw new Error("filepath required");
 	if (pageSize <= 0) throw new RangeError("pageSize must be > 0");
 	if (prefetch <= 0) throw new RangeError("prefetch must be >= 1");
 	if (backward && useWorker) throw new Error("backward not supported with useWorker");
 	return useWorker ? createWorkerReader(filepath, {
+		chunkSize,
 		pageSize,
 		prefetch,
 		delimiter
 	}) : backward ? createBackwardReader(filepath, {
+		chunkSize,
 		pageSize,
 		prefetch,
 		delimiter
 	}) : createForwardReader(filepath, {
+		chunkSize,
 		pageSize,
 		prefetch,
 		delimiter

package/dist/worker.cjs

@@ -1,9 +1,8 @@
-const require_constants = require('./constants-Cyb_UQAC.cjs');
 let node_fs_promises = require("node:fs/promises");
 let node_worker_threads = require("node:worker_threads");
 
 //#region src/worker.ts
-const { filepath, pageSize, delimiter } = node_worker_threads.workerData;
+const { filepath, chunkSize, pageSize, delimiter } = node_worker_threads.workerData;
 (async () => {
 	const fd = await (0, node_fs_promises.open)(filepath, "r");
 	const { size } = await fd.stat();
@@ -11,7 +10,7 @@ const { filepath, pageSize, delimiter } = node_worker_threads.workerData;
 	let buffer = "";
 	const local = [];
 	while (pos < size) {
-		const readSize = Math.min(require_constants.CHUNK_SIZE, size - pos);
+		const readSize = Math.min(chunkSize, size - pos);
 		const buf = Buffer.allocUnsafe(readSize);
 		const { bytesRead } = await fd.read(buf, 0, readSize, pos);
 		pos += bytesRead;

package/dist/worker.mjs

@@ -1,9 +1,8 @@
-import { t as CHUNK_SIZE } from "./constants-BNKQoOqH.mjs";
 import { open } from "node:fs/promises";
 import { parentPort, workerData } from "node:worker_threads";
 
 //#region src/worker.ts
-const { filepath, pageSize, delimiter } = workerData;
+const { filepath, chunkSize, pageSize, delimiter } = workerData;
 (async () => {
 	const fd = await open(filepath, "r");
 	const { size } = await fd.stat();
@@ -11,7 +10,7 @@ const { filepath, pageSize, delimiter } = workerData;
 	let buffer = "";
 	const local = [];
 	while (pos < size) {
-		const readSize = Math.min(CHUNK_SIZE, size - pos);
+		const readSize = Math.min(chunkSize, size - pos);
 		const buf = Buffer.allocUnsafe(readSize);
 		const { bytesRead } = await fd.read(buf, 0, readSize, pos);
 		pos += bytesRead;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "readline-pager",
-  "version": "0.2.2",
+  "version": "0.2.4",
   "scripts": {
     "build": "tsdown",
     "pretest": "npm run build",
@@ -8,6 +8,7 @@
     "prepublishOnly": "npm run build && npm run test"
   },
   "devDependencies": {
+    "@types/bun": "~1.3.9",
     "@types/node": "~25.3.0",
     "prettier": "~3.8.1",
     "prettier-plugin-organize-imports": "~4.3.0",

package/dist/constants-BNKQoOqH.mjs

@@ -1,6 +0,0 @@
-//#region src/constants.ts
-const CHUNK_SIZE = 64 * 1024;
-const ENCODING = "utf8";
-
-//#endregion
-export { ENCODING as n, CHUNK_SIZE as t };

package/dist/constants-Cyb_UQAC.cjs

@@ -1,18 +0,0 @@
-
-//#region src/constants.ts
-const CHUNK_SIZE = 64 * 1024;
-const ENCODING = "utf8";
-
-//#endregion
-Object.defineProperty(exports, 'CHUNK_SIZE', {
-  enumerable: true,
-  get: function () {
-    return CHUNK_SIZE;
-  }
-});
-Object.defineProperty(exports, 'ENCODING', {
-  enumerable: true,
-  get: function () {
-    return ENCODING;
-  }
-});