@thi.ng/block-fs 0.3.0 → 0.4.1

package/CHANGELOG.md

@@ -1,6 +1,6 @@
 # Change Log
 
-- **Last updated**: 2025-04-02T18:47:59Z
+- **Last updated**: 2025-04-16T11:11:14Z
 - **Generator**: [thi.ng/monopub](https://thi.ng/monopub)
 
 All notable changes to this project will be documented in this file.
@@ -11,6 +11,27 @@ See [Conventional Commits](https://conventionalcommits.org/) for commit guidelin
 **Note:** Unlisted _patch_ versions only involve non-code or otherwise excluded changes
 and/or version bumps of transitive dependencies.
 
+### [0.4.1](https://github.com/thi-ng/umbrella/tree/@thi.ng/block-fs@0.4.1) (2025-04-16)
+
+#### ♻️ Refactoring
+
+- update Entry memory layout ([b5416bd](https://github.com/thi-ng/umbrella/commit/b5416bd))
+  - move block start & end ID locations
+
+## [0.4.0](https://github.com/thi-ng/umbrella/tree/@thi.ng/block-fs@0.4.0) (2025-04-06)
+
+#### 🚀 Features
+
+- add support for wrapping `ArrayBuffer` ([e23f008](https://github.com/thi-ng/umbrella/commit/e23f008))
+  - update `MemoryBlockStorageOpts.buffer` to allow array buffers
+  - update `MemoryBlockStorage` ctor
+- auto-infer MIME type in `.readAsObjectURL()` ([8fbcebd](https://github.com/thi-ng/umbrella/commit/8fbcebd))
+  - use `preferredTypeForPath()` as MIME type fallback
+  - update deps
+- update CLI, add include/exclude regexp, logging ([ef04e09](https://github.com/thi-ng/umbrella/commit/ef04e09))
+  - add support for multiple include/exclude regexps in `convert` command
+  - add `--quiet` flag to disable logging
+
 ## [0.3.0](https://github.com/thi-ng/umbrella/tree/@thi.ng/block-fs@0.3.0) (2025-04-02)
 
 #### 🚀 Features

package/README.md

@@ -28,6 +28,8 @@
 - [Installation](#installation)
 - [Dependencies](#dependencies)
 - [API](#api)
+  - [Basic usage](#basic-usage)
+  - [Working with a converted file system blob](#working-with-a-converted-file-system-blob)
 - [Authors](#authors)
 - [License](#license)
 
@@ -94,16 +96,33 @@ entries/sizes. The default [`Entry`
 implementation](https://docs.thi.ng/umbrella/block-fs/classes/Entry.html)
 requires 64 bytes.
 
-TODO diagram
+![Memory layout diagram for a single directory entry](https://raw.githubusercontent.com/thi-ng/umbrella/develop/assets/block-fs/direntry-01.png)
 
 #### File blocks
 
 Files are stored as linked lists of blocks, with the first few bytes of each
-block reserved for linkage and number of data bytes in the block. The number of
-bytes effectively available for data depends on the configured block size and
-the max. number of blocks in the storage backend.
+block reserved for linkage and number of data bytes in the block.
 
-TODO diagram
+![Memory layout diagram for a single file block](https://raw.githubusercontent.com/thi-ng/umbrella/develop/assets/block-fs/block-layout-01.png)
+
+The number of bytes effectively available for data depends on the configured
+block size and the max. number of blocks in the storage backend. For example, a
+max. block count of 65536 and a block size of 256 bytes only requires a two
+bytes for linkage and a third byte for storing the number of data bytes used in
+the block. Hence, in this configuration 253 bytes per block are available for
+data.
+
+The following diagram shows a block which links to block ID 0x1234 and uses the
+full 253 (0xfd in hex) bytes of data available:
+
+![Memory layout diagram for a single file block](https://raw.githubusercontent.com/thi-ng/umbrella/develop/assets/block-fs/block-layout-02.png)
+
+The last block of a file uses a special sentinel marker to indicate that no
+other blocks follow. This sentinel value again depends on the configured max.
+block count, and in this example is 0xffff. This example block only stores 64
+(0x40 in hex) bytes of data, with the remainder zeroed out.
+
+![Memory layout diagram for a sentinel file block](https://raw.githubusercontent.com/thi-ng/umbrella/develop/assets/block-fs/block-layout-03.png)
 
 ### Command line app
 
@@ -119,7 +138,8 @@ Once bundled, the binary blob can then be used together with
 [`MemoryBlockStorage`](https://docs.thi.ng/umbrella/block-fs/classes/MemoryBlockStorage.html)
 and [`BlockFS`](https://docs.thi.ng/umbrella/block-fs/classes/BlockFS.html) for
 other purposes (e.g. distributed with your web app to provide a virtual
-filesystem).
+filesystem). Also see [API example further
+below](#working-with-a-converted-file-system-blob).
 
 Example usage to bundle the source directory of this package:
 
@@ -138,15 +158,31 @@ General usage:
 ```text
 npx @thi.ng/block-fs convert --help
 
-Usage: blockfs <cmd> [opts] input
+ █ █   █           │
+██ █               │
+ █ █ █ █   █ █ █ █ │ @thi.ng/block-fs 0.4.0
+ █ █ █ █ █ █ █ █ █ │ Block-based storage & file system layer
+                 █ │
+               █ █ │
+
+Usage: blockfs <cmd> [opts] input [...]
+       blockfs <cmd> --help
+
+Available commands:
+
+convert         : Convert file tree into single BlockFS blob
+list            : List file tree of a BlockFS blob
 
 Flags:
 
--v, --verbose                   Display extra process information
+-q, --quiet                     Disable logging
+-v, --verbose                   Display extra logging information
 
 Main:
 
 -bs BYTES, --block-size BYTES   Block size (default: 1024)
+-i EXT, --exclude EXT           [multiple] File exclusion regexp
+-i EXT, --include EXT           [multiple] File inclusion regexp
 -n INT, --num-blocks INT        Number of blocks (multiple of 8)
 -o STR, --out STR               [required] Output file path
 ```
@@ -252,7 +288,7 @@ For Node.js REPL:
 const bf = await import("@thi.ng/block-fs");
 ```
 
-Package sizes (brotli'd, pre-treeshake): ESM: 4.43 KB
+Package sizes (brotli'd, pre-treeshake): ESM: 4.50 KB
 
 ## Dependencies
 
@@ -264,6 +300,7 @@ Package sizes (brotli'd, pre-treeshake): ESM: 4.43 KB
 - [@thi.ng/errors](https://github.com/thi-ng/umbrella/tree/develop/packages/errors)
 - [@thi.ng/file-io](https://github.com/thi-ng/umbrella/tree/develop/packages/file-io)
 - [@thi.ng/logger](https://github.com/thi-ng/umbrella/tree/develop/packages/logger)
+- [@thi.ng/mime](https://github.com/thi-ng/umbrella/tree/develop/packages/mime)
 - [@thi.ng/random](https://github.com/thi-ng/umbrella/tree/develop/packages/random)
 - [@thi.ng/strings](https://github.com/thi-ng/umbrella/tree/develop/packages/strings)
 
@@ -273,7 +310,9 @@ Note: @thi.ng/api is in _most_ cases a type-only import (not used at runtime)
 
 [Generated API docs](https://docs.thi.ng/umbrella/block-fs/)
 
-```ts tangle:export/readme.ts
+### Basic usage
+
+```ts tangle:export/readme-1.ts
 import { BlockFS, MemoryBlockStorage } from "@thi.ng/block-fs";
 
 // create in-memory storage (64KB)
@@ -335,6 +374,45 @@ for await (let entry of fs.root.tree()) {
 // /deeply/nested/paths/are-ok 4n 2025-04-01T20:18:55.919Z
 ```
 
+### Working with a converted file system blob
+
+This example shows how to use a binary blob created via the [CLI `blockfs
+convert` command](#convert-file-tree-into-single-blockfs-blob) as a virtual file
+system...
+
+```ts tangle:export/readme-2.ts
+import { BlockFS, MemoryBlockStorage } from "@thi.ng/block-fs";
+
+// load binary blob
+const response = await fetch("./blocks.dat");
+const buffer = await response.arrayBuffer();
+
+// wrap as block storage
+const storage = new MemoryBlockStorage({
+    buffer,
+    blockSize: 1024,
+    numBlocks: buffer.byteLength / 1024
+});
+
+// wrap as file system
+const fs = new BlockFS(storage);
+
+// list all entries (recursive)
+for await(let f of fs.root.tree()) {
+    console.log(f.path);
+}
+
+// list all entries in a directory
+const dir = (await fs.entryForPath("/path/to/dir")).directory;
+for await (let f of dir) {
+    console.log(f.path);
+}
+
+// load an image as blob URL (MIME type is inferred automatically)
+const img = new Image();
+img.src = await fs.readAsObjectURL("/assets/test.jpg");
+```
+
 ## Authors
 
 - [Karsten Schmidt](https://thi.ng)

package/cli.d.ts

@@ -1,10 +1,12 @@
 import { type Command, type CommandCtx } from "@thi.ng/args";
 interface CLIOpts {
     verbose: boolean;
+    quiet: boolean;
 }
 interface ConvertOpts extends CLIOpts {
     numBlocks?: number;
     blockSize: number;
+    exclude?: string[];
     include?: string[];
     out: string;
 }

package/cli.js

@@ -28,25 +28,30 @@ const ARG_BLOCKSIZE = {
     }
   })
 };
-const collectFiles = (ctx) => {
-  const root = resolve(ctx.inputs[0]);
+const collectFiles = ({
+  opts: { include, exclude },
+  inputs
+}) => {
+  const root = resolve(inputs[0]);
   const filtered = [];
   const dirs = /* @__PURE__ */ new Set();
+  const $include = include?.map((x) => new RegExp(x));
+  const $exclude = exclude?.map((x) => new RegExp(x));
   let total = 0;
   for (let f of files(root)) {
     const stats = statSync(f);
-    if (stats.isFile()) {
-      const dest = relative(root, f);
-      filtered.push({
-        src: f,
-        dest,
-        size: stats.size,
-        ctime: stats.ctimeMs,
-        mtime: stats.mtimeMs
-      });
-      dirs.add(dirname(dest));
-      total += stats.size;
-    }
+    if ($exclude && $exclude.some((x) => x.test(f))) continue;
+    if ($include && !$include.some((x) => x.test(f))) continue;
+    const dest = relative(root, f);
+    filtered.push({
+      src: f,
+      dest,
+      size: stats.size,
+      ctime: stats.ctimeMs,
+      mtime: stats.mtimeMs
+    });
+    dirs.add(dirname(dest));
+    total += stats.size;
   }
   return { files: filtered, dirs: [...dirs], size: total };
 };
@@ -77,9 +82,14 @@ const CONVERT = {
       desc: "Output file path",
       optional: false
     }),
+    exclude: strings({
+      alias: "e",
+      desc: "File exclusion regexp",
+      hint: "EXT"
+    }),
     include: strings({
       alias: "i",
-      desc: "Only include file extensions",
+      desc: "File inclusion regexp",
       hint: "EXT"
     })
   },
@@ -102,7 +112,7 @@ const CONVERT = {
     const bfs = new BlockFS(storage, { logger: ctx.logger });
     await bfs.init();
     for (let f of collected.files) {
-      ctx.logger.debug("writing file:", f.dest);
+      ctx.logger.info("writing file:", f.dest);
       await bfs.writeFile(f.dest, readBinary(f.src));
       const entry = await bfs.entryForPath(f.dest);
       entry.ctime = f.ctime;
@@ -179,7 +189,11 @@ cliApp({
   opts: {
     verbose: flag({
       alias: "v",
-      desc: "Display extra process information"
+      desc: "Display extra logging information"
+    }),
+    quiet: flag({
+      alias: "q",
+      desc: "Disable logging"
     })
   },
   commands: {
@@ -188,7 +202,8 @@ cliApp({
   },
   name: "blockfs",
   ctx: async (ctx) => {
-    if (ctx.opts.verbose) ctx.logger.level = LogLevel.DEBUG;
+    if (ctx.opts.quiet) ctx.logger.level = LogLevel.NONE;
+    else if (ctx.opts.verbose) ctx.logger.level = LogLevel.DEBUG;
     return ctx;
   },
   start: 3,

package/entry.d.ts

@@ -20,16 +20,16 @@ export declare class Entry implements IEntry {
     set owner(owner: number);
     get name(): string;
     set name(name: string);
+    get start(): number;
+    set start(block: number);
+    get end(): number;
+    set end(block: number);
     get size(): bigint;
     set size(size: bigint);
     get ctime(): number;
     set ctime(epoch: number);
     get mtime(): number;
     set mtime(epoch: number);
-    get start(): number;
-    set start(block: number);
-    get end(): number;
-    set end(block: number);
     set(spec: EntrySpec): void;
     release(): void;
     save(): Promise<void>;

package/entry.js

@@ -66,35 +66,35 @@ class Entry {
       this.data.subarray(offset, offset + Entry.NAME_MAX_LENGTH)
     );
   }
+  get start() {
+    return this.view.getUint32(32, true);
+  }
+  set start(block) {
+    this.view.setUint32(32, block, true);
+  }
+  get end() {
+    return this.view.getUint32(36, true);
+  }
+  set end(block) {
+    this.view.setUint32(36, block, true);
+  }
   get size() {
-    return this.view.getBigUint64(32, true);
+    return this.view.getBigUint64(40, true);
   }
   set size(size) {
-    this.view.setBigUint64(32, size, true);
+    this.view.setBigUint64(40, size, true);
   }
   get ctime() {
-    return Number(this.view.getBigUint64(40, true));
+    return Number(this.view.getBigUint64(48, true));
   }
   set ctime(epoch) {
-    this.view.setBigUint64(40, BigInt(epoch), true);
+    this.view.setBigUint64(48, BigInt(epoch), true);
   }
   get mtime() {
-    return Number(this.view.getBigUint64(48, true));
+    return Number(this.view.getBigUint64(56, true));
   }
   set mtime(epoch) {
-    this.view.setBigUint64(48, BigInt(epoch), true);
-  }
-  get start() {
-    return this.view.getUint32(56, true);
-  }
-  set start(block) {
-    this.view.setUint32(56, block, true);
-  }
-  get end() {
-    return this.view.getUint32(60, true);
-  }
-  set end(block) {
-    this.view.setUint32(60, block, true);
+    this.view.setBigUint64(56, BigInt(epoch), true);
   }
   set(spec) {
     this.type = spec.type;

package/fs.d.ts

@@ -160,6 +160,10 @@ export declare class BlockFS {
      * object URL, optionally typed with given MIME type.
      *
      * @remarks
+     * If `type` is omitted, it will be attempted to be inferred automatically
+     * via [thi.ng/mime](https://thi.ng/mime).
+     *
+     * @remarks
      * Reference:
      *
      * - https://developer.mozilla.org/en-US/docs/Web/API/Blob#creating_a_url_representing_the_contents_of_a_typed_array

package/fs.js

@@ -5,6 +5,7 @@ import { assert } from "@thi.ng/errors/assert";
 import { illegalArgs } from "@thi.ng/errors/illegal-arguments";
 import { illegalState } from "@thi.ng/errors/illegal-state";
 import { NULL_LOGGER } from "@thi.ng/logger/null";
+import { preferredTypeForPath } from "@thi.ng/mime";
 import {
   EntryType
 } from "./api.js";
@@ -289,6 +290,10 @@ class BlockFS {
    * object URL, optionally typed with given MIME type.
    *
    * @remarks
+   * If `type` is omitted, it will be attempted to be inferred automatically
+   * via [thi.ng/mime](https://thi.ng/mime).
+   *
+   * @remarks
    * Reference:
    *
    * - https://developer.mozilla.org/en-US/docs/Web/API/Blob#creating_a_url_representing_the_contents_of_a_typed_array
@@ -298,7 +303,9 @@ class BlockFS {
    */
   async readAsObjectURL(path, type) {
     return URL.createObjectURL(
-      new Blob([await this.readFile(path)], { type })
+      new Blob([await this.readFile(path)], {
+        type: type ?? (isString(path) ? preferredTypeForPath(path) : void 0)
+      })
     );
   }
   /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@thi.ng/block-fs",
-  "version": "0.3.0",
+  "version": "0.4.1",
   "description": "Customizable block-based storage, adapters & file system layer",
   "type": "module",
   "module": "./index.js",
@@ -42,32 +42,35 @@
     "tool:tangle": "../../node_modules/.bin/tangle src/**/*.ts"
   },
   "dependencies": {
-    "@thi.ng/api": "^8.11.25",
-    "@thi.ng/args": "^2.3.66",
-    "@thi.ng/binary": "^3.4.48",
-    "@thi.ng/bitfield": "^2.4.0",
-    "@thi.ng/checks": "^3.7.5",
-    "@thi.ng/errors": "^2.5.31",
-    "@thi.ng/file-io": "^2.1.34",
-    "@thi.ng/logger": "^3.1.6",
-    "@thi.ng/random": "^4.1.16",
-    "@thi.ng/strings": "^3.9.10"
+    "@thi.ng/api": "^8.11.26",
+    "@thi.ng/args": "^2.3.67",
+    "@thi.ng/binary": "^3.4.49",
+    "@thi.ng/bitfield": "^2.4.1",
+    "@thi.ng/checks": "^3.7.6",
+    "@thi.ng/errors": "^2.5.32",
+    "@thi.ng/file-io": "^2.1.35",
+    "@thi.ng/logger": "^3.1.7",
+    "@thi.ng/mime": "^2.7.8",
+    "@thi.ng/random": "^4.1.17",
+    "@thi.ng/strings": "^3.9.11"
   },
   "devDependencies": {
-    "@types/node": "^22.13.14",
+    "@types/node": "^22.14.1",
     "esbuild": "^0.25.2",
-    "typedoc": "^0.28.1",
-    "typescript": "^5.8.2"
+    "typedoc": "^0.28.2",
+    "typescript": "^5.8.3"
   },
   "keywords": [
+    "async",
     "binary",
     "block",
     "cli",
     "conversion",
     "file",
-    "file-system",
+    "filesystem",
     "memory",
     "memory-mapped",
+    "mime",
     "nodejs",
     "path",
     "storage",
@@ -129,5 +132,5 @@
     "status": "alpha",
     "year": 2024
   },
-  "gitHead": "549e798e6a82254ee68727f12229c5252892b8f1\n"
+  "gitHead": "c464b6948f92cba90c2ea75b59203dad894fb450\n"
 }

package/storage/memory.d.ts

@@ -16,7 +16,7 @@ export interface MemoryBlockStorageOpts extends BlockStorageOpts {
      * Optional, pre-defined/loaded byte buffer. Must have at least `numBlocks *
      * blockSize` capacity.
      */
-    buffer?: Uint8Array;
+    buffer?: Uint8Array | ArrayBufferLike;
 }
 export declare class MemoryBlockStorage extends ABlockStorage<MemoryBlock> {
     buffer: Uint8Array;

package/storage/memory.js

@@ -1,3 +1,4 @@
+import { isArrayBufferLike } from "@thi.ng/checks/is-arraybufferlike";
 import { illegalArgs } from "@thi.ng/errors/illegal-arguments";
 import { ABlockStorage } from "./astorage.js";
 class MemoryBlock {
@@ -27,12 +28,13 @@ class MemoryBlockStorage extends ABlockStorage {
   constructor(opts) {
     super(opts);
     const size = this.numBlocks * this.blockSize;
-    if (opts.buffer && opts.buffer.length < size) {
+    const buffer = opts.buffer ? isArrayBufferLike(opts.buffer) ? new Uint8Array(opts.buffer) : opts.buffer : void 0;
+    if (buffer && buffer.length < size) {
       illegalArgs(
         `given buffer is too small, expected at least ${size} bytes`
       );
     }
-    this.buffer = opts.buffer ?? new Uint8Array(size);
+    this.buffer = buffer ?? new Uint8Array(size);
   }
   async hasBlock(id) {
     this.ensureValidID(id);