facetful 0.4.0 → 0.5.0

package/README.md

@@ -25,6 +25,22 @@ browser** — facet counts, pivots, top-k, filters — at interaction speed.
 
 ## Quickstart
 
+One install gives the browser library **and** the `facetful` command:
+
+```
+npm install facetful
+npx facetful convert data.csv data.facetful        # streaming: any file size, ~20 MB of memory
+npx facetful query data.facetful "select country, count(*) as n from t group by country order by n desc limit 5"
+```
+
+The command runs the same wasm engine the browser does (Node is the second
+runtime), so a `.facetful` built here is exactly what `openParquet` would have
+built, and SQL — including your registered functions, via `--udf module.mjs` —
+behaves identically in both places. CSV can also be opened directly in the
+browser with `loadCsv` (below). The ready-made functions (`json_extract`,
+`to_tz`, `date_trunc`, …, see "User-defined functions") are registered by
+default in both.
+
 ```js
 import { Facetful } from "facetful";
 
@@ -53,13 +69,31 @@ The SQL table is always named `t`. Multiple tables coexist:
 | method | use when |
 |---|---|
 | `openParquet(name, buf)` | the default: Parquet in, OPFS-cached compiled image, instant repeat visits |
-| `load(name, buf)` | you already have a `.facetful` image (built by the native CLI) |
+| `load(name, buf)` | you already have a `.facetful` image (built by `facetful convert`) |
+| `loadCsv(name, fileOrBuffer, {persist})` | a CSV: streamed through the converter in the worker, types inferred, optionally persisted |
 | `loadOpfs(name, path, {cacheBytes})` | the file is in OPFS; open lazily, spill-over for big data |
 
 Persistence is always explicit (`storeOpfs`), never write-behind. OPFS needs a
 secure context (https or localhost); everything degrades to memory-only
 without one.
 
+## The `facetful` command
+
+```
+facetful convert in.csv out.facetful [--row-group-size N]
+facetful query file.facetful ["select …"] [--table name=other.facetful …] [--udf module.mjs …]
+facetful materialize in.facetful "select …" out.facetful [--table …] [--udf …]
+```
+
+`convert` streams: two passes over the CSV (types and dictionaries, then
+encoding), row groups written as they finish, memory bounded by the distinct
+values plus one row group — a 200 MB / 3M-row CSV converts in ~4 s at ~70 MB
+of process memory (Node included). Types: int (narrowed), float, ISO date and
+datetime, text; repeated text is dictionary-encoded. `query` without SQL is a
+REPL; tables open lazily, so large files don't load into memory. The
+ready-made functions are registered; a `--udf` module's default export adds
+your own, as an array of `{ name, signature, fn }`.
+
 ## Derived tables: `materialize`
 
 ```js
@@ -104,6 +138,50 @@ for the join once. Correlated subqueries beyond `inner.k = outer.k` equalities,
 `NOT IN` follows SQL's NULL rule (a NULL in the set makes it select nothing);
 `NOT EXISTS` doesn't, and is usually what you mean.
 
+## User-defined functions
+
+```js
+// vectorized: called once per lane (a row group, a group table, or — for a
+// dictionary column — the dictionary itself); `out.values` is the result lane
+await db.registerFunction("regexp", { params: ["text", "text"], returns: "bool" }, (() => {
+  const cache = new Map();
+  return (args, len, out) => {
+    const [s, pattern] = args;                 // pattern is a literal: broadcast, one value
+    let re = cache.get(pattern.values[0]);
+    if (!re) cache.set(pattern.values[0], (re = new RegExp(pattern.values[0])));
+    for (let i = 0; i < len; i++) out.values[i] = re.test(s.values[i]) ? 1 : 0;
+  };
+})());
+await db.query("select fuel, count(*) from t where regexp(plant_name, '^(Big|Little) ') group by fuel");
+
+// per row: simpler, ~10x slower on large lanes
+await db.registerFunction("mw_to_gw", { params: ["float"], returns: "float", perRow: true }, (mw) => mw / 1000);
+```
+
+**Ready-made functions**, registered by default (`Facetful.open({ udfs: false })`
+opts out; the list is importable from `facetful/udfs`): `regexp(s, pattern[, flags])`,
+`regexp_extract(s, pattern[, group])`, `regexp_replace(s, pattern, replacement)`,
+`json_extract(doc, '$.a.b[0]')`, `to_tz(ts, 'Europe/London')`
+(Intl's time-zone tables — hundreds of KB the wasm never has to carry),
+`date_trunc('month', ts)`, `date_add(d, 1, 'month')`, `weekday`, `quarter`,
+`country_name('DE')`, `format_number(x, 'en-US:compact')`, `unaccent('Zürich')`,
+`url_host(url)`. Each is a few lines of ordinary JavaScript over what the
+browser already ships; they're as much a set of patterns as a library.
+
+Kinds: `int`, `float`, `bool`, `text`, `date`, `timestamp` (dates and timestamps
+arrive as days / ms numbers). Functions bind like built-ins — wrong argument
+types are caret-diagnosed, the declared return type is the column's type — and
+their results go through the same caches, so a `regexp()` filter is evaluated
+once per pattern and served from the mask cache after. `strict` (default)
+gives NULL out for NULL in without calling you; `optional: n` makes the last
+`n` parameters omittable, a parameter kind of `"any"` accepts every type, and
+`variadic` repeats the last parameter; `unregisterFunction(name)` removes one. The function runs in the
+worker: pass a self-contained function (its source is sent — an IIFE for
+state, as above — no closures over your variables) or `{ moduleUrl }`. A
+throwing function fails the query with its message. `regexp()` is the
+browser's `RegExp` here and the `regex` crate in the native CLI — the same SQL
+on both sides.
+
 ## Parquet support
 
 Reading uses [hyparquet](https://github.com/hyparam/hyparquet) (~20 KB gz),
@@ -122,7 +200,8 @@ browser transcoder (the native CLI has no such limit).
 SELECT-only, SQLite semantics (3-valued logic, null-skipping aggregates,
 truncating integer division, NULL-first ascending sorts). Idioms: `IN`,
 `BETWEEN`, `IS [NOT] NULL`, `[NOT] LIKE`, `CASE WHEN`, `CAST`,
-`COUNT(DISTINCT x)`, `||`, `select *`, `JOIN`/`WITH`/subqueries as above. Aggregates: count, sum, avg, min, max,
+`COUNT(DISTINCT x)`, `||`, `select *`, `JOIN`/`WITH`/subqueries and
+user-defined functions as above. Aggregates: count, sum, avg, min, max,
 count(distinct), median, stddev, group_concat. Scalars: math (abs, round,
 floor, ceil, sqrt, pow, exp, ln, sign), text (lower, upper, length, substr,
 trim/ltrim/rtrim, replace, instr, concat), null handling (coalesce, ifnull,

package/bin/facetful.mjs

@@ -0,0 +1,190 @@
+#!/usr/bin/env node
+// The facetful command, on the same wasm engine the browser runs (design.sv
+// d51): one `npm install facetful` gives the library and this.
+//
+//   facetful convert in.csv out.facetful [--row-group-size N]
+//   facetful query file.facetful ["select …"] [--table name=other.facetful …] [--udf module.mjs …]
+//   facetful materialize in.facetful "select …" out.facetful [--table name=path …] [--udf …]
+//
+// Conversion streams: two passes over the CSV in 1 MB chunks, row groups
+// written as they finish, memory bounded whatever the file size. Tables open
+// lazily through positional reads (the browser's OPFS import, here fs.readSync).
+// The ready-made functions (../udfs.js) are registered; a --udf module's
+// default export adds more: an array of { name, signature, fn }.
+import { readFileSync, openSync, readSync, writeSync, closeSync, fstatSync } from "node:fs";
+import { pathToFileURL } from "node:url";
+import { instantiate, QueryError } from "../core.js";
+import { udfs as builtinUdfs } from "../udfs.js";
+
+const args = process.argv.slice(2);
+const cmd = args.shift();
+const usage = () => {
+  console.error(`usage: facetful convert in.csv out.facetful [--row-group-size N]
+       facetful query file.facetful ["select …"] [--table name=path.facetful …] [--udf module.mjs …]
+       facetful materialize in.facetful "select …" out.facetful [--table name=path …] [--udf module.mjs …]`);
+  process.exit(2);
+};
+
+// positional reads for lazily opened tables: fileId -> fd
+const fds = new Map();
+let nextId = 1;
+const opfsRead = (fileId, offset, dest) => {
+  const fd = fds.get(fileId);
+  if (fd === undefined) return -1;
+  return readSync(fd, dest, 0, dest.length, offset);
+};
+const engine = await instantiate(readFileSync(new URL("../facetful_wasm.wasm", import.meta.url)), opfsRead);
+for (const u of builtinUdfs) engine.registerFunction(u.name, u.signature, u.fn);
+
+function openLazy(path) {
+  const fd = openSync(path, "r");
+  const id = nextId++;
+  fds.set(id, fd);
+  return engine.openOpfsTable(id, fstatSync(fd).size, 256 << 20);
+}
+
+async function registerUdfs(paths) {
+  for (const p of paths) {
+    const mod = await import(pathToFileURL(p).href);
+    const list = Array.isArray(mod.default) ? mod.default : Object.values(mod.default ?? mod);
+    for (const u of list) engine.registerFunction(u.name, u.signature, u.fn);
+  }
+}
+
+/** Split flags out of a positional list: --table name=path, --udf path, --row-group-size N. */
+function parse(argv) {
+  const pos = [], tables = [], udfs = [];
+  let groupSize = 65536;
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === "--table") tables.push(argv[++i]);
+    else if (a === "--udf") udfs.push(argv[++i]);
+    else if (a === "--row-group-size") groupSize = Number(argv[++i]);
+    else if (a.startsWith("--")) usage();
+    else pos.push(a);
+  }
+  return { pos, tables, udfs, groupSize };
+}
+
+const text = (c, i) => engine.dec.decode(c.bytes.subarray(c.offsets[i], c.offsets[i + 1]));
+const isoDate = (d) => new Date(d * 86400000).toISOString().slice(0, 10);
+const isoTs = (ms) => new Date(ms).toISOString().replace("T", " ").slice(0, 19);
+function cell(c, i) {
+  if (!((c.validity[i >> 3] >> (i & 7)) & 1)) return "";
+  switch (c.kind) {
+    case "text": return text(c, i);
+    case "date": return isoDate(c.values[i]);
+    case "timestamp": return isoTs(c.values[i]);
+    case "bool": return c.values[i] ? "1" : "0";
+    default: return String(c.values[i]);
+  }
+}
+function printResult(r, ms) {
+  const rows = [];
+  for (let i = 0; i < r.rowCount; i++) rows.push(r.columns.map((c) => cell(c, i)));
+  const widths = r.columns.map((c, j) => Math.max(c.name.length, ...rows.map((row) => row[j].length)));
+  const line = (cells) => cells.map((s, j) => s.padEnd(widths[j])).join("  ");
+  console.log(line(r.columns.map((c) => c.name)));
+  console.log(widths.map((w) => "-".repeat(w)).join("  "));
+  for (const row of rows) console.log(line(row));
+  console.log(`(${r.rowCount} row${r.rowCount === 1 ? "" : "s"}, ${ms.toFixed(1)} ms)`);
+}
+
+function convert(input, output, groupSize) {
+  const CHUNK = 1 << 20;
+  const h = engine.w.convert_begin(groupSize);
+  const fail = () => {
+    const n = engine.w.convert_error(h, engine.scratch, 4096);
+    console.error(engine.dec.decode(new Uint8Array(engine.mem(), engine.scratch, n)));
+    process.exit(1);
+  };
+  const out = openSync(output, "w");
+  let written = 0;
+  const drain = () => {
+    const n = engine.w.convert_output_len(h);
+    if (!n) return;
+    const p = engine.w.alloc(n);
+    engine.w.convert_output_copy(h, p);
+    writeSync(out, new Uint8Array(engine.mem(), p, n));
+    engine.w.dealloc(p, n);
+    written += n;
+  };
+  const feedFile = () => {
+    const fd = openSync(input, "r");
+    const buf = engine.w.alloc(CHUNK);
+    for (;;) {
+      // the view is rebuilt per read: memory may grow while the converter runs
+      const n = readSync(fd, new Uint8Array(engine.mem(), buf, CHUNK), 0, CHUNK, null);
+      if (n === 0) break;
+      if (engine.w.convert_feed(h, buf, n) < 0) fail();
+      drain();
+    }
+    engine.w.dealloc(buf, CHUNK);
+    closeSync(fd);
+  };
+  const t0 = performance.now();
+  feedFile();
+  if (engine.w.convert_pass2(h) < 0) fail();
+  const n = engine.w.convert_schema(h, engine.scratch, 4096);
+  const schema = engine.dec.decode(new Uint8Array(engine.mem(), engine.scratch, n)).trimEnd();
+  feedFile();
+  if (engine.w.convert_finish(h) < 0) fail();
+  drain();
+  closeSync(out);
+  const rows = engine.w.convert_rows(h);
+  engine.w.convert_free(h);
+  console.error(`${input}: ${rows} rows`);
+  for (const l of schema.split("\n")) console.error("  " + l.replace("\t", ": "));
+  console.error(`${output}: ${written} bytes (${Math.ceil(rows / groupSize)} row groups, ${(performance.now() - t0).toFixed(0)} ms)`);
+}
+
+try {
+  if (cmd === "convert") {
+    const { pos, groupSize } = parse(args);
+    if (pos.length !== 2) usage();
+    convert(pos[0], pos[1], groupSize);
+  } else if (cmd === "query" || cmd === "materialize") {
+    const { pos, tables, udfs } = parse(args);
+    if (pos.length < 1) usage();
+    await registerUdfs(udfs);
+    const { handle } = openLazy(pos[0]);
+    for (const spec of tables) {
+      const eq = spec.indexOf("=");
+      if (eq < 0) usage();
+      engine.catalogRegister(spec.slice(0, eq), openLazy(spec.slice(eq + 1)).handle);
+    }
+    if (cmd === "materialize") {
+      if (pos.length !== 3) usage();
+      const t0 = performance.now();
+      const img = engine.materialize(handle, pos[1]);
+      const bytes = engine.imageBytes(img);
+      writeSync(openSync(pos[2], "w"), bytes);
+      const { rows } = engine.openImage(img);
+      console.error(`${pos[2]}: ${rows} rows, ${bytes.byteLength} bytes (${(performance.now() - t0).toFixed(1)} ms)`);
+    } else if (pos.length >= 2) {
+      const t0 = performance.now();
+      const r = engine.query(handle, pos[1]);
+      printResult(r, performance.now() - t0);
+    } else {
+      // REPL: one statement per line
+      const rl = (await import("node:readline")).createInterface({ input: process.stdin, output: process.stdout, prompt: "facetful> " });
+      rl.prompt();
+      rl.on("line", (line) => {
+        const sql = line.trim();
+        if (sql) {
+          try {
+            const t0 = performance.now();
+            printResult(engine.query(handle, sql), performance.now() - t0);
+          } catch (e) {
+            console.error(e instanceof QueryError ? e.message : e);
+          }
+        }
+        rl.prompt();
+      });
+      rl.on("close", () => process.exit(0));
+    }
+  } else usage();
+} catch (e) {
+  console.error(e instanceof QueryError ? e.message : e);
+  process.exit(1);
+}

package/core.js

@@ -10,10 +10,12 @@ const KINDS = { 1: "int", 2: "float", 3: "bool", 4: "text", 5: "date", 6: "times
 // Re-interpret as unsigned before building any view over memory.
 const u32 = (n) => n >>> 0;
 
-// The wasm module declares one import: env.opfs_read(fileId, offset, len, destPtr)
-// -> bytes read. The browser worker supplies a real implementation over OPFS
-// sync access handles; environments without OPFS (Node smoke test) get a stub
-// that fails any read (memory-backed tables never call it).
+// The wasm module declares two imports. env.opfs_read(fileId, offset, len,
+// destPtr) -> bytes read: the browser worker supplies a real implementation
+// over OPFS sync access handles; environments without OPFS (Node smoke test)
+// get a stub that fails any read (memory-backed tables never call it).
+// env.udf_call(id, argc, argsPtr, outPtr, len) evaluates a registered
+// user-defined function over whole lanes (see Engine.registerFunction).
 export async function instantiate(wasmBytes, opfsRead) {
   let engine = null;
   const env = {
@@ -22,18 +24,155 @@ export async function instantiate(wasmBytes, opfsRead) {
       // the view must be built per call: memory.buffer detaches on growth
       return opfsRead(fileId, offset, new Uint8Array(engine.mem(), u32(destPtr), u32(len)));
     },
+    udf_call: (id, argc, argsPtr, outPtr, len) => engine._udfCall(id, argc, u32(argsPtr), u32(outPtr), len),
   };
   const { instance } = await WebAssembly.instantiate(wasmBytes, { env });
   engine = new Engine(instance.exports);
   return engine;
 }
 
+/** Lane kinds on the UDF wire (the numbers are the ABI). */
+export const KIND = { int: 0, float: 1, bool: 2, text: 3, date: 4, timestamp: 5 };
+/** parameter-only kind: accepts any argument type (the lane carries its real kind) */
+const ANY = 6;
+const KIND_NAMES = Object.keys(KIND);
+// descriptor words: see facetful-wasm `WasmHost`
+const D_KIND = 0, D_LEN = 1, D_FLAGS = 2, D_DATA = 3, D_AUX = 4, D_VALID = 5, D_BYTES = 6, D_ERR = 7, D_WORDS = 8;
+
 export class Engine {
   constructor(exports) {
     this.w = exports;
     this.scratch = u32(this.w.alloc(4096));
     this.enc = new TextEncoder();
     this.dec = new TextDecoder();
+    /** id -> { fn, sig, name } for registered user-defined functions */
+    this.udfs = new Map();
+  }
+
+  /**
+   * Register a user-defined scalar function (design.sv d49). `sig` is
+   * `{ params: kind[], returns: kind, strict?, variadic?, optional?, perRow? }`
+   * with kinds from KIND (by name; a parameter may also be "any"). `optional`
+   * is how many trailing parameters may be omitted; `variadic` lets the last
+   * repeat. The function is called ONCE per lane —
+   * `fn(args, len, out)` where each arg is `{ kind, values, valid, broadcast }`
+   * (`values` a Float64Array for numbers/dates/bools, `string[]` for text;
+   * `broadcast` = a literal, one value; `valid` a bitmap or null) and `out` is
+   * `{ values, valid }` to fill (`values` a typed array or `string[]`; set a
+   * result to null to make it NULL). With `perRow: true` the function is
+   * instead called per row with plain values and returns one (or null).
+   * Strict (the default) skips NULL inputs and NULL-fills those outputs.
+   */
+  registerFunction(name, sig, fn) {
+    const kinds = (sig.params || []).map((k) => (k === "any" ? ANY : kindOf(k)));
+    const ret = kindOf(sig.returns);
+    const nameB = this.enc.encode(name);
+    const p = u32(this.w.alloc(nameB.byteLength + kinds.length + 1));
+    new Uint8Array(this.mem(), p, nameB.byteLength).set(nameB);
+    new Uint8Array(this.mem(), p + nameB.byteLength, kinds.length).set(kinds);
+    const flags = (sig.strict === false ? 0 : 1) | (sig.variadic ? 2 : 0) | ((sig.optional || 0) << 2);
+    const id = this.w.udf_register(p, nameB.byteLength, p + nameB.byteLength, kinds.length, ret, flags);
+    this.w.dealloc(p, nameB.byteLength + kinds.length + 1);
+    if (!id) {
+      const n = this.w.udf_error(this.scratch, 4096);
+      throw new Error(`registerFunction('${name}'): ${this.dec.decode(new Uint8Array(this.mem(), this.scratch, n))}`);
+    }
+    for (const [k, v] of this.udfs) if (v.name === name) this.udfs.delete(k);
+    this.udfs.set(id, { fn, sig: { ...sig, strict: sig.strict !== false }, name });
+    return id;
+  }
+
+  unregisterFunction(name) {
+    const b = this.enc.encode(name);
+    const p = u32(this.w.alloc(b.byteLength));
+    new Uint8Array(this.mem(), p, b.byteLength).set(b);
+    const ok = this.w.udf_unregister(p, b.byteLength) !== 0;
+    this.w.dealloc(p, b.byteLength);
+    for (const [k, v] of this.udfs) if (v.name === name) this.udfs.delete(k);
+    return ok;
+  }
+
+  /** The udf_call import: lanes in wasm memory -> the registered function -> the output lane. */
+  _udfCall(id, argc, argsPtr, outPtr, len) {
+    const entry = this.udfs.get(id);
+    try {
+      if (!entry) throw new Error(`no function registered under id ${id}`);
+      const mem = this.mem();
+      const args = [];
+      for (let i = 0; i < argc; i++) {
+        const d = new Uint32Array(mem, argsPtr + i * D_WORDS * 4, D_WORDS);
+        const n = d[D_LEN];
+        const kind = KIND_NAMES[d[D_KIND]];
+        const valid = d[D_VALID] ? new Uint8Array(mem, d[D_VALID], (n + 7) >> 3) : null;
+        let values;
+        if (kind === "text") {
+          values = this._decodeLane(new Uint8Array(mem, d[D_DATA], d[D_BYTES]), new Uint32Array(mem, d[D_AUX], n + 1), n);
+        } else {
+          values = new Float64Array(mem, d[D_DATA], n);
+        }
+        args.push({ kind, values, valid, broadcast: (d[D_FLAGS] & 1) !== 0 });
+      }
+      const od = new Uint32Array(mem, outPtr, D_WORDS);
+      const retKind = KIND_NAMES[od[D_KIND]];
+      const outValid = new Uint8Array(mem, od[D_VALID], (len + 7) >> 3);
+      const out = {
+        values: retKind === "text" ? new Array(len).fill("") : retKind === "bool" ? new Uint8Array(mem, od[D_DATA], len) : new Float64Array(mem, od[D_DATA], len),
+        valid: outValid,
+      };
+      const { fn, sig } = entry;
+      if (sig.perRow) {
+        const at = (a, i) => {
+          const j = a.broadcast ? 0 : i;
+          if (a.valid && !((a.valid[j >> 3] >> (j & 7)) & 1)) return null;
+          return a.values[j];
+        };
+        for (let i = 0; i < len; i++) {
+          const row = args.map((a) => at(a, i));
+          if (sig.strict && row.includes(null)) { outValid[i >> 3] &= ~(1 << (i & 7)); continue; }
+          const v = fn(...row);
+          if (v === null || v === undefined) outValid[i >> 3] &= ~(1 << (i & 7));
+          else out.values[i] = v;
+        }
+      } else {
+        fn(args, len, out);
+        // null results in a plain array mark NULL
+        if (retKind === "text") for (let i = 0; i < len; i++) if (out.values[i] == null) { outValid[i >> 3] &= ~(1 << (i & 7)); out.values[i] = ""; }
+      }
+      if (retKind === "text") {
+        const parts = out.values.map((s) => this.enc.encode(typeof s === "string" ? s : String(s)));
+        let total = 0;
+        for (const q of parts) total += q.byteLength;
+        const bp = total ? u32(this.w.alloc(total)) : 0;
+        const bytes = new Uint8Array(this.mem(), bp, total);
+        const offs = new Uint32Array(this.mem(), od[D_AUX], len + 1);
+        let pos = 0;
+        offs[0] = 0;
+        for (let i = 0; i < len; i++) { bytes.set(parts[i], pos); pos += parts[i].byteLength; offs[i + 1] = pos; }
+        const od2 = new Uint32Array(this.mem(), outPtr, D_WORDS);
+        od2[D_DATA] = bp;
+        od2[D_BYTES] = total;
+      }
+      return 0;
+    } catch (e) {
+      const msg = this.enc.encode(`${entry ? entry.name : "udf"}(): ${e && e.message ? e.message : e}`);
+      const p = u32(this.w.alloc(msg.byteLength));
+      new Uint8Array(this.mem(), p, msg.byteLength).set(msg);
+      new Uint32Array(this.mem(), outPtr, D_WORDS)[D_ERR] = p;
+      return -msg.byteLength;
+    }
+  }
+
+  /** A text lane to string[]: one decode when the bytes are ASCII (decoded
+   *  length == byte length, so byte offsets are char offsets), else per string. */
+  _decodeLane(bytes, offsets, n) {
+    const whole = this.dec.decode(bytes);
+    const out = new Array(n);
+    if (whole.length === bytes.byteLength) {
+      for (let i = 0; i < n; i++) out[i] = whole.slice(offsets[i], offsets[i + 1]);
+    } else {
+      for (let i = 0; i < n; i++) out[i] = this.dec.decode(bytes.subarray(offsets[i], offsets[i + 1]));
+    }
+    return out;
   }
   mem() {
     return this.w.memory.buffer;
@@ -157,6 +296,59 @@ export class Engine {
     return { handle, rows: this.w.table_total_rows(handle) };
   }
 
+  /**
+   * Stream a CSV into a .facetful image (design.sv d51): `chunks()` returns a
+   * fresh (async) iterable of Uint8Array chunks each time it is called — the
+   * converter reads the input twice (types and dictionaries, then encoding)
+   * in bounded memory; the image is assembled here. Returns
+   * { bytes, rows, schema: [{ name, kind }] }.
+   */
+  async convertCsv(chunks, { groupTarget = 65536 } = {}) {
+    const h = this.w.convert_begin(groupTarget);
+    const fail = () => {
+      const n = this.w.convert_error(h, this.scratch, 4096);
+      const msg = this.dec.decode(new Uint8Array(this.mem(), this.scratch, n));
+      this.w.convert_free(h);
+      throw new Error(`convert: ${msg}`);
+    };
+    const parts = [];
+    let total = 0;
+    const drain = () => {
+      const n = this.w.convert_output_len(h);
+      if (!n) return;
+      const p = u32(this.w.alloc(n));
+      this.w.convert_output_copy(h, p);
+      parts.push(new Uint8Array(this.mem(), p, n).slice());
+      this.w.dealloc(p, n);
+      total += n;
+    };
+    const feed = async () => {
+      for await (const c of chunks()) {
+        const bytes = c instanceof Uint8Array ? c : new Uint8Array(c);
+        const p = u32(this.w.alloc(bytes.byteLength));
+        new Uint8Array(this.mem(), p, bytes.byteLength).set(bytes);
+        const rc = this.w.convert_feed(h, p, bytes.byteLength);
+        this.w.dealloc(p, bytes.byteLength);
+        if (rc < 0) fail();
+        drain();
+      }
+    };
+    await feed();
+    if (this.w.convert_pass2(h) < 0) fail();
+    const n = this.w.convert_schema(h, this.scratch, 4096);
+    const schema = this.dec.decode(new Uint8Array(this.mem(), this.scratch, n)).trimEnd().split("\n")
+      .filter(Boolean).map((l) => { const [name, kind] = l.split("\t"); return { name, kind }; });
+    await feed();
+    if (this.w.convert_finish(h) < 0) fail();
+    drain();
+    const rows = this.w.convert_rows(h);
+    this.w.convert_free(h);
+    const bytes = new Uint8Array(total);
+    let pos = 0;
+    for (const q of parts) { bytes.set(q, pos); pos += q.byteLength; }
+    return { bytes, rows, schema };
+  }
+
   /** Make `handle` reachable by `name` from other tables' queries (FROM / JOIN). */
   catalogRegister(name, handle) {
     const b = this.enc.encode(name);
@@ -226,3 +418,9 @@ export function transferables(result) {
   }
   return t;
 }
+
+function kindOf(k) {
+  if (typeof k === "number") return k;
+  if (k in KIND) return KIND[k];
+  throw new Error(`unknown lane kind '${k}' (expected one of ${Object.keys(KIND).join(", ")})`);
+}

package/index.d.ts

@@ -14,6 +14,8 @@ export interface OpenOptions {
    * explicit URL when running without one.
    */
   hyparquetUrl?: string;
+  /** register the ready-made functions from `facetful/udfs` at open (default true) */
+  udfs?: boolean;
 }
 
 export interface QueryStats {
@@ -51,6 +53,36 @@ export declare class Result {
   rows(): Generator<Record<string, CellValue>>;
 }
 
+export type LaneKind = "int" | "float" | "bool" | "text" | "date" | "timestamp";
+export interface UdfSignature {
+  /** parameter types; "any" accepts every type (the lane still carries its real kind) */
+  params: (LaneKind | "any")[];
+  returns: LaneKind;
+  /** how many trailing parameters may be omitted */
+  optional?: number;
+  /** NULL in → NULL out without calling the function for that row (default true) */
+  strict?: boolean;
+  /** the last parameter type repeats */
+  variadic?: boolean;
+  /** call per row with plain values instead of once per lane */
+  perRow?: boolean;
+}
+export interface UdfLane {
+  kind: LaneKind;
+  /** Float64Array for int/float/bool/date/timestamp, string[] for text */
+  values: Float64Array | string[];
+  /** validity bitmap (bit set = present), or null when all present */
+  valid: Uint8Array | null;
+  /** a literal argument: one value, applies to every row */
+  broadcast: boolean;
+}
+export type UdfVectorFn = (
+  args: UdfLane[],
+  len: number,
+  out: { values: Float64Array | Uint8Array | (string | null)[]; valid: Uint8Array },
+) => void;
+export type UdfRowFn = (...values: (number | string | null)[]) => number | string | boolean | null;
+
 export interface LoadResult {
   name: string;
   rows: number;
@@ -104,6 +136,24 @@ export declare class Facetful {
     options?: { table?: string; persist?: string },
   ): Promise<{ rows: number; bytes: number; elapsedMs: number }>;
   storeOpfs(path: string, buffer: ArrayBuffer): Promise<{ bytes: number }>;
+  /**
+   * Register a user-defined scalar function (runs in the worker; pass a
+   * self-contained function, its source, or `{ moduleUrl }`). Vectorized by
+   * default — `fn(args, len, out)` once per lane; `perRow: true` calls
+   * `fn(...values)` per row, returning a value or null.
+   */
+  /** Stream a CSV (File/Blob or ArrayBuffer) into a table; two passes, bounded memory. */
+  loadCsv(
+    name: string,
+    source: Blob | ArrayBuffer,
+    options?: { persist?: string; groupTarget?: number },
+  ): Promise<{ rows: number; bytes: number; schema: { name: string; kind: string }[]; elapsedMs: number }>;
+  registerFunction(
+    name: string,
+    signature: UdfSignature,
+    fn: UdfVectorFn | UdfRowFn | string | { moduleUrl: string },
+  ): Promise<{ ok: true }>;
+  unregisterFunction(name: string): Promise<{ ok: boolean }>;
 
   /**
    * Open a table over an OPFS file: metadata reads now, column segments load

package/index.js

@@ -9,7 +9,7 @@
 //   [...r.rows()]              // row objects, materialized lazily
 
 export class Facetful {
-  static async open({ wasmUrl, workerUrl, hyparquetUrl } = {}) {
+  static async open({ wasmUrl, workerUrl, hyparquetUrl, udfs = true } = {}) {
     // the no-argument form must stay a literal `new Worker(new URL(...))`
     // expression: bundlers (Vite, webpack) statically analyze exactly that
     // pattern to compile the worker graph
@@ -21,6 +21,7 @@ export class Facetful {
       cmd: "init",
       wasmUrl: String(wasmUrl ?? new URL("./facetful_wasm.wasm", import.meta.url)),
       hyparquetUrl,
+      udfs,
     });
     return db;
   }
@@ -85,6 +86,40 @@ export class Facetful {
     return this._call({ cmd: "materialize", name, sql, table, persist });
   }
 
+  /**
+   * Convert a CSV (a File/Blob, or an ArrayBuffer) into a table named `name`,
+   * streaming in bounded memory — two passes over the input, so a File is
+   * read twice. Types are inferred (int, float, date, timestamp, text;
+   * repeated text becomes a dictionary). Optionally persist the image to OPFS.
+   */
+  async loadCsv(name, source, { persist, groupTarget } = {}) {
+    const transfer = source instanceof ArrayBuffer ? [source] : [];
+    return this._call({ cmd: "loadCsv", name, source, persist, groupTarget }, transfer);
+  }
+
+  /**
+   * Register a user-defined scalar function, callable from any query.
+   * `signature` = { params: kind[], returns: kind, strict?, variadic?, perRow? }
+   * with kinds "int" | "float" | "bool" | "text" | "date" | "timestamp".
+   * `fn` runs in the worker, so pass a self-contained function (its source is
+   * sent — no closures over your variables) or `{ moduleUrl }` whose default
+   * export is the function. Vectorized by default: fn(args, len, out) is
+   * called once per lane (see core.js Engine.registerFunction); with
+   * `perRow: true` it is called per row with plain values and returns one.
+   */
+  async registerFunction(name, signature, fn) {
+    const msg = { cmd: "registerFunction", name, signature };
+    if (typeof fn === "function") msg.source = fn.toString();
+    else if (fn && fn.moduleUrl) msg.moduleUrl = fn.moduleUrl;
+    else if (typeof fn === "string") msg.source = fn;
+    else throw new Error("registerFunction: pass a function, its source text, or { moduleUrl }");
+    return this._call(msg);
+  }
+
+  async unregisterFunction(name) {
+    return this._call({ cmd: "unregisterFunction", name });
+  }
+
   /** Persist a .facetful image into OPFS at `path` (e.g. "facetful/plants.facetful"). */
   async storeOpfs(path, buffer) {
     return this._call({ cmd: "storeOpfs", path, buffer }, [buffer]);

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "facetful",
-  "version": "0.4.0",
-  "description": "Tiny columnar SQL engine for browser faceting — Rust/wasm, ~170KB gz, zero dependencies. Open Parquet instantly, facet 1M rows in milliseconds, persist to OPFS.",
+  "version": "0.5.0",
+  "description": "Tiny columnar SQL engine for browser faceting — Rust/wasm, ~240KB gz, zero dependencies. Open Parquet or CSV instantly, facet 1M rows in milliseconds, persist to OPFS; one install gives the library and the `facetful` command.",
   "license": "MIT",
   "author": "David Raznick",
   "type": "module",
@@ -15,7 +15,11 @@
     "./core": "./core.js",
     "./worker": "./worker.js",
     "./parquet": "./parquet.js",
-    "./facetful_wasm.wasm": "./facetful_wasm.wasm"
+    "./facetful_wasm.wasm": "./facetful_wasm.wasm",
+    "./udfs": {
+      "types": "./udfs.d.ts",
+      "default": "./udfs.js"
+    }
   },
   "files": [
     "index.js",
@@ -23,6 +27,9 @@
     "core.js",
     "worker.js",
     "parquet.js",
+    "udfs.js",
+    "udfs.d.ts",
+    "bin/facetful.mjs",
     "facetful_wasm.wasm",
     "README.md",
     "LICENSE"
@@ -43,7 +50,9 @@
     "opfs",
     "facets",
     "analytics",
-    "browser"
+    "browser",
+    "csv",
+    "cli"
   ],
   "sideEffects": false,
   "repository": {
@@ -54,5 +63,8 @@
   "homepage": "https://github.com/kindly/facetful#readme",
   "bugs": {
     "url": "https://github.com/kindly/facetful/issues"
+  },
+  "bin": {
+    "facetful": "bin/facetful.mjs"
   }
 }

package/udfs.d.ts

@@ -0,0 +1,15 @@
+import type { UdfSignature, UdfVectorFn, UdfRowFn } from "./index.js";
+
+export interface ReadyMadeUdf {
+  name: string;
+  signature: UdfSignature;
+  fn: UdfVectorFn | UdfRowFn;
+}
+
+/**
+ * The ready-made functions Facetful.open() registers by default: regexp,
+ * json_extract, to_tz, date_trunc, date_add, weekday, quarter, country_name,
+ * format_number, unaccent, url_host.
+ */
+export const udfs: ReadyMadeUdf[];
+export default udfs;

package/udfs.js

@@ -0,0 +1,205 @@
+// Ready-made user-defined functions (design.sv d49/d51): things the browser
+// does natively that would cost the wasm engine tens or hundreds of KB —
+// JSON, the Intl tables (timezones, country names, number formats), Unicode
+// normalization, URL parsing — plus the temporal long tail over Date.UTC.
+//
+// Registered by default by Facetful.open() and the `facetful` command
+// (`open({ udfs: false })` opts out). The list is exported for callers that
+// drive core.js directly or want a subset.
+//
+// Each entry is { name, signature, fn }. Functions are self-contained (they run
+// in the worker from their source), vectorized where it pays, per-row where it
+// reads better. Dates are days since 1970-01-01, timestamps ms since the epoch.
+
+const DAY = 86400000;
+
+export const udfs = [
+  // --- regular expressions ---------------------------------------------------
+  // regexp(s, pattern[, flags]) -> bool, the browser's RegExp (JIT-compiled, cached per
+  // pattern); the native CLI answers the same SQL through the `regex` crate
+  {
+    name: "regexp",
+    signature: { params: ["text", "text", "text"], returns: "bool", optional: 1 },
+    fn: (() => {
+      const cache = new Map();
+      return (args, len, out) => {
+        const [s, p] = args;
+        const key = p.values[0] + "\u0000" + (args[2] ? args[2].values[0] : "");
+        let re = cache.get(key);
+        if (!re) cache.set(key, (re = new RegExp(p.values[0], args[2] ? args[2].values[0] : "")));
+        for (let i = 0; i < len; i++) out.values[i] = re.test(s.values[i]) ? 1 : 0;
+      };
+    })(),
+  },
+  // regexp_extract(s, pattern[, group]) -> text: the match, or group n (number) / a
+  // named group (text); NULL when nothing matches
+  {
+    name: "regexp_extract",
+    signature: { params: ["text", "text", "any"], returns: "text", optional: 1 },
+    fn: (() => {
+      const cache = new Map();
+      return (args, len, out) => {
+        const [s, p] = args;
+        const pat = p.values[0];
+        let re = cache.get(pat);
+        if (!re) cache.set(pat, (re = new RegExp(pat, "u")));
+        const g = args[2] ? args[2].values[0] : 0;
+        for (let i = 0; i < len; i++) {
+          const m = re.exec(s.values[i]);
+          if (!m) { out.values[i] = null; continue; }
+          const v = typeof g === "number" ? m[g] : m.groups ? m.groups[g] : undefined;
+          out.values[i] = v === undefined ? null : v;
+        }
+      };
+    })(),
+  },
+  // regexp_replace(s, pattern, replacement[, flags]) -> text: every match replaced;
+  // $1 and $<name> refer to groups (the native CLI accepts the same spelling)
+  {
+    name: "regexp_replace",
+    signature: { params: ["text", "text", "text", "text"], returns: "text", optional: 1 },
+    fn: (() => {
+      const cache = new Map();
+      return (args, len, out) => {
+        const [s, p, r] = args;
+        const flags = args[3] ? args[3].values[0] : "";
+        const key = p.values[0] + "\u0000" + flags;
+        let re = cache.get(key);
+        if (!re) cache.set(key, (re = new RegExp(p.values[0], flags.includes("g") ? flags : flags + "g")));
+        const repl = r.values[0];
+        for (let i = 0; i < len; i++) out.values[i] = s.values[i].replace(re, repl);
+      };
+    })(),
+  },
+  // --- JSON ---------------------------------------------------------------
+  // json_extract(doc, '$.a.b[0]') -> text (numbers/bools stringified, null for missing)
+  {
+    name: "json_extract",
+    signature: { params: ["text", "text"], returns: "text" },
+    fn: (() => {
+      const paths = new Map();
+      const parsePath = (p) => {
+        let keys = paths.get(p);
+        if (!keys) {
+          keys = [];
+          for (const m of p.replace(/^\$\.?/, "").matchAll(/([^.[\]]+)|\[(\d+)\]/g)) keys.push(m[2] !== undefined ? Number(m[2]) : m[1]);
+          paths.set(p, keys);
+        }
+        return keys;
+      };
+      return (args, len, out) => {
+        const [doc, path] = args;
+        const keys = parsePath(path.values[0]);
+        for (let i = 0; i < len; i++) {
+          let v;
+          try { v = JSON.parse(doc.values[i]); } catch { out.values[i] = null; continue; }
+          for (const k of keys) { if (v == null) break; v = v[k]; }
+          out.values[i] = v == null ? null : typeof v === "object" ? JSON.stringify(v) : String(v);
+        }
+      };
+    })(),
+  },
+  // --- time zones and the temporal long tail -------------------------------
+  // to_tz(ts, 'America/New_York') -> text "YYYY-MM-DD HH:MM:SS" in that zone (Intl's IANA tables)
+  {
+    name: "to_tz",
+    signature: { params: ["timestamp", "text"], returns: "text" },
+    fn: (() => {
+      const fmts = new Map();
+      return (args, len, out) => {
+        const [ts, tz] = args;
+        const zone = tz.values[0];
+        let f = fmts.get(zone);
+        if (!f) fmts.set(zone, (f = new Intl.DateTimeFormat("sv-SE", { timeZone: zone, year: "numeric", month: "2-digit", day: "2-digit", hour: "2-digit", minute: "2-digit", second: "2-digit", hour12: false })));
+        for (let i = 0; i < len; i++) out.values[i] = f.format(ts.values[i]);
+      };
+    })(),
+  },
+  // date_trunc('month', ts) -> timestamp at the start of the unit (UTC); units: year quarter month week day hour
+  {
+    name: "date_trunc",
+    signature: { params: ["text", "timestamp"], returns: "timestamp" },
+    fn: (args, len, out) => {
+      const [unit, ts] = args;
+      const u = unit.values[0];
+      for (let i = 0; i < len; i++) {
+        const d = new Date(ts.values[i]);
+        let r;
+        switch (u) {
+          case "year": r = Date.UTC(d.getUTCFullYear(), 0, 1); break;
+          case "quarter": r = Date.UTC(d.getUTCFullYear(), d.getUTCMonth() - (d.getUTCMonth() % 3), 1); break;
+          case "month": r = Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), 1); break;
+          case "week": { const dow = (d.getUTCDay() + 6) % 7; r = Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate() - dow); break; } // Monday
+          case "day": r = Date.UTC(d.getUTCFullYear(), d.getUTCMonth(), d.getUTCDate()); break;
+          case "hour": r = ts.values[i] - (ts.values[i] % 3600000); break;
+          default: throw new Error(`date_trunc: unknown unit '${u}'`);
+        }
+        out.values[i] = r;
+      }
+    },
+  },
+  // date_add(d, n, 'month') -> date; units: day week month year (month arithmetic clamps to the month's end)
+  {
+    name: "date_add",
+    signature: { params: ["date", "int", "text"], returns: "date", perRow: true },
+    fn: (d, n, unit) => {
+      const t = new Date(d * DAY);
+      switch (unit) {
+        case "day": return d + n;
+        case "week": return d + 7 * n;
+        case "month": case "year": {
+          const months = unit === "year" ? 12 * n : n;
+          const y = t.getUTCFullYear(), m = t.getUTCMonth() + months, day = t.getUTCDate();
+          const last = new Date(Date.UTC(y, m + 1, 0)).getUTCDate();
+          return Date.UTC(y, m, Math.min(day, last)) / DAY;
+        }
+        default: throw new Error(`date_add: unknown unit '${unit}'`);
+      }
+    },
+  },
+  // weekday(d) -> int, 0 = Monday … 6 = Sunday; quarter(d) -> 1..4
+  { name: "weekday", signature: { params: ["date"], returns: "int" }, fn: (args, len, out) => { for (let i = 0; i < len; i++) out.values[i] = (new Date(args[0].values[i] * DAY).getUTCDay() + 6) % 7; } },
+  { name: "quarter", signature: { params: ["date"], returns: "int" }, fn: (args, len, out) => { for (let i = 0; i < len; i++) out.values[i] = Math.floor(new Date(args[0].values[i] * DAY).getUTCMonth() / 3) + 1; } },
+  // --- Intl tables -----------------------------------------------------------
+  // country_name('DE') -> 'Germany' (ISO 3166 alpha-2; optional locale second arg)
+  {
+    name: "country_name",
+    signature: { params: ["text", "text"], returns: "text", optional: 1 },
+    fn: (() => {
+      const names = new Map();
+      return (args, len, out) => {
+        const locale = args.length > 1 ? args[1].values[0] : "en";
+        let dn = names.get(locale);
+        if (!dn) names.set(locale, (dn = new Intl.DisplayNames([locale], { type: "region" })));
+        for (let i = 0; i < len; i++) {
+          const code = args[0].values[i].toUpperCase();
+          try { out.values[i] = /^[A-Z]{2}$/.test(code) ? dn.of(code) : null; } catch { out.values[i] = null; }
+        }
+      };
+    })(),
+  },
+  // format_number(x, 'en-US') -> '1,234,567.9' ; compact form via 'en-US:compact' -> '1.2M'
+  {
+    name: "format_number",
+    signature: { params: ["float", "text"], returns: "text" },
+    fn: (() => {
+      const fmts = new Map();
+      return (args, len, out) => {
+        const spec = args[1].values[0];
+        let f = fmts.get(spec);
+        if (!f) {
+          const [locale, style] = spec.split(":");
+          fmts.set(spec, (f = new Intl.NumberFormat(locale, style === "compact" ? { notation: "compact", maximumFractionDigits: 1 } : { maximumFractionDigits: 1 })));
+        }
+        for (let i = 0; i < len; i++) out.values[i] = f.format(args[0].values[i]);
+      };
+    })(),
+  },
+  // --- Unicode and URLs -------------------------------------------------------
+  // unaccent('Zürich') -> 'Zurich' : NFD then strip combining marks (\p{M} — the regex class wasm can't afford)
+  { name: "unaccent", signature: { params: ["text"], returns: "text" }, fn: (args, len, out) => { for (let i = 0; i < len; i++) out.values[i] = args[0].values[i].normalize("NFD").replace(/\p{M}+/gu, ""); } },
+  // url_host('https://www.eia.gov/x?y') -> 'www.eia.gov' (null when not a URL)
+  { name: "url_host", signature: { params: ["text"], returns: "text", perRow: true }, fn: (s) => { try { return new URL(s).hostname; } catch { return null; } } },
+];
+
+export default udfs;

package/worker.js

@@ -5,6 +5,7 @@
 
 import { instantiate, transferables, QueryError } from "./core.js";
 import { parquetToColumns } from "./parquet.js";
+import { udfs as builtinUdfs } from "./udfs.js";
 
 let engine = null;
 const tables = new Map(); // name -> handle
@@ -126,6 +127,9 @@ self.onmessage = async (e) => {
       if (e.data.hyparquetUrl) hyparquetUrl = e.data.hyparquetUrl;
       const wasmBytes = await (await fetch(e.data.wasmUrl)).arrayBuffer();
       engine = await instantiate(wasmBytes, opfsRead);
+      // the ready-made functions (udfs.js) are on by default: they cost
+      // nothing until a query calls one, and agents expect date_trunc to exist
+      if (e.data.udfs !== false) for (const u of builtinUdfs) engine.registerFunction(u.name, u.signature, u.fn);
       reply({ ok: true });
     } else if (cmd === "load") {
       const { handle, rows } = engine.openTable(e.data.buffer);
@@ -186,6 +190,30 @@ self.onmessage = async (e) => {
       const { handle, rows } = engine.openImage(img);
       setTable(e.data.name, handle);
       reply({ ok: true, rows, bytes, elapsedMs: performance.now() - t0 });
+    } else if (cmd === "loadCsv") {
+      // a File/Blob streams twice through the converter; a buffer is one chunk
+      const src = e.data.source;
+      const chunks = typeof src.stream === "function"
+        ? () => src.stream()
+        : async function* () { yield new Uint8Array(src); };
+      const t0 = performance.now();
+      const { bytes, rows, schema } = await engine.convertCsv(chunks, { groupTarget: e.data.groupTarget });
+      if (e.data.persist) await opfsWrite(e.data.persist, bytes);
+      const { handle } = engine.openTable(bytes);
+      setTable(e.data.name, handle);
+      lastTable = handle;
+      reply({ ok: true, rows, bytes: bytes.byteLength, schema, elapsedMs: performance.now() - t0 });
+    } else if (cmd === "registerFunction") {
+      // functions don't cross postMessage: `source` is the function's text (an
+      // expression — an arrow function, or an IIFE returning one for state)
+      const fn = e.data.moduleUrl
+        ? (await import(e.data.moduleUrl)).default
+        : new Function(`return (${e.data.source});`)();
+      if (typeof fn !== "function") throw new Error(`registerFunction('${e.data.name}'): source is not a function`);
+      engine.registerFunction(e.data.name, e.data.signature, fn);
+      reply({ ok: true });
+    } else if (cmd === "unregisterFunction") {
+      reply({ ok: engine.unregisterFunction(e.data.name) });
     } else if (cmd === "storeOpfs") {
       await opfsWrite(e.data.path, new Uint8Array(e.data.buffer));
       reply({ ok: true, bytes: e.data.buffer.byteLength });