vectrojs 0.1.1 → 0.1.2

package/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 0.1.2
+
+A correctness release, from comparing about 1,700 new SQL queries with DuckDB
+on the same data. No API changes. Every case below now gives DuckDB's answer,
+or a clear error.
+
+**Engine**
+- A query too big for the 4GB memory (a join with billions of output rows)
+  now fails with `out of memory (4GB limit)`. Before, the size wrapped around
+  in the allocator, and later queries on the same engine could return wrong
+  numbers.
+- Missing text (`''`) sorts last, up or down, as missing numbers do, in
+  `sort`, `top`, window ordering and SQL `ORDER BY`.
+- `null` and `undefined` in a text column load as missing text, also when
+  the column starts with one (`engine.table`, `tableFromRows`).
+- A missing text key never matches in a join, as the docs say.
+- A filter with no column in it (`'1'`, `'1 == 0'`) keeps every row or none,
+  instead of failing.
+- Comparing a text column with `<` or `>` and then grouping gives a clear
+  error instead of overflowing the stack. On datasets, the unknown-column
+  error lists every column.
+
+**SQL**
+- Conditions follow SQL's three values everywhere: a condition on a missing
+  value is NULL when selected, `IS TRUE` / `IS FALSE` treat it as neither,
+  comparing two conditions is unknown if either is, and a `NULL` literal in
+  `AND` / `OR` is unknown rather than true.
+- `WHERE TRUE`, `WHERE 1 = 1`, `WHERE NULL` and other constant conditions work.
+- `ORDER BY text NULLS FIRST` works.
+- `x NOT IN (SELECT ...)` keeps every row when the subquery is empty and no
+  row when it contains a NULL.
+- `LEAST` and `GREATEST` skip missing values (NULL only if all are).
+- The same aggregate twice (`SUM(x) AS a, SUM(x) AS b`) gives NULL both times
+  over no values.
+- Subqueries in the SELECT list or HAVING of a GROUP BY query work.
+
 ## 0.1.1
 
 A performance release. No API changes.

package/README.md

@@ -5,7 +5,7 @@ and compiles each query, filter included, into one SIMD WebAssembly loop, then
 runs it on every CPU core. It runs in Node, in browsers (any page) and on
 Cloudflare Workers, with no native addon, compiler or server.
 
-> Version 0.1.1. Node 20+ and modern browsers. MIT license.
+> Version 0.1.2. Node 20+ and modern browsers. MIT license.
 
 **Website and docs: [vectrojs.dev](https://vectrojs.dev)**. [Quick start](https://vectrojs.dev/docs/quick-start) · [API reference](https://vectrojs.dev/api) · [Race it in your browser](https://vectrojs.dev/live) · [Benchmarks](https://vectrojs.dev/benchmarks)
 
@@ -30,18 +30,27 @@ orders.groupBy('region').agg({ revenue: 'sum(price * qty)' }).toRows();
 npm install vectrojs
 ```
 
+Save this as `try.mjs` (or set `"type": "module"` in package.json) and run `node try.mjs`:
+
 ```js
 import { Engine, sql } from 'vectrojs';
 
-const engine = new Engine();
-const orders = engine.readCSV('orders.csv');
-sql('SELECT region, COUNT(*) AS n, SUM(price) AS revenue FROM orders GROUP BY region ORDER BY revenue DESC', { orders }).toRows();
+const engine = new Engine();                         // uses every core
+const orders = engine.table({
+  region: ['EU', 'US', 'EU', 'APAC', 'US'],
+  price:  [120, 80, 300, 45, 99],
+});
+console.log(sql('SELECT region, COUNT(*) AS n, SUM(price) AS revenue FROM orders GROUP BY region ORDER BY revenue DESC', { orders }).toRows());
+// [ { region: 'EU', n: 2, revenue: 420 }, { region: 'US', n: 2, revenue: 179 }, { region: 'APAC', n: 1, revenue: 45 } ]
+await engine.close();
 ```
 
+For your own data, `engine.readCSV('orders.csv')` reads a CSV file on every core.
+
 Serve a folder of CSV / Arrow / `.vectro` files to BI tools, with no code:
 
 ```sh
-npx vectro serve --root ./data --pg 5433      # HTTP on 8787, PostgreSQL on 5433
+npx -y -p vectrojs vectro serve --root ./data --pg 5433      # HTTP on 8787, PostgreSQL on 5433
 psql -h 127.0.0.1 -p 5433 -U vectro vectro     # or Power BI, Tableau, Metabase, Grafana
 ```
 
@@ -408,8 +417,8 @@ there (1.4x at a billion rows, 1.5x at 100M).
 ## SQL over HTTP: Excel, Power BI, Google Sheets, pandas
 
 ```sh
-npx vectro serve --root ~/data        # every CSV, TSV, JSON-lines, Arrow / Feather and .vectro file is a table
-curl 'http://127.0.0.1:8787/sql?q=SELECT region, COUNT(*) AS n FROM orders GROUP BY 1&format=csv'
+npx -y -p vectrojs vectro serve --root ~/data        # every CSV, TSV, JSON-lines, Arrow / Feather and .vectro file is a table
+curl -G http://127.0.0.1:8787/sql --data-urlencode 'q=SELECT region, COUNT(*) AS n FROM orders GROUP BY 1' -d format=csv
 ```
 
 | Endpoint | Gives |
@@ -441,7 +450,7 @@ On 10M orders: a group by region answers in 9 ms over HTTP; 1M rows export in 0.
 ## The PostgreSQL protocol: Power BI, Tableau, Metabase, Grafana, psql
 
 ```sh
-npx vectro serve --root ~/data --pg 5433 --user me --password secret
+npx -y -p vectrojs vectro serve --root ~/data --pg 5433 --user me --password secret
 psql -h 127.0.0.1 -p 5433 -U me vectro
 ```
 
@@ -496,7 +505,7 @@ ODBC, and every driver. Queries arrive in each tool's own SQL and run on Vectro'
 ## Arrow Flight SQL: pandas, Polars, DuckDB
 
 ```sh
-npx vectro serve --root ~/data --flight 31337            # grpc://localhost:31337 (--user / --password, --tls-cert / --tls-key)
+npx -y -p vectrojs vectro serve --root ~/data --flight 31337            # grpc://localhost:31337 (--user / --password, --tls-cert / --tls-key)
 ```
 
 ```python
@@ -729,6 +738,7 @@ writeFileSync('eu.feather', sales.filter("region == 'EU'").toArrow({ file: true
 // split tables: await engine.fromArrowAsync(bytes), await table.toArrowAsync()
 
 // streams: any Arrow IPC stream over TCP or WebSocket, e.g. from pyarrow
+const split = new SplitEngine();                                     // import { SplitEngine } from 'vectrojs'
 const orders = await split.streamAsync({ format: 'arrow', columns: { ts: 'number', region: 'category' } });
 // ...or Vectro's Emitter speaking Arrow: Emitter.listen({ columns, format: 'arrow' })
 ```

package/mcp/server.mjs

@@ -14,7 +14,7 @@ import { createInterface } from 'node:readline';
 import { Engine, sql } from '../src/node.mjs';
 import { LoadError, safePath as safePathIn, loadFile, columnsOf, kindOf } from '../src/serve/catalog.mjs';
 
-const VERSION = '0.1.1';
+const VERSION = '0.1.2';
 const PROTOCOLS = ['2025-06-18', '2025-03-26', '2024-11-05'];
 const MAX_ROWS = 500, DEFAULT_ROWS = 50;
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vectrojs",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "description": "Multi-core analytics engine for JavaScript: columnar tables, SQL and fused SIMD WebAssembly kernels for Node, browsers and edge runtimes, with no native code. Serves tables to Power BI, Tableau, Metabase, Grafana, pandas and Polars.",
   "type": "module",
   "license": "MIT",

package/src/dataset.mjs

@@ -834,7 +834,12 @@ export function* runPlanG(plan, sink = null) {
     for (let n = plan; n.op !== 'scan' && n.op !== 'union'; ) { n = n.op === 'join' ? n.left : n.input; path.push(n); }
     const scan = path[path.length - 1];
     if (!scan.table?.isDataset) result = yield { run: plan };
-    else result = yield* runChunksG(plan, path, sink);
+    else {
+      try { result = yield* runChunksG(plan, path, sink); } catch (err) { // (chunks are read with only the columns used: name them all)
+        if (/^vectro: unknown column /.test(err?.message)) err.message = err.message.replace(/\(columns: [^)]*\)$/, `(columns: ${scan.table.columnNames.join(', ')})`);
+        throw err;
+      }
+    }
     if (sink && result) { // (the result itself was built in memory)
       if (result.cols) sink.append(result);
       else sink.append(Object.fromEntries(Object.entries(result).map(([k, v]) => [k, [v]])));

package/src/engine.mjs

@@ -32,7 +32,7 @@ const DEFAULT_MIN_PARALLEL_WASM = {
 const PAGE = 65536;
 const MAX_PAGES = 65536; // 4GB, the most a 32-bit WASM address can reach
 const HEAP_START = 64;   // keep address 0 unused
-const align16 = (n) => (n + 15) & ~15;
+const align16 = (n) => Math.ceil(n / 16) * 16; // (not bitwise: sizes may pass 2^31 bytes, and must not wrap)
 
 // Can this runtime compile WebAssembly from bytes? (Cloudflare Workers and
 // Vercel Edge can't: they only run modules bundled ahead of time.)
@@ -141,6 +141,8 @@ export class Engine {
   // WASM SIMD kernels; other shared arrays fall back to JS kernels.
   alloc(Type, n) {
     const size = align16(Math.max(16, n * Type.BYTES_PER_ELEMENT));
+    // too big for the 4GB memory (a huge join output, say): fail before touching the free list
+    if (!(size <= MAX_PAGES * PAGE - 64)) throw new RangeError('vectro: out of memory (4GB limit)');
     let ptr = -1;
     for (let i = 0; i < this.freeList.length; i++) {
       const blk = this.freeList[i];

package/src/fuse.mjs

@@ -125,6 +125,12 @@ export function walk(n, f) {
   else if (n.t === 'cond') { walk(n.c, f); walk(n.x, f); walk(n.y, f); }
 }
 
+// text left in an expression a kernel computes (a text column was compared with < or >, or
+// isn't one): a clear error, not a loop between the value and condition emitters
+function noText(n) {
+  if (n.t === 'str') throw new SyntaxError(`vectro: text like ${JSON.stringify(n.v)} can only be compared with == or != to a text column (to compare dates, load the column as dates: { dates: ['name'] })`);
+}
+
 function checkElementwise(n, src) {
   walk(n, (m) => {
     if (m.t === 'str') throw new SyntaxError(`vectro: text like ${JSON.stringify(m.v)} can only be compared (== / !=) with a text column, in "${src}"`);
@@ -162,7 +168,10 @@ export function analyze(spec) {
 
 // Constant parts computed now: 2 * 3 -> 6, floor(2.5) -> 2, (1 ? x : y) -> x.
 // Same results as the kernels (IEEE doubles; f32 kernels round literals).
-const FOLD_BIN = { '+': (a, b) => a + b, '-': (a, b) => a - b, '*': (a, b) => a * b, '/': (a, b) => a / b };
+const FOLD_BIN = { '+': (a, b) => a + b, '-': (a, b) => a - b, '*': (a, b) => a * b, '/': (a, b) => a / b,
+  // as the kernels compute them: 1 / 0, a comparison with NaN false (but != true), NaN counting as true in && and ||
+  '==': (a, b) => +(a === b), '!=': (a, b) => +(a !== b), '<': (a, b) => +(a < b), '<=': (a, b) => +(a <= b), '>': (a, b) => +(a > b), '>=': (a, b) => +(a >= b),
+  '&&': (a, b) => +(a !== 0 && b !== 0), '||': (a, b) => +(a !== 0 || b !== 0) };
 const FOLD_FN = { abs: Math.abs, sqrt: Math.sqrt, floor: Math.floor, ceil: Math.ceil, min: Math.min, max: Math.max };
 export function fold(n) {
   const isNum = (x) => x.t === 'num';
@@ -255,8 +264,8 @@ export class Emitter {
     this.memo.set(m, l);
   }
 
-  vec(n, k) { return this.#reuse(n, k, V128, () => this.#vec(n, k)); }
-  sca(n) { return this.#reuse(n, 'sca', this.ty.val, () => this.#sca(n)); }
+  vec(n, k) { noText(n); return this.#reuse(n, k, V128, () => this.#vec(n, k)); }
+  sca(n) { noText(n); return this.#reuse(n, 'sca', this.ty.val, () => this.#sca(n)); }
 
   #vec(n, k) {
     const { f, env } = this, V = VEC[this.t];

package/src/join.mjs

@@ -415,7 +415,7 @@ function* buildRight(engine, t, L, R, lk, rk, how) {
     for (let k = 0; k < nk; k++) {
       const a = L.cols.get(lk[k]), b = R.cols.get(rk[k]);
       if (!b.dict) { build.rightKeys.push(b.data); continue; }
-      const lut = b.dict.map((label) => a.codes?.get(label) ?? NaN); // a label the left table lacks matches nothing
+      const lut = b.dict.map((label) => (label === '' ? NaN : a.codes?.get(label) ?? NaN)); // missing text, or a label the left table lacks, matches nothing
       const mapped = alloc(L.type, nR);
       const src = engine.live(b.data), dst = engine.live(mapped);
       for (let i = 0; i < nR; i++) dst[i] = lut[src[i]];

package/src/sort.mjs

@@ -22,10 +22,11 @@ export function parseKeys(by, cols) {
 }
 
 // Ranks for a category column's codes: code -> position of its label A..Z.
+// Missing text ('') ranks NaN, so it comes last either way, as a missing number does.
 export function labelRanks(dict, desc) {
   const order = dict.map((_, i) => i).sort((a, b) => (dict[a] < dict[b] ? -1 : dict[a] > dict[b] ? 1 : 0));
   const rank = new Float64Array(dict.length);
-  order.forEach((code, r) => { rank[code] = desc ? -r : r; });
+  order.forEach((code, r) => { rank[code] = dict[code] === '' || dict[code] == null ? NaN : desc ? -r : r; });
   return rank;
 }
 

package/src/sql.mjs

@@ -872,20 +872,39 @@ function compileSelect(ast, env, outer, tail) {
   }
   // join rel (its columns `cols`) on outer key ASTs = rel key columns; the
   // rel's columns get fresh names; returns them
-  function attach(rel, outerKeys, relKeys, cols) {
+  // a subquery's columns looked up on the outer rows by key; in a grouped SELECT (ctx.step deferred),
+  // done after grouping, so the columns are there for the grouped rows
+  function attach(rel, outerKeys, relKeys, cols, ctx) {
     const tag = n.hidden++;
     const fresh = (c) => `__s${tag}_${c}`;
-    let rq = rel.query;
-    const all = [...new Set([...relKeys, ...cols])];
-    all.forEach((c) => { rq = rq.derive(fresh(c), c); });
-    rq = rq.select(...all.map(fresh));
-    let lefts = outerKeys.map((t) => hide(t));
-    let rights = relKeys.map(fresh);
-    if (!lefts.length) { const k = `__x${n.hidden++}`; q = q.derive(k, '0'); rq = rq.derive(k, '0'); lefts = [k]; rights = [k]; }
-    q = q.join(rq, { on: { left: lefts, right: rights }, how: 'left', view: true }); // (its keys are unique: right columns looked up, left rows not copied)
+    const run = () => {
+      let rq = rel.query;
+      const all = [...new Set([...relKeys, ...cols])];
+      all.forEach((c) => { rq = rq.derive(fresh(c), c); });
+      rq = rq.select(...all.map(fresh));
+      let lefts = outerKeys.map((t) => hide(t));
+      let rights = relKeys.map(fresh);
+      if (!lefts.length) { const k = `__x${n.hidden++}`; q = q.derive(k, '0'); rq = rq.derive(k, '0'); lefts = [k]; rights = [k]; }
+      q = q.join(rq, { on: { left: lefts, right: rights }, how: 'left', view: true }); // (its keys are unique: right columns looked up, left rows not copied)
+    };
+    if (ctx?.step) ctx.step({ name: fresh(cols[0]), inputs: [], run }); else run();
     return cols.map(fresh);
   }
-  function subqueryValue(e, ctx) {
+  // x NOT IN (subquery), as SQL has it: true where the subquery has no rows (x missing or not);
+  // else true only where x is present, not among the values, and none of them is missing
+  function notInValue(e, ctx, rel, vcol, mcol, relKeys, outerKeys) { // (rel: one row per value and key, mcol 1 on each)
+    const missing = rel.kinds?.get(vcol) === 'text' ? `${vcol} == ''` : `${vcol} != ${vcol}`;
+    let sq = rel.query;
+    let keys = relKeys;
+    if (!keys.length) { sq = sq.derive('__g', '0'); keys = ['__g']; }
+    sq = sq.groupBy(keys).agg({ __n: 'count()', __nn: `count(${missing})` });
+    const [cn, cnn] = attach({ ...rel, query: sq, names: [...keys, '__n', '__nn'] }, outerKeys, relKeys, ['__n', '__nn'], ctx);
+    const [m] = attach(rel, [translate(e.x, ctx), ...outerKeys], [vcol, ...relKeys], [mcol], ctx);
+    const isIn = bin('==', v(m), num(1));
+    const none = bin('==', call('coalesce', v(cn), num(0)), num(0));
+    return bin('||', none, bin('&&', bin('&&', present(e.x, ctx), { t: 'un', op: '!', x: isIn }), bin('==', call('coalesce', v(cnn), num(0)), num(0))));
+  }
+  function subqueryValue(e, ctx, negate = false) {
     if (e.k === 'exists') {
       const { inner, pairs } = correlate(e.q);
       const body = inner.body;
@@ -893,7 +912,7 @@ function compileSelect(ast, env, outer, tail) {
       const sel = { ...body, distinct: true, items: [...keyItems, { e: { k: 'num', v: 1 }, alias: '__m' }], groupBy: [] };
       if (body.groupBy?.length || body.having) fail('EXISTS (SELECT ... GROUP BY ...) is not supported');
       const rel = compileQuery({ ...inner, body: sel, orderBy: [], limit: pairs.length ? null : 1, offset: 0 }, env, outerHere);
-      const [m] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__m']);
+      const [m] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__m'], ctx);
       return bin('==', v(m), num(1));
     }
     if (e.k === 'insub') {
@@ -904,7 +923,8 @@ function compileSelect(ast, env, outer, tail) {
           const rel = compileQuery(inner, env, outerHere);
           if (rel.names.length !== 1) fail('IN (SELECT ...) needs exactly one column');
           const dist = { ...rel, query: rel.query.groupBy([rel.names[0]]).agg({ __m: 'count()' }).derive('__m1', '1') };
-          const [m] = attach({ ...dist, names: [rel.names[0], '__m1'] }, [translate(e.x, ctx)], [rel.names[0]], ['__m1']);
+          if (negate) return notInValue(e, ctx, { ...dist, names: [rel.names[0], '__m', '__m1'] }, rel.names[0], '__m1', [], []);
+          const [m] = attach({ ...dist, names: [rel.names[0], '__m1'] }, [translate(e.x, ctx)], [rel.names[0]], ['__m1'], ctx);
           return bin('==', v(m), num(1));
         }
         fail('IN (SELECT ...) needs exactly one column');
@@ -913,7 +933,8 @@ function compileSelect(ast, env, outer, tail) {
       const sel = { ...body, distinct: !body.groupBy?.length, items: [{ e: body.items[0].e, alias: '__v' }, ...keyItems, ...(body.groupBy?.length ? [] : [{ e: { k: 'num', v: 1 }, alias: '__m' }])] };
       let rel = compileQuery({ ...inner, body: sel, orderBy: [], limit: null, offset: 0 }, env, outerHere);
       if (body.groupBy?.length) { const keys = ['__v', ...keyItems.map((it) => it.alias)]; rel = { ...rel, query: rel.query.groupBy(keys).agg({ __c: 'count()' }).derive('__m', '1'), names: [...keys, '__m'] }; }
-      const [m] = attach(rel, [translate(e.x, ctx), ...pairs.map((pr) => translate(pr.outer, ctx))], ['__v', ...keyItems.map((it) => it.alias)], ['__m']);
+      if (negate) return notInValue(e, ctx, rel, '__v', '__m', keyItems.map((it) => it.alias), pairs.map((pr) => translate(pr.outer, ctx)));
+      const [m] = attach(rel, [translate(e.x, ctx), ...pairs.map((pr) => translate(pr.outer, ctx))], ['__v', ...keyItems.map((it) => it.alias)], ['__m'], ctx);
       return bin('==', v(m), num(1));
     }
     // (SELECT one value)
@@ -923,14 +944,14 @@ function compileSelect(ast, env, outer, tail) {
     if (!pairs.length) {
       const rel = compileQuery(inner, env, outerHere);
       if (rel.names.length !== 1) fail('a subquery used as a value needs exactly one column');
-      const [c] = attach(rel, [], [], [rel.names[0]]);
+      const [c] = attach(rel, [], [], [rel.names[0]], ctx);
       return v(c);
     }
     if (!(body.items.length === 1 && hasAgg(body.items[0].e)) || body.groupBy?.length) fail('a correlated subquery used as a value must compute one aggregate, e.g. (SELECT MAX(x) FROM t WHERE t.k = o.k)');
     const keyItems = pairs.map((pr, i) => ({ e: pr.inner, alias: `__k${i}` }));
     const sel = { ...body, items: [...keyItems, { e: body.items[0].e, alias: '__v' }], groupBy: keyItems.map((it) => it.e) };
     const rel = compileQuery({ ...inner, body: sel, orderBy: [], limit: null, offset: 0 }, env, outerHere);
-    const [c] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__v']);
+    const [c] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__v'], ctx);
     const it = body.items[0].e;
     return it.k === 'call' && it.fn === 'COUNT' ? call('coalesce', v(c), num(0)) : v(c); // (no rows: COUNT is 0)
   }
@@ -1099,9 +1120,19 @@ function compileSelect(ast, env, outer, tail) {
   // true where a value is there (SQL's NULL is NaN for numbers, '' for text); null: always there (a literal)
   function present(e, ctx) {
     if (e.k === 'num' || e.k === 'str' || e.k === 'date' || e.k === 'interval') return num(1);
+    if (isCond(e)) return bin('||', translate(e, ctx), negated(e, ctx)); // (a condition: known where it's true or false)
     const x = translate(e, ctx);
     return kindOf(e, ctx) === 'text' ? bin('!=', x, { t: 'str', v: '' }) : bin('==', x, x);
   }
+  // a true / false / unknown condition (comparisons, AND, OR, NOT, IS, IN, LIKE...): translate()
+  // gives where it's true, negated() where it's false; elsewhere it's unknown
+  function isCond(e) {
+    if (e.k === 'num') return !!e.bool;
+    if (e.k === 'bin') return ['AND', 'OR', 'IS', '=', '<>', '!=', '<', '<=', '>', '>=', '~', '~*', '!~', '!~*'].includes(e.op);
+    return ['not', 'isnull', 'like', 'in', 'between', 'exists', 'insub'].includes(e.k);
+  }
+  // a condition as a value: 1 true, 0 false, NaN unknown (SQL's NULL boolean)
+  function condValue(e, ctx, t) { return cond(t, num(1), cond(negated(e, ctx), num(0), num(NaN))); }
   // NOT x with SQL's three values: where x is unknown (a missing value), so is NOT x, and the
   // row is dropped. Comparisons become their opposites (false on missing values already).
   const FLIP = { '=': '<>', '<>': '=', '!=': '=', '<': '>=', '<=': '>', '>': '<=', '>=': '<' };
@@ -1118,7 +1149,8 @@ function compileSelect(ast, env, outer, tail) {
       case 'in':
         if (kindOf(x.x, ctx) === 'text') return bin('&&', { t: 'un', op: '!', x: T(x) }, present(x.x, ctx));
         return x.list.map((i) => T({ k: 'bin', op: '<>', l: x.x, r: i })).reduce((a, b) => bin('&&', a, b));
-      case 'like': case 'insub': return bin('&&', { t: 'un', op: '!', x: T(x) }, present(x.x, ctx));
+      case 'like': return bin('&&', { t: 'un', op: '!', x: T(x) }, present(x.x, ctx));
+      case 'insub': return subqueryValue(x, ctx, true);
       case 'exists': case 'isnull': return { t: 'un', op: '!', x: T(x) };
       default: { // a true / false value (1 / 0): NOT of a missing one stays missing
         const v = T(x);
@@ -1146,9 +1178,16 @@ function compileSelect(ast, env, outer, tail) {
       case 'exists': case 'insub': case 'subq': return subqueryValue(e, ctx);
       case 'window': return ctx.window(e);
       case 'bin': {
-        if (e.op === 'AND') return bin('&&', T(e.l), T(e.r));
-        if (e.op === 'OR') return bin('||', T(e.l), T(e.r));
+        const B = (x) => (x.k === 'null' ? num(0) : T(x)); // (a NULL condition is unknown: never true)
+        if (e.op === 'AND') return bin('&&', B(e.l), B(e.r));
+        if (e.op === 'OR') return bin('||', B(e.l), B(e.r));
         const op = { '=': '==', '<>': '!=' }[e.op] ?? e.op;
+        if (op === 'IS' && e.r.k === 'num' && e.r.bool) { // x IS TRUE / IS FALSE: never unknown (unknown is neither)
+          const x = e.l;
+          if (isCond(x)) return e.r.v ? T(x) : negated(x, ctx);
+          if (x.k === 'null') return num(0);
+          return e.r.v ? bin('&&', bin('!=', T(x), num(0)), present(x, ctx)) : bin('==', T(x), num(0));
+        }
         if (op === 'IS') { // a IS b: equal, or both missing
           const a = T(e.l), b = T(e.r);
           return bin('||', bin('==', a, b), bin('&&', call('isMissing', a), call('isMissing', b)));
@@ -1159,6 +1198,8 @@ function compileSelect(ast, env, outer, tail) {
           return op.startsWith('!') ? bin('==', m, num(0)) : bin('==', m, num(1));
         }
         if (['==', '!=', '<', '<=', '>', '>='].includes(op)) {
+          if (isCond(e.l) || isCond(e.r)) // (conditions compared: unknown if either one is)
+            return bin('&&', bin(op, T(e.l), T(e.r)), bin('&&', present(e.l, ctx), present(e.r, ctx)));
           const lk = kindOf(e.l, ctx), rk = kindOf(e.r, ctx);
           if (lk === 'text' || rk === 'text') {
             if ((lk === 'date' && e.r.k === 'str') || (rk === 'date' && e.l.k === 'str')) // a date written as text
@@ -1265,7 +1306,11 @@ function compileSelect(ast, env, outer, tail) {
             const s = bin('*', a[0], num(f));
             return bin('/', cond(bin('>=', s, num(0)), call('floor', bin('+', s, num(0.5))), call('ceil', bin('-', s, num(0.5)))), num(f));
           }
-          case 'LEAST': case 'GREATEST': if (a.length < 2) fail(`${e.fn}() needs two values or more`); return a.reduce((x, y) => call(e.fn === 'LEAST' ? 'min' : 'max', x, y));
+          case 'LEAST': case 'GREATEST': { // missing values skipped, as in PostgreSQL and DuckDB (NULL only if all are)
+            if (a.length < 2) fail(`${e.fn}() needs two values or more`);
+            const f = e.fn === 'LEAST' ? 'min' : 'max';
+            return a.reduce((x, y) => call(f, call('coalesce', x, y), call('coalesce', y, x)));
+          }
           case 'COALESCE': case 'IFNULL': case 'NVL': if (a.length < 2) fail(`${e.fn}() needs two values or more`); return call('coalesce', ...a);
           case 'NULLIF': need(2); return cond(bin('==', a[0], a[1]), num(NaN), a[0]);
           case 'DATE_TRUNC': {
@@ -1303,7 +1348,7 @@ function compileSelect(ast, env, outer, tail) {
   if (ast.where) {
     if (hasAgg(ast.where)) fail('aggregates (SUM, COUNT...) can\'t be used in WHERE: use HAVING');
     if (hasWindow(ast.where)) fail('window functions can\'t be used in WHERE (filter on them in an outer query)');
-    const t = translate(ast.where, plain);
+    const t = ast.where.k === 'null' ? num(0) : translate(ast.where, plain); // (WHERE NULL: unknown, no row)
     q = q.filter(src(t));
   }
 
@@ -1440,7 +1485,9 @@ function compileSelect(ast, env, outer, tail) {
     // aggregates, each once
     const aggs = new Map(); // sqlKey -> name
     const spec = {}, extraKeys = [];
-    const aggOf = (e) => {
+    const aggDone = new Map(); // (an aggregate used twice: the same expression, SUM's NULL check included)
+    const aggOf = (e) => { const k = key(e); if (!aggDone.has(k)) aggDone.set(k, aggOnce(e)); return aggDone.get(k); };
+    const aggOnce = (e) => {
       const k = key(e);
       if (aggs.has(k)) return v(aggs.get(k));
       const name = `__a${aggs.size}_${n.hidden++}`;
@@ -1480,8 +1527,9 @@ function compileSelect(ast, env, outer, tail) {
       return translate(e, post);
     };
     planWindows(translatePost, (x) => kindOf(x, post));
-    const outs = withCols(items.map((it) => ({ ...it, ast: textConst(translatePost(it.e), post) })));
-    const havingAst = ast.having ? translatePost(ast.having) : null;
+    const tri = (e, t) => { if (!isCond(e) || e.k === 'num' || t.t === 'var') return t; try { return condValue(e, post, t); } catch { return t; } }; // (built from GROUP BY expressions: as it is)
+    const outs = withCols(items.map((it) => { const t = translatePost(it.e); return { ...it, ast: textConst(tri(it.e, t), post) }; }));
+    const havingAst = ast.having ? (ast.having.k === 'null' ? num(0) : translatePost(ast.having)) : null;
     const orderAsts = orderBy.map((o) => ({ ...o, ast: orderTarget(o.e, outs, translatePost) }));
     q = q.groupBy([...keys.map((k) => k.name), ...extraKeys]).agg(Object.keys(spec).length ? spec : { __a_rows: 'count()' });
     const late = new Set(windowVars.values()), after = []; // (those using window values: after the windows)
@@ -1490,16 +1538,16 @@ function compileSelect(ast, env, outer, tail) {
     applyWindows();
     for (const st of after) st.run();
     for (const o of outs) if (o.ast.t !== 'var') q = q.derive(o.col, src(o.ast));
-    for (const o of orderAsts) sortKey(o);
+    for (const o of orderAsts) sortKey(o, orderKind(o, outs, post));
     return finish(outs, (o) => (o.e.k === 'col' ? post.kindOfCol(o.e) ?? kinds.get(resolveSafe(o.e)) : kindOf(o.e, post)), post);
   }
   const pre = windowCtx(plain);
   planWindows((e) => (e.k === 'col' ? colOf(e) : translate(e, pre)), (x) => kindOf(x, pre));
   applyWindows();
-  const outs = withCols(items.map((it) => ({ ...it, ast: textConst(it.e.k === 'col' ? colOf(it.e) : translate(it.e, pre), pre) })));
+  const outs = withCols(items.map((it) => { const t = it.e.k === 'col' ? colOf(it.e) : translate(it.e, pre); return { ...it, ast: textConst(isCond(it.e) && it.e.k !== 'num' ? condValue(it.e, pre, t) : t, pre) }; }));
   const orderAsts = orderBy.map((o) => ({ ...o, ast: orderTarget(o.e, outs, (e) => (e.k === 'col' ? colOf(e) : translate(e, pre))) }));
   for (const o of outs) if (o.ast.t !== 'var') q = q.derive(o.col, src(o.ast));
-  for (const o of orderAsts) sortKey(o);
+  for (const o of orderAsts) sortKey(o, orderKind(o, outs, pre));
   if (ast.distinct) q = q.groupBy([...new Set(outs.map((o) => o.col))]).agg({ __distinct_rows: 'count()' });
   return finish(outs, (o) => (o.e.k === 'col' ? (sessionRef(o.e)?.kind ?? kinds.get(o.e.resolved ? o.e.name : resolve(o.e))) : kindOf(o.e, pre)), pre);
 
@@ -1516,11 +1564,17 @@ function compileSelect(ast, env, outer, tail) {
     const t = tr(e), same = outs.find((o) => toSourceAst(o.ast) === toSourceAst(t));
     return same ? v(same.col) : t;
   }
-  function sortKey(o) {
+  // the kind (text or not) of what an ORDER BY item sorts: a select item by number or name, or the expression
+  function orderKind(o, outs, ctx) {
+    const e = o.e.k === 'num' ? outs[o.e.v - 1]?.e : (o.e.k === 'col' && !o.e.table && outs.find((x) => x.name === o.e.name)?.e) || o.e;
+    try { return kindOf(e, ctx); } catch { return null; }
+  }
+  function sortKey(o, kind) {
     let k;
     if (o.ast.t === 'var') k = o.ast.name;
     else { k = `__o${n.hidden++}`; q = q.derive(k, src(o.ast)); }
-    if (o.nullsFirst) { const h = `__nf${n.hidden++}`; q = q.derive(h, `${k} == ${k} ? 1 : 0`); sortKeys.push(h); }
+    // missing sorts last on its own (NaN, and '' for text); NULLS FIRST sorts present values after missing ones first
+    if (o.nullsFirst) { const h = `__nf${n.hidden++}`; q = q.derive(h, kind === 'text' ? `${k} == '' ? 0 : 1` : `${k} == ${k} ? 1 : 0`); sortKeys.push(h); }
     sortKeys.push(o.desc ? `-${k}` : k);
   }
   function finish(outs, kindFn, ctx) {

package/src/table.mjs

@@ -7,7 +7,7 @@
 
 import { parseKeys, labelRanks, sortIndex } from './sort.mjs';
 import { ZONE_ROWS, computeZones, zoneRange } from './zones.mjs';
-import { parse, toSource, walk, analyze, isMask } from './fuse.mjs';
+import { parse, toSource, walk, analyze, isMask, fold } from './fuse.mjs';
 import { generateGroupBy, generateCompact, generateHashGroups, generateMarkPairs, generateStridedKeys } from './table-kernels.mjs';
 import { decodeString } from './csv-kernel.mjs';
 import { maxUnits } from './kernels.mjs';
@@ -107,17 +107,18 @@ export class Table {
     for (const name of Object.keys(rows[0])) {
       checkName(name);
       const data = engine.alloc(type, n);
-      if (typeof rows[0][name] === 'string') {
+      const first = firstPresent(rows.map((r) => r[name]));
+      if (typeof first === 'string') {
         const codes = new Map(), dict = [];
         for (let i = 0; i < n; i++) {
-          const s = rows[i][name];
+          const s = rows[i][name] ?? ''; // (null / undefined: missing text)
           let c = codes.get(s);
           if (c === undefined) { c = dict.length; dict.push(s); codes.set(s, c); }
           data[i] = c;
         }
         cols.set(name, { data, dict, codes });
       } else {
-        const date = rows[0][name] instanceof Date;
+        const date = first instanceof Date;
         if (date && type === Float32Array) throw new Error(f32Dates(name));
         for (let i = 0; i < n; i++) { const v = rows[i][name]; data[i] = v === null || v === undefined ? NaN : date ? toMillis(v) : v; } // (null: missing)
         cols.set(name, { data, dict: null, codes: null, ...(date ? { date: true } : {}) });
@@ -314,7 +315,14 @@ export class Table {
   // Lazy filter: returns a view of the same columns with a condition attached.
   filter(expr, params = {}) {
     const merged = this.#mergeParams(params);
-    const { src } = this.#bind(expr, merged);
+    let { ast, src } = this.#bind(expr, merged);
+    walk(ast, (m) => { if (m.t === 'str') throw new SyntaxError(`vectro: text like ${JSON.stringify(m.v)} can only be compared with == or != to a text column, in "${expr}" (to compare dates, load the column as dates: { dates: ['name'] })`); });
+    const k = fold(ast);
+    if (k.t === 'num') { // no column in it (1 == 1, 0): every row, or none (a kernel needs a column to read)
+      if (k.v !== 0) return new Table(this.engine, this.cols, this.n, { pred: this.pred, params: merged, zones: this.zones });
+      const c = [...this.cols.keys()].find((x) => !this.cols.get(x).strings);
+      src = `${c} != ${c} && ${c} == ${c}`;
+    }
     const pred = this.pred ? `(${this.pred}) && (${src})` : src;
     return new Table(this.engine, this.cols, this.n, { pred, params: merged, zones: this.zones });
   }
@@ -1891,7 +1899,9 @@ function codedText(engine, type, { codes: src, labels }) {
 }
 
 const isDates = (values) => Array.isArray(values) && values.find((v) => v != null) instanceof Date;
-const isText = (values) => !ArrayBuffer.isView(values) && values.length > 0 && typeof values[0] === 'string';
+// the first value present decides (a column may start with null / undefined: missing)
+const firstPresent = (values) => { for (const v of values) if (v !== null && v !== undefined) return v; return undefined; };
+const isText = (values) => !ArrayBuffer.isView(values) && values.length > 0 && typeof firstPresent(values) === 'string';
 
 function copyNumbers(engine, type, values) {
   const data = engine.alloc(type, values.length);
@@ -1904,7 +1914,7 @@ function encodeText(engine, type, values) {
   const data = engine.alloc(type, values.length);
   const codes = new Map(), dict = [];
   for (let i = 0; i < values.length; i++) {
-    const s = values[i];
+    const s = values[i] ?? ''; // (null / undefined: missing text, as in JSON)
     let c = codes.get(s);
     if (c === undefined) { c = dict.length; dict.push(s); codes.set(s, c); }
     data[i] = c;