vectrojs 0.1.0 → 0.1.2

package/CHANGELOG.md

@@ -1,5 +1,59 @@
 # Changelog
 
+## 0.1.2
+
+A correctness release, from comparing about 1,700 new SQL queries with DuckDB
+on the same data. No API changes. Every case below now gives DuckDB's answer,
+or a clear error.
+
+**Engine**
+- A query too big for the 4GB memory (a join with billions of output rows)
+  now fails with `out of memory (4GB limit)`. Before, the size wrapped around
+  in the allocator, and later queries on the same engine could return wrong
+  numbers.
+- Missing text (`''`) sorts last, up or down, as missing numbers do, in
+  `sort`, `top`, window ordering and SQL `ORDER BY`.
+- `null` and `undefined` in a text column load as missing text, also when
+  the column starts with one (`engine.table`, `tableFromRows`).
+- A missing text key never matches in a join, as the docs say.
+- A filter with no column in it (`'1'`, `'1 == 0'`) keeps every row or none,
+  instead of failing.
+- Comparing a text column with `<` or `>` and then grouping gives a clear
+  error instead of overflowing the stack. On datasets, the unknown-column
+  error lists every column.
+
+**SQL**
+- Conditions follow SQL's three values everywhere: a condition on a missing
+  value is NULL when selected, `IS TRUE` / `IS FALSE` treat it as neither,
+  comparing two conditions is unknown if either is, and a `NULL` literal in
+  `AND` / `OR` is unknown rather than true.
+- `WHERE TRUE`, `WHERE 1 = 1`, `WHERE NULL` and other constant conditions work.
+- `ORDER BY text NULLS FIRST` works.
+- `x NOT IN (SELECT ...)` keeps every row when the subquery is empty and no
+  row when it contains a NULL.
+- `LEAST` and `GREATEST` skip missing values (NULL only if all are).
+- The same aggregate twice (`SUM(x) AS a, SUM(x) AS b`) gives NULL both times
+  over no values.
+- Subqueries in the SELECT list or HAVING of a GROUP BY query work.
+
+## 0.1.1
+
+A performance release. No API changes.
+
+**Faster group-by**
+- SQL `SUM` no longer computes a hidden count of non-missing values when the
+  column has none missing, and sum, mean, min and max skip the per-row
+  missing-value check on such columns.
+- On many threads, a group-by remembers its key range and merges the
+  threads' partial results with a WebAssembly kernel instead of a JavaScript
+  loop. This also removes the 31-thread limit for queries with many groups.
+- Measured on 2M orders (`SELECT product_id, sum(amount) ... GROUP BY 1`,
+  `compare/bench-2m-groupby.mjs`, numbers in `compare/results.json`): with
+  1,000 products, 3.0 ms on 1 thread (DuckDB 2.0: 6.8 ms) and 2.3 ms on 8
+  threads (DuckDB 2.0: 2.4 ms). Before this release Vectro was 1.9x slower
+  than DuckDB on 8 threads in this case; it is now level or faster at every
+  size from 100 to 100,000 products.
+
 ## 0.1.0 (first release)
 
 The first public release. Everything below is new.

package/README.md

@@ -5,7 +5,7 @@ and compiles each query, filter included, into one SIMD WebAssembly loop, then
 runs it on every CPU core. It runs in Node, in browsers (any page) and on
 Cloudflare Workers, with no native addon, compiler or server.
 
-> Version 0.1.0, the first release. Node 20+ and modern browsers. MIT license.
+> Version 0.1.2. Node 20+ and modern browsers. MIT license.
 
 **Website and docs: [vectrojs.dev](https://vectrojs.dev)**. [Quick start](https://vectrojs.dev/docs/quick-start) · [API reference](https://vectrojs.dev/api) · [Race it in your browser](https://vectrojs.dev/live) · [Benchmarks](https://vectrojs.dev/benchmarks)
 
@@ -30,18 +30,27 @@ orders.groupBy('region').agg({ revenue: 'sum(price * qty)' }).toRows();
 npm install vectrojs
 ```
 
+Save this as `try.mjs` (or set `"type": "module"` in package.json) and run `node try.mjs`:
+
 ```js
 import { Engine, sql } from 'vectrojs';
 
-const engine = new Engine();
-const orders = engine.readCSV('orders.csv');
-sql('SELECT region, COUNT(*) AS n, SUM(price) AS revenue FROM orders GROUP BY region ORDER BY revenue DESC', { orders }).toRows();
+const engine = new Engine();                         // uses every core
+const orders = engine.table({
+  region: ['EU', 'US', 'EU', 'APAC', 'US'],
+  price:  [120, 80, 300, 45, 99],
+});
+console.log(sql('SELECT region, COUNT(*) AS n, SUM(price) AS revenue FROM orders GROUP BY region ORDER BY revenue DESC', { orders }).toRows());
+// [ { region: 'EU', n: 2, revenue: 420 }, { region: 'US', n: 2, revenue: 179 }, { region: 'APAC', n: 1, revenue: 45 } ]
+await engine.close();
 ```
 
+For your own data, `engine.readCSV('orders.csv')` reads a CSV file on every core.
+
 Serve a folder of CSV / Arrow / `.vectro` files to BI tools, with no code:
 
 ```sh
-npx vectro serve --root ./data --pg 5433      # HTTP on 8787, PostgreSQL on 5433
+npx -y -p vectrojs vectro serve --root ./data --pg 5433      # HTTP on 8787, PostgreSQL on 5433
 psql -h 127.0.0.1 -p 5433 -U vectro vectro     # or Power BI, Tableau, Metabase, Grafana
 ```
 
@@ -82,10 +91,10 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
 
   | Keys | Vectro | DuckDB |
   |---|---|---|
-  | region + day (2,920 pairs) | 8 ms | 34 ms |
-  | price (100K amounts) | 23 ms | 42 ms |
-  | account (1M big IDs) | 29 ms | 61 ms |
-  | account + region (5.7M pairs) | 42 ms | 115 ms |
+  | region + day (2,920 pairs) | 6 ms | 39 ms |
+  | price (100K amounts) | 27 ms | 49 ms |
+  | account (1M big IDs) | 36 ms | 88 ms |
+  | account + region (5.7M pairs) | 50 ms | 139 ms |
 
   Vectro's groups also come back sorted by key; DuckDB's come back unsorted.
 - **Missing values** are NaN for numbers and `''` for text. `sum`, `mean`,
@@ -124,9 +133,9 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
   by the hash table first, or, with many of them, sorted by key range on
   every thread (as a group-by with many groups) and counted in runs; values
   a whole step apart (IDs like 1e9 + 7919k) get the byte map too. On 10M rows
-  vs DuckDB (`compare/bench-distinct.mjs`): 365 days 4 ms vs 4; regions per
-  day 7 ms vs 31; 1M accounts 9 ms vs 44 (1M random IDs up to 2^40, no common
-  step: 24 ms vs 46); accounts per region (5.7M pairs) 30 ms vs 85.
+  vs DuckDB (`compare/bench-distinct.mjs`): 365 days 5 ms vs 5; regions per
+  day 9 ms vs 36; 1M accounts 11 ms vs 51 (1M random IDs up to 2^40, no common
+  step, from an earlier run: 24 ms vs 46); accounts per region (5.7M pairs) 35 ms vs 103.
 - **Compressed columns:** `table.compress()` (or `engine.table(cols, { compress:
   true })`, `parseCSV(text, { compress: true })`) stores each number and
   category column in 1, 2 or 4 bytes per row when every value comes back
@@ -198,11 +207,11 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
   dates, and comparing text with `<` / `>`. Each function runs once per
   distinct value, not per row (a category column has few labels), and the
   answer is a category column again, so it groups, sorts and joins like any
-  other: `GROUP BY UPPER(region)` on 10M rows 30 ms (DuckDB 42), `region ||
-  '-' || qty` as a key 85 ms (92), `STRFTIME(ts, '%Y-%m')` 48 ms (61; times
-  are rounded to the month first), ILIKE on a column 29 ms (46). A CASE whose
+  other: `GROUP BY UPPER(region)` on 10M rows 31 ms (DuckDB 47), `region ||
+  '-' || qty` as a key 85 ms (106), `STRFTIME(ts, '%Y-%m')` 51 ms (70; times
+  are rounded to the month first), ILIKE on a column 30 ms (54). A CASE whose
   results are all quoted labels is worked out as label codes in the fused
-  pass (CASE text + group 5.7 ms vs 21). Missing text
+  pass (CASE text + group 6.2 ms vs 25). Missing text
   is `''`, as everywhere in Vectro, and positions count characters from 1 as
   in DuckDB.
 
@@ -210,7 +219,7 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
   rows (FULL: plus the left rows without a match) followed by the right rows
   whose keys no left row has. The join marks which right rows it matched as
   it probes, so the rows without a match come from the same pass (RIGHT JOIN
-  + group 27 vs 54 ms; FULL JOIN + count 17 vs 26 ms).
+  + group 30 vs 64 ms; FULL JOIN + count 19 vs 29 ms).
 
   `compare/check-sql.mjs` runs 40 such queries (CTEs, set operations,
   correlated subqueries, windows, percentiles, text functions on awkward
@@ -219,15 +228,15 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
   correlation on anything but equalities, MIN / MAX of text, and STRING_AGG.
   The MCP server has
   a `sql` tool. The same SQL text vs DuckDB on 10M orders + 100K customers
-  (`compare/bench-sql.mjs`, answers checked, one run): Vectro is faster on all
-  20 queries: filtered totals 3.4 vs 10.1 ms, group by region 7.8 vs
-  30.7 ms, join + filter + group 17.3 vs 29.0 ms, top 10 customers 12.9 vs
-  42.5 ms, distinct customers per month 30.8 vs 70.6 ms, top 20 rows 8.2 vs
-  22.3 ms, CASE inside SUM 7.6 vs 35.2 ms, IN (subquery) 19.7 vs 25.2 ms,
-  correlated subquery 16.0 vs 50.6 ms, top 3 per region with RANK 36 vs
-  119 ms, ROW_NUMBER over all rows 124 vs 179 ms, running total by day 8.6
-  vs 14.6 ms, MEDIAN per region 33.6 vs 71.7 ms, FULL JOIN + count 16.9 vs
-  25.5 ms. A filter like `rn <= 3` on ROW_NUMBER or RANK keeps
+  (`compare/bench-sql.mjs`, answers checked, best of 5): Vectro is faster on all
+  20 queries (1.3–6.0x): filtered totals 4.3 vs 10.1 ms, group by region 7.4 vs
+  36.8 ms, join + filter + group 19.8 vs 32.8 ms, top 10 customers 13.1 vs
+  50.3 ms, distinct customers per month 34.2 vs 82.5 ms, top 20 rows 9.1 vs
+  25.3 ms, CASE inside SUM 6.6 vs 39.6 ms, IN (subquery) 23.4 vs 29.7 ms,
+  correlated subquery 16.8 vs 61.4 ms, top 3 per region with RANK 38 vs
+  139 ms, ROW_NUMBER over all rows 129 vs 221 ms, running total by day 9.8
+  vs 17.3 ms, MEDIAN per region 39.1 vs 83.9 ms, FULL JOIN + count 19.3 vs
+  29.3 ms. A filter like `rn <= 3` on ROW_NUMBER or RANK keeps
   only the rows that can rank that high before sorting (picked on every
   thread); percentiles of a few partitions count values in buckets on every
   thread and select only within the wanted buckets. Window functions and
@@ -257,10 +266,10 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
 
   | Join | Vectro | DuckDB |
   |---|---|---|
-  | × 100K customers | 65 ms | 222 ms |
-  | × 1M accounts | 135 ms | 255 ms |
-  | left join, half the keys match | 76 ms | 191 ms |
-  | two keys: region + day (2,920 rows) | 64 ms | 167 ms |
+  | × 100K customers | 77 ms | 210 ms |
+  | × 1M accounts | 151 ms | 251 ms |
+  | left join, half the keys match | 82 ms | 206 ms |
+  | two keys: region + day (2,920 rows) | 61 ms | 189 ms |
 - **Text functions** on category columns: `startsWith(col, 'x')`,
   `endsWith`, `contains`, `in(col, 'a', 'b')` and `like(col, 'EU-%')`. They run
   once per distinct label (not per row) and become a few range checks on the
@@ -282,8 +291,8 @@ What changed: [CHANGELOG.md](CHANGELOG.md). Problems and ideas:
   of 50,000+ rows run on every core: a parallel radix sort (11-bit digits,
   digits that never change are skipped, whole numbers sort as offsets), then
   every column is gathered in the new order on every core. Sorting 10M rows
-  by `-price` takes ~130 ms (one thread: ~350 ms; a JavaScript comparator
-  sort: ~4.6 s).
+  by `-price` takes ~117 ms (one thread: ~350 ms; a JavaScript comparator
+  sort: ~4.1 s).
 - **Async API:** every query has an `…Async` version that runs on the worker
   threads while your server or page keeps working.
 - **Expressions** are a small, safe language: numbers, columns, `+ - * /`,
@@ -383,31 +392,33 @@ formulas exact in both, answers compared; best of 3):
 
 | Query | 100M rows: Vectro | DuckDB | 1 billion rows: Vectro | DuckDB |
 |---|---|---|---|---|
-| scan everything (count, revenue) | 49 ms | 61 ms | 460 ms | 648 ms |
-| filter: region and quantity | 70 ms | 72 ms | 665 ms | 723 ms |
-| the last day (1 chunk read) | 3 ms | 1 ms | 3 ms | 5 ms |
-| group by region | 114 ms | 124 ms | 1.13 s | 1.24 s |
-| top 10 of 1M customers | 460 ms | 487 ms | 4.46 s | 4.62 s |
-| top 10 rows by price | 10 ms | 87 ms | 11 ms | 827 ms |
-| join 1M customers, group by tier | 84 ms | 68 ms | 745 ms | 587 ms |
-| distinct customers per region | 247 ms | 517 ms | 2.45 s | 6.50 s |
+| scan everything (count, revenue) | 70 ms | 73 ms | 478 ms | 662 ms |
+| filter: region and quantity | 100 ms | 87 ms | 691 ms | 729 ms |
+| the last day (1 chunk read) | 3 ms | 1 ms | 3 ms | 4 ms |
+| group by region | 142 ms | 149 ms | 1.05 s | 1.25 s |
+| top 10 of 1M customers | 529 ms | 637 ms | 4.39 s | 4.50 s |
+| top 10 rows by price | 15 ms | 100 ms | 11 ms | 843 ms |
+| join 1M customers, group by tier | 126 ms | 85 ms | 880 ms | 627 ms |
+| distinct customers per region | 268 ms | 650 ms | 2.43 s | 6.84 s |
 | file on disk | 0.5 GB | 0.5 GB | 5.1 GB | 5.4 GB |
-| peak memory (RSS) | 343 MB | 3.1 GB | **0.37 GB** | 6.6 GB |
+| peak memory (RSS) | 344 MB | 2.9 GB | **0.37 GB** | 6.9 GB |
 
 A billion rows would take 37 GB as plain columns in memory (this laptop has
-16 GB); Vectro's engine never used more than 163 MB for them. Vectro is level
-or faster on 7 of the 8 queries. Top rows read only the blocks of rows whose
+16 GB); Vectro's engine never used more than 163 MB for them. At a billion
+rows Vectro is level or faster on 7 of the 8 queries (only the join is
+slower, 1.4x); at 100M rows it is slower on three: the filter (1.1x), the
+last day (3 vs 1 ms) and the join (1.5x). Top rows read only the blocks of rows whose
 stored min / max can still beat the rows kept so far (here 1 chunk of 239),
 so they take the same few milliseconds at any size. A join whose right side
 has unique whole-number keys and a GROUP BY on its columns looks each left
 key's group up in one array instead of joining rows; DuckDB is still faster
-there (1.3x).
+there (1.4x at a billion rows, 1.5x at 100M).
 
 ## SQL over HTTP: Excel, Power BI, Google Sheets, pandas
 
 ```sh
-npx vectro serve --root ~/data        # every CSV, TSV, JSON-lines, Arrow / Feather and .vectro file is a table
-curl 'http://127.0.0.1:8787/sql?q=SELECT region, COUNT(*) AS n FROM orders GROUP BY 1&format=csv'
+npx -y -p vectrojs vectro serve --root ~/data        # every CSV, TSV, JSON-lines, Arrow / Feather and .vectro file is a table
+curl -G http://127.0.0.1:8787/sql --data-urlencode 'q=SELECT region, COUNT(*) AS n FROM orders GROUP BY 1' -d format=csv
 ```
 
 | Endpoint | Gives |
@@ -439,7 +450,7 @@ On 10M orders: a group by region answers in 9 ms over HTTP; 1M rows export in 0.
 ## The PostgreSQL protocol: Power BI, Tableau, Metabase, Grafana, psql
 
 ```sh
-npx vectro serve --root ~/data --pg 5433 --user me --password secret
+npx -y -p vectrojs vectro serve --root ~/data --pg 5433 --user me --password secret
 psql -h 127.0.0.1 -p 5433 -U me vectro
 ```
 
@@ -494,7 +505,7 @@ ODBC, and every driver. Queries arrive in each tool's own SQL and run on Vectro'
 ## Arrow Flight SQL: pandas, Polars, DuckDB
 
 ```sh
-npx vectro serve --root ~/data --flight 31337            # grpc://localhost:31337 (--user / --password, --tls-cert / --tls-key)
+npx -y -p vectrojs vectro serve --root ~/data --flight 31337            # grpc://localhost:31337 (--user / --password, --tls-cert / --tls-key)
 ```
 
 ```python
@@ -589,9 +600,9 @@ no index to build or tune, results identical to brute force.
 
 | Embeddings | Vectro | Plain JS loop (Float32Array) |
 |---|---|---|
-| 100,000 (154 MB) | 11.7 ms | 35.9 ms (3.1×) |
-| 200,000 (307 MB) | 13.8 ms | 71.5 ms (5.2×) |
-| 500,000 (768 MB) | 22.8 ms | 179.3 ms (7.9×) |
+| 100,000 (154 MB) | 9.3 ms | 36.6 ms (3.9×) |
+| 200,000 (307 MB) | 13.6 ms | 72.6 ms (5.3×) |
+| 500,000 (768 MB), an earlier run | 22.8 ms | 179.3 ms (7.9×) |
 
 With a `SplitEngine`, every worker searches its own rows in one round trip
 and the page keeps the best k. In Chrome (`examples/check/vectors.html?n=200000`),
@@ -727,6 +738,7 @@ writeFileSync('eu.feather', sales.filter("region == 'EU'").toArrow({ file: true
 // split tables: await engine.fromArrowAsync(bytes), await table.toArrowAsync()
 
 // streams: any Arrow IPC stream over TCP or WebSocket, e.g. from pyarrow
+const split = new SplitEngine();                                     // import { SplitEngine } from 'vectrojs'
 const orders = await split.streamAsync({ format: 'arrow', columns: { ts: 'number', region: 'category' } });
 // ...or Vectro's Emitter speaking Arrow: Emitter.listen({ columns, format: 'arrow' })
 ```
@@ -763,25 +775,30 @@ iPhone) and Cloudflare Workers.
 
 | Query | Vectro | DuckDB 2.0 | SQLite* | Arquero |
 |---|---|---|---|---|
-| Filtered revenue | 3.3 ms | 9.9 ms | 757 ms | 65 ms |
-| Dashboard, 4 aggregates | 3.6 ms | 22.8 ms | 345 ms | 169 ms |
-| Group by region (8 groups) | 7.7 ms | 35.1 ms | 2.75 s | 1.02 s |
-| Group by customer (100K groups) | 11.0 ms | 11.7 ms | 4.32 s | 723 ms |
-| Filter into a new table | 5.4 ms | 59.6 ms | | 123 ms |
-| New column | 3.3 ms | 25.6 ms | 1.75 s | 114 ms |
+| Filtered revenue | 3.2 ms | 10.1 ms | 772 ms | 74 ms |
+| Dashboard, 4 aggregates | 4.1 ms | 23.9 ms | 359 ms | 171 ms |
+| Group by region (8 groups) | 6.5 ms | 39.7 ms | 2.59 s | 1.00 s |
+| Group by customer (100K groups) | 8.1 ms | 13.2 ms | 3.90 s | 772 ms |
+| Filter into a new table | 6.4 ms | 55.7 ms | 540 ms | 124 ms |
+| New column | 3.7 ms | 32.3 ms | 1.50 s | 125 ms |
 
 Best of 5, each system using all 8 cores (Arquero is single-threaded), the
-same answers from every system. \*SQLite from an earlier run
-(`npm run bench:sqlite`).
+same answers from every system. Vectro is 1.6–8.7x faster than DuckDB here
+(3–9x on five of the six, 1.6x with 100K groups). \*SQLite from a separate
+run (`npm run bench:sqlite`, best of 3).
 
 Run them yourself: `cd compare && npm install && npm run bench`. Timings vary
 with the machine's state (battery, background load), so compare systems in the
-same run. More comparisons in `compare/`: SQL (`bench-sql.mjs`: Vectro faster
+same run. The numbers in this README were measured on 1 November 2026
+(commit b2a3033). These runs shared the laptop with other jobs (load average 3–8 for
+the in-memory runs, up to 15 for the billion-row run), and ratios move by up
+to about ±30% between runs. More comparisons in `compare/`: SQL (`bench-sql.mjs`: Vectro faster
 on all 20 queries), joins (`bench-join.mjs`: faster on all 4), group-by
-(`bench-groupby.mjs`: faster on all 4, 2.1–2.7x with millions of groups),
+(`bench-groupby.mjs`: faster on all 4, 2.4–2.8x with millions of groups),
 distinct counts (`bench-distinct.mjs`: level or faster on all 4). On files
 on disk (`bench-dataset.mjs`) DuckDB is still faster at joins with a GROUP
-BY on the joined table (1.2–1.3x).
+BY on the joined table (1.4–1.5x), and at 100M rows also at the filter (1.1x)
+and the last-day lookup (3 vs 1 ms).
 
 ## Layout
 

package/mcp/server.mjs

@@ -14,7 +14,7 @@ import { createInterface } from 'node:readline';
 import { Engine, sql } from '../src/node.mjs';
 import { LoadError, safePath as safePathIn, loadFile, columnsOf, kindOf } from '../src/serve/catalog.mjs';
 
-const VERSION = '0.1.0';
+const VERSION = '0.1.2';
 const PROTOCOLS = ['2025-06-18', '2025-03-26', '2024-11-05'];
 const MAX_ROWS = 500, DEFAULT_ROWS = 50;
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vectrojs",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Multi-core analytics engine for JavaScript: columnar tables, SQL and fused SIMD WebAssembly kernels for Node, browsers and edge runtimes, with no native code. Serves tables to Power BI, Tableau, Metabase, Grafana, pandas and Polars.",
   "type": "module",
   "license": "MIT",

package/src/dataset.mjs

@@ -834,7 +834,12 @@ export function* runPlanG(plan, sink = null) {
     for (let n = plan; n.op !== 'scan' && n.op !== 'union'; ) { n = n.op === 'join' ? n.left : n.input; path.push(n); }
     const scan = path[path.length - 1];
     if (!scan.table?.isDataset) result = yield { run: plan };
-    else result = yield* runChunksG(plan, path, sink);
+    else {
+      try { result = yield* runChunksG(plan, path, sink); } catch (err) { // (chunks are read with only the columns used: name them all)
+        if (/^vectro: unknown column /.test(err?.message)) err.message = err.message.replace(/\(columns: [^)]*\)$/, `(columns: ${scan.table.columnNames.join(', ')})`);
+        throw err;
+      }
+    }
     if (sink && result) { // (the result itself was built in memory)
       if (result.cols) sink.append(result);
       else sink.append(Object.fromEntries(Object.entries(result).map(([k, v]) => [k, [v]])));

package/src/engine.mjs

@@ -32,7 +32,7 @@ const DEFAULT_MIN_PARALLEL_WASM = {
 const PAGE = 65536;
 const MAX_PAGES = 65536; // 4GB, the most a 32-bit WASM address can reach
 const HEAP_START = 64;   // keep address 0 unused
-const align16 = (n) => (n + 15) & ~15;
+const align16 = (n) => Math.ceil(n / 16) * 16; // (not bitwise: sizes may pass 2^31 bytes, and must not wrap)
 
 // Can this runtime compile WebAssembly from bytes? (Cloudflare Workers and
 // Vercel Edge can't: they only run modules bundled ahead of time.)
@@ -141,6 +141,8 @@ export class Engine {
   // WASM SIMD kernels; other shared arrays fall back to JS kernels.
   alloc(Type, n) {
     const size = align16(Math.max(16, n * Type.BYTES_PER_ELEMENT));
+    // too big for the 4GB memory (a huge join output, say): fail before touching the free list
+    if (!(size <= MAX_PAGES * PAGE - 64)) throw new RangeError('vectro: out of memory (4GB limit)');
     let ptr = -1;
     for (let i = 0; i < this.freeList.length; i++) {
       const blk = this.freeList[i];

package/src/fuse.mjs

@@ -125,6 +125,12 @@ export function walk(n, f) {
   else if (n.t === 'cond') { walk(n.c, f); walk(n.x, f); walk(n.y, f); }
 }
 
+// text left in an expression a kernel computes (a text column was compared with < or >, or
+// isn't one): a clear error, not a loop between the value and condition emitters
+function noText(n) {
+  if (n.t === 'str') throw new SyntaxError(`vectro: text like ${JSON.stringify(n.v)} can only be compared with == or != to a text column (to compare dates, load the column as dates: { dates: ['name'] })`);
+}
+
 function checkElementwise(n, src) {
   walk(n, (m) => {
     if (m.t === 'str') throw new SyntaxError(`vectro: text like ${JSON.stringify(m.v)} can only be compared (== / !=) with a text column, in "${src}"`);
@@ -162,7 +168,10 @@ export function analyze(spec) {
 
 // Constant parts computed now: 2 * 3 -> 6, floor(2.5) -> 2, (1 ? x : y) -> x.
 // Same results as the kernels (IEEE doubles; f32 kernels round literals).
-const FOLD_BIN = { '+': (a, b) => a + b, '-': (a, b) => a - b, '*': (a, b) => a * b, '/': (a, b) => a / b };
+const FOLD_BIN = { '+': (a, b) => a + b, '-': (a, b) => a - b, '*': (a, b) => a * b, '/': (a, b) => a / b,
+  // as the kernels compute them: 1 / 0, a comparison with NaN false (but != true), NaN counting as true in && and ||
+  '==': (a, b) => +(a === b), '!=': (a, b) => +(a !== b), '<': (a, b) => +(a < b), '<=': (a, b) => +(a <= b), '>': (a, b) => +(a > b), '>=': (a, b) => +(a >= b),
+  '&&': (a, b) => +(a !== 0 && b !== 0), '||': (a, b) => +(a !== 0 || b !== 0) };
 const FOLD_FN = { abs: Math.abs, sqrt: Math.sqrt, floor: Math.floor, ceil: Math.ceil, min: Math.min, max: Math.max };
 export function fold(n) {
   const isNum = (x) => x.t === 'num';
@@ -255,8 +264,8 @@ export class Emitter {
     this.memo.set(m, l);
   }
 
-  vec(n, k) { return this.#reuse(n, k, V128, () => this.#vec(n, k)); }
-  sca(n) { return this.#reuse(n, 'sca', this.ty.val, () => this.#sca(n)); }
+  vec(n, k) { noText(n); return this.#reuse(n, k, V128, () => this.#vec(n, k)); }
+  sca(n) { noText(n); return this.#reuse(n, 'sca', this.ty.val, () => this.#sca(n)); }
 
   #vec(n, k) {
     const { f, env } = this, V = VEC[this.t];

package/src/join.mjs

@@ -415,7 +415,7 @@ function* buildRight(engine, t, L, R, lk, rk, how) {
     for (let k = 0; k < nk; k++) {
       const a = L.cols.get(lk[k]), b = R.cols.get(rk[k]);
       if (!b.dict) { build.rightKeys.push(b.data); continue; }
-      const lut = b.dict.map((label) => a.codes?.get(label) ?? NaN); // a label the left table lacks matches nothing
+      const lut = b.dict.map((label) => (label === '' ? NaN : a.codes?.get(label) ?? NaN)); // missing text, or a label the left table lacks, matches nothing
       const mapped = alloc(L.type, nR);
       const src = engine.live(b.data), dst = engine.live(mapped);
       for (let i = 0; i < nR; i++) dst[i] = lut[src[i]];

package/src/sort.mjs

@@ -22,10 +22,11 @@ export function parseKeys(by, cols) {
 }
 
 // Ranks for a category column's codes: code -> position of its label A..Z.
+// Missing text ('') ranks NaN, so it comes last either way, as a missing number does.
 export function labelRanks(dict, desc) {
   const order = dict.map((_, i) => i).sort((a, b) => (dict[a] < dict[b] ? -1 : dict[a] > dict[b] ? 1 : 0));
   const rank = new Float64Array(dict.length);
-  order.forEach((code, r) => { rank[code] = desc ? -r : r; });
+  order.forEach((code, r) => { rank[code] = dict[code] === '' || dict[code] == null ? NaN : desc ? -r : r; });
   return rank;
 }
 

package/src/sql.mjs

@@ -872,20 +872,39 @@ function compileSelect(ast, env, outer, tail) {
   }
   // join rel (its columns `cols`) on outer key ASTs = rel key columns; the
   // rel's columns get fresh names; returns them
-  function attach(rel, outerKeys, relKeys, cols) {
+  // a subquery's columns looked up on the outer rows by key; in a grouped SELECT (ctx.step deferred),
+  // done after grouping, so the columns are there for the grouped rows
+  function attach(rel, outerKeys, relKeys, cols, ctx) {
     const tag = n.hidden++;
     const fresh = (c) => `__s${tag}_${c}`;
-    let rq = rel.query;
-    const all = [...new Set([...relKeys, ...cols])];
-    all.forEach((c) => { rq = rq.derive(fresh(c), c); });
-    rq = rq.select(...all.map(fresh));
-    let lefts = outerKeys.map((t) => hide(t));
-    let rights = relKeys.map(fresh);
-    if (!lefts.length) { const k = `__x${n.hidden++}`; q = q.derive(k, '0'); rq = rq.derive(k, '0'); lefts = [k]; rights = [k]; }
-    q = q.join(rq, { on: { left: lefts, right: rights }, how: 'left', view: true }); // (its keys are unique: right columns looked up, left rows not copied)
+    const run = () => {
+      let rq = rel.query;
+      const all = [...new Set([...relKeys, ...cols])];
+      all.forEach((c) => { rq = rq.derive(fresh(c), c); });
+      rq = rq.select(...all.map(fresh));
+      let lefts = outerKeys.map((t) => hide(t));
+      let rights = relKeys.map(fresh);
+      if (!lefts.length) { const k = `__x${n.hidden++}`; q = q.derive(k, '0'); rq = rq.derive(k, '0'); lefts = [k]; rights = [k]; }
+      q = q.join(rq, { on: { left: lefts, right: rights }, how: 'left', view: true }); // (its keys are unique: right columns looked up, left rows not copied)
+    };
+    if (ctx?.step) ctx.step({ name: fresh(cols[0]), inputs: [], run }); else run();
     return cols.map(fresh);
   }
-  function subqueryValue(e, ctx) {
+  // x NOT IN (subquery), as SQL has it: true where the subquery has no rows (x missing or not);
+  // else true only where x is present, not among the values, and none of them is missing
+  function notInValue(e, ctx, rel, vcol, mcol, relKeys, outerKeys) { // (rel: one row per value and key, mcol 1 on each)
+    const missing = rel.kinds?.get(vcol) === 'text' ? `${vcol} == ''` : `${vcol} != ${vcol}`;
+    let sq = rel.query;
+    let keys = relKeys;
+    if (!keys.length) { sq = sq.derive('__g', '0'); keys = ['__g']; }
+    sq = sq.groupBy(keys).agg({ __n: 'count()', __nn: `count(${missing})` });
+    const [cn, cnn] = attach({ ...rel, query: sq, names: [...keys, '__n', '__nn'] }, outerKeys, relKeys, ['__n', '__nn'], ctx);
+    const [m] = attach(rel, [translate(e.x, ctx), ...outerKeys], [vcol, ...relKeys], [mcol], ctx);
+    const isIn = bin('==', v(m), num(1));
+    const none = bin('==', call('coalesce', v(cn), num(0)), num(0));
+    return bin('||', none, bin('&&', bin('&&', present(e.x, ctx), { t: 'un', op: '!', x: isIn }), bin('==', call('coalesce', v(cnn), num(0)), num(0))));
+  }
+  function subqueryValue(e, ctx, negate = false) {
     if (e.k === 'exists') {
       const { inner, pairs } = correlate(e.q);
       const body = inner.body;
@@ -893,7 +912,7 @@ function compileSelect(ast, env, outer, tail) {
       const sel = { ...body, distinct: true, items: [...keyItems, { e: { k: 'num', v: 1 }, alias: '__m' }], groupBy: [] };
       if (body.groupBy?.length || body.having) fail('EXISTS (SELECT ... GROUP BY ...) is not supported');
       const rel = compileQuery({ ...inner, body: sel, orderBy: [], limit: pairs.length ? null : 1, offset: 0 }, env, outerHere);
-      const [m] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__m']);
+      const [m] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__m'], ctx);
       return bin('==', v(m), num(1));
     }
     if (e.k === 'insub') {
@@ -904,7 +923,8 @@ function compileSelect(ast, env, outer, tail) {
           const rel = compileQuery(inner, env, outerHere);
           if (rel.names.length !== 1) fail('IN (SELECT ...) needs exactly one column');
           const dist = { ...rel, query: rel.query.groupBy([rel.names[0]]).agg({ __m: 'count()' }).derive('__m1', '1') };
-          const [m] = attach({ ...dist, names: [rel.names[0], '__m1'] }, [translate(e.x, ctx)], [rel.names[0]], ['__m1']);
+          if (negate) return notInValue(e, ctx, { ...dist, names: [rel.names[0], '__m', '__m1'] }, rel.names[0], '__m1', [], []);
+          const [m] = attach({ ...dist, names: [rel.names[0], '__m1'] }, [translate(e.x, ctx)], [rel.names[0]], ['__m1'], ctx);
           return bin('==', v(m), num(1));
         }
         fail('IN (SELECT ...) needs exactly one column');
@@ -913,7 +933,8 @@ function compileSelect(ast, env, outer, tail) {
       const sel = { ...body, distinct: !body.groupBy?.length, items: [{ e: body.items[0].e, alias: '__v' }, ...keyItems, ...(body.groupBy?.length ? [] : [{ e: { k: 'num', v: 1 }, alias: '__m' }])] };
       let rel = compileQuery({ ...inner, body: sel, orderBy: [], limit: null, offset: 0 }, env, outerHere);
       if (body.groupBy?.length) { const keys = ['__v', ...keyItems.map((it) => it.alias)]; rel = { ...rel, query: rel.query.groupBy(keys).agg({ __c: 'count()' }).derive('__m', '1'), names: [...keys, '__m'] }; }
-      const [m] = attach(rel, [translate(e.x, ctx), ...pairs.map((pr) => translate(pr.outer, ctx))], ['__v', ...keyItems.map((it) => it.alias)], ['__m']);
+      if (negate) return notInValue(e, ctx, rel, '__v', '__m', keyItems.map((it) => it.alias), pairs.map((pr) => translate(pr.outer, ctx)));
+      const [m] = attach(rel, [translate(e.x, ctx), ...pairs.map((pr) => translate(pr.outer, ctx))], ['__v', ...keyItems.map((it) => it.alias)], ['__m'], ctx);
       return bin('==', v(m), num(1));
     }
     // (SELECT one value)
@@ -923,14 +944,14 @@ function compileSelect(ast, env, outer, tail) {
     if (!pairs.length) {
       const rel = compileQuery(inner, env, outerHere);
       if (rel.names.length !== 1) fail('a subquery used as a value needs exactly one column');
-      const [c] = attach(rel, [], [], [rel.names[0]]);
+      const [c] = attach(rel, [], [], [rel.names[0]], ctx);
       return v(c);
     }
     if (!(body.items.length === 1 && hasAgg(body.items[0].e)) || body.groupBy?.length) fail('a correlated subquery used as a value must compute one aggregate, e.g. (SELECT MAX(x) FROM t WHERE t.k = o.k)');
     const keyItems = pairs.map((pr, i) => ({ e: pr.inner, alias: `__k${i}` }));
     const sel = { ...body, items: [...keyItems, { e: body.items[0].e, alias: '__v' }], groupBy: keyItems.map((it) => it.e) };
     const rel = compileQuery({ ...inner, body: sel, orderBy: [], limit: null, offset: 0 }, env, outerHere);
-    const [c] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__v']);
+    const [c] = attach(rel, pairs.map((pr) => translate(pr.outer, ctx)), keyItems.map((it) => it.alias), ['__v'], ctx);
     const it = body.items[0].e;
     return it.k === 'call' && it.fn === 'COUNT' ? call('coalesce', v(c), num(0)) : v(c); // (no rows: COUNT is 0)
   }
@@ -1099,9 +1120,19 @@ function compileSelect(ast, env, outer, tail) {
   // true where a value is there (SQL's NULL is NaN for numbers, '' for text); null: always there (a literal)
   function present(e, ctx) {
     if (e.k === 'num' || e.k === 'str' || e.k === 'date' || e.k === 'interval') return num(1);
+    if (isCond(e)) return bin('||', translate(e, ctx), negated(e, ctx)); // (a condition: known where it's true or false)
     const x = translate(e, ctx);
     return kindOf(e, ctx) === 'text' ? bin('!=', x, { t: 'str', v: '' }) : bin('==', x, x);
   }
+  // a true / false / unknown condition (comparisons, AND, OR, NOT, IS, IN, LIKE...): translate()
+  // gives where it's true, negated() where it's false; elsewhere it's unknown
+  function isCond(e) {
+    if (e.k === 'num') return !!e.bool;
+    if (e.k === 'bin') return ['AND', 'OR', 'IS', '=', '<>', '!=', '<', '<=', '>', '>=', '~', '~*', '!~', '!~*'].includes(e.op);
+    return ['not', 'isnull', 'like', 'in', 'between', 'exists', 'insub'].includes(e.k);
+  }
+  // a condition as a value: 1 true, 0 false, NaN unknown (SQL's NULL boolean)
+  function condValue(e, ctx, t) { return cond(t, num(1), cond(negated(e, ctx), num(0), num(NaN))); }
   // NOT x with SQL's three values: where x is unknown (a missing value), so is NOT x, and the
   // row is dropped. Comparisons become their opposites (false on missing values already).
   const FLIP = { '=': '<>', '<>': '=', '!=': '=', '<': '>=', '<=': '>', '>': '<=', '>=': '<' };
@@ -1118,7 +1149,8 @@ function compileSelect(ast, env, outer, tail) {
       case 'in':
         if (kindOf(x.x, ctx) === 'text') return bin('&&', { t: 'un', op: '!', x: T(x) }, present(x.x, ctx));
         return x.list.map((i) => T({ k: 'bin', op: '<>', l: x.x, r: i })).reduce((a, b) => bin('&&', a, b));
-      case 'like': case 'insub': return bin('&&', { t: 'un', op: '!', x: T(x) }, present(x.x, ctx));
+      case 'like': return bin('&&', { t: 'un', op: '!', x: T(x) }, present(x.x, ctx));
+      case 'insub': return subqueryValue(x, ctx, true);
       case 'exists': case 'isnull': return { t: 'un', op: '!', x: T(x) };
       default: { // a true / false value (1 / 0): NOT of a missing one stays missing
         const v = T(x);
@@ -1146,9 +1178,16 @@ function compileSelect(ast, env, outer, tail) {
       case 'exists': case 'insub': case 'subq': return subqueryValue(e, ctx);
       case 'window': return ctx.window(e);
       case 'bin': {
-        if (e.op === 'AND') return bin('&&', T(e.l), T(e.r));
-        if (e.op === 'OR') return bin('||', T(e.l), T(e.r));
+        const B = (x) => (x.k === 'null' ? num(0) : T(x)); // (a NULL condition is unknown: never true)
+        if (e.op === 'AND') return bin('&&', B(e.l), B(e.r));
+        if (e.op === 'OR') return bin('||', B(e.l), B(e.r));
         const op = { '=': '==', '<>': '!=' }[e.op] ?? e.op;
+        if (op === 'IS' && e.r.k === 'num' && e.r.bool) { // x IS TRUE / IS FALSE: never unknown (unknown is neither)
+          const x = e.l;
+          if (isCond(x)) return e.r.v ? T(x) : negated(x, ctx);
+          if (x.k === 'null') return num(0);
+          return e.r.v ? bin('&&', bin('!=', T(x), num(0)), present(x, ctx)) : bin('==', T(x), num(0));
+        }
         if (op === 'IS') { // a IS b: equal, or both missing
           const a = T(e.l), b = T(e.r);
           return bin('||', bin('==', a, b), bin('&&', call('isMissing', a), call('isMissing', b)));
@@ -1159,6 +1198,8 @@ function compileSelect(ast, env, outer, tail) {
           return op.startsWith('!') ? bin('==', m, num(0)) : bin('==', m, num(1));
         }
         if (['==', '!=', '<', '<=', '>', '>='].includes(op)) {
+          if (isCond(e.l) || isCond(e.r)) // (conditions compared: unknown if either one is)
+            return bin('&&', bin(op, T(e.l), T(e.r)), bin('&&', present(e.l, ctx), present(e.r, ctx)));
           const lk = kindOf(e.l, ctx), rk = kindOf(e.r, ctx);
           if (lk === 'text' || rk === 'text') {
             if ((lk === 'date' && e.r.k === 'str') || (rk === 'date' && e.l.k === 'str')) // a date written as text
@@ -1265,7 +1306,11 @@ function compileSelect(ast, env, outer, tail) {
             const s = bin('*', a[0], num(f));
             return bin('/', cond(bin('>=', s, num(0)), call('floor', bin('+', s, num(0.5))), call('ceil', bin('-', s, num(0.5)))), num(f));
           }
-          case 'LEAST': case 'GREATEST': if (a.length < 2) fail(`${e.fn}() needs two values or more`); return a.reduce((x, y) => call(e.fn === 'LEAST' ? 'min' : 'max', x, y));
+          case 'LEAST': case 'GREATEST': { // missing values skipped, as in PostgreSQL and DuckDB (NULL only if all are)
+            if (a.length < 2) fail(`${e.fn}() needs two values or more`);
+            const f = e.fn === 'LEAST' ? 'min' : 'max';
+            return a.reduce((x, y) => call(f, call('coalesce', x, y), call('coalesce', y, x)));
+          }
           case 'COALESCE': case 'IFNULL': case 'NVL': if (a.length < 2) fail(`${e.fn}() needs two values or more`); return call('coalesce', ...a);
           case 'NULLIF': need(2); return cond(bin('==', a[0], a[1]), num(NaN), a[0]);
           case 'DATE_TRUNC': {
@@ -1303,7 +1348,7 @@ function compileSelect(ast, env, outer, tail) {
   if (ast.where) {
     if (hasAgg(ast.where)) fail('aggregates (SUM, COUNT...) can\'t be used in WHERE: use HAVING');
     if (hasWindow(ast.where)) fail('window functions can\'t be used in WHERE (filter on them in an outer query)');
-    const t = translate(ast.where, plain);
+    const t = ast.where.k === 'null' ? num(0) : translate(ast.where, plain); // (WHERE NULL: unknown, no row)
     q = q.filter(src(t));
   }
 
@@ -1440,7 +1485,9 @@ function compileSelect(ast, env, outer, tail) {
     // aggregates, each once
     const aggs = new Map(); // sqlKey -> name
     const spec = {}, extraKeys = [];
-    const aggOf = (e) => {
+    const aggDone = new Map(); // (an aggregate used twice: the same expression, SUM's NULL check included)
+    const aggOf = (e) => { const k = key(e); if (!aggDone.has(k)) aggDone.set(k, aggOnce(e)); return aggDone.get(k); };
+    const aggOnce = (e) => {
       const k = key(e);
       if (aggs.has(k)) return v(aggs.get(k));
       const name = `__a${aggs.size}_${n.hidden++}`;
@@ -1480,8 +1527,9 @@ function compileSelect(ast, env, outer, tail) {
       return translate(e, post);
     };
     planWindows(translatePost, (x) => kindOf(x, post));
-    const outs = withCols(items.map((it) => ({ ...it, ast: textConst(translatePost(it.e), post) })));
-    const havingAst = ast.having ? translatePost(ast.having) : null;
+    const tri = (e, t) => { if (!isCond(e) || e.k === 'num' || t.t === 'var') return t; try { return condValue(e, post, t); } catch { return t; } }; // (built from GROUP BY expressions: as it is)
+    const outs = withCols(items.map((it) => { const t = translatePost(it.e); return { ...it, ast: textConst(tri(it.e, t), post) }; }));
+    const havingAst = ast.having ? (ast.having.k === 'null' ? num(0) : translatePost(ast.having)) : null;
     const orderAsts = orderBy.map((o) => ({ ...o, ast: orderTarget(o.e, outs, translatePost) }));
     q = q.groupBy([...keys.map((k) => k.name), ...extraKeys]).agg(Object.keys(spec).length ? spec : { __a_rows: 'count()' });
     const late = new Set(windowVars.values()), after = []; // (those using window values: after the windows)
@@ -1490,16 +1538,16 @@ function compileSelect(ast, env, outer, tail) {
     applyWindows();
     for (const st of after) st.run();
     for (const o of outs) if (o.ast.t !== 'var') q = q.derive(o.col, src(o.ast));
-    for (const o of orderAsts) sortKey(o);
+    for (const o of orderAsts) sortKey(o, orderKind(o, outs, post));
     return finish(outs, (o) => (o.e.k === 'col' ? post.kindOfCol(o.e) ?? kinds.get(resolveSafe(o.e)) : kindOf(o.e, post)), post);
   }
   const pre = windowCtx(plain);
   planWindows((e) => (e.k === 'col' ? colOf(e) : translate(e, pre)), (x) => kindOf(x, pre));
   applyWindows();
-  const outs = withCols(items.map((it) => ({ ...it, ast: textConst(it.e.k === 'col' ? colOf(it.e) : translate(it.e, pre), pre) })));
+  const outs = withCols(items.map((it) => { const t = it.e.k === 'col' ? colOf(it.e) : translate(it.e, pre); return { ...it, ast: textConst(isCond(it.e) && it.e.k !== 'num' ? condValue(it.e, pre, t) : t, pre) }; }));
   const orderAsts = orderBy.map((o) => ({ ...o, ast: orderTarget(o.e, outs, (e) => (e.k === 'col' ? colOf(e) : translate(e, pre))) }));
   for (const o of outs) if (o.ast.t !== 'var') q = q.derive(o.col, src(o.ast));
-  for (const o of orderAsts) sortKey(o);
+  for (const o of orderAsts) sortKey(o, orderKind(o, outs, pre));
   if (ast.distinct) q = q.groupBy([...new Set(outs.map((o) => o.col))]).agg({ __distinct_rows: 'count()' });
   return finish(outs, (o) => (o.e.k === 'col' ? (sessionRef(o.e)?.kind ?? kinds.get(o.e.resolved ? o.e.name : resolve(o.e))) : kindOf(o.e, pre)), pre);
 
@@ -1516,11 +1564,17 @@ function compileSelect(ast, env, outer, tail) {
     const t = tr(e), same = outs.find((o) => toSourceAst(o.ast) === toSourceAst(t));
     return same ? v(same.col) : t;
   }
-  function sortKey(o) {
+  // the kind (text or not) of what an ORDER BY item sorts: a select item by number or name, or the expression
+  function orderKind(o, outs, ctx) {
+    const e = o.e.k === 'num' ? outs[o.e.v - 1]?.e : (o.e.k === 'col' && !o.e.table && outs.find((x) => x.name === o.e.name)?.e) || o.e;
+    try { return kindOf(e, ctx); } catch { return null; }
+  }
+  function sortKey(o, kind) {
     let k;
     if (o.ast.t === 'var') k = o.ast.name;
     else { k = `__o${n.hidden++}`; q = q.derive(k, src(o.ast)); }
-    if (o.nullsFirst) { const h = `__nf${n.hidden++}`; q = q.derive(h, `${k} == ${k} ? 1 : 0`); sortKeys.push(h); }
+    // missing sorts last on its own (NaN, and '' for text); NULLS FIRST sorts present values after missing ones first
+    if (o.nullsFirst) { const h = `__nf${n.hidden++}`; q = q.derive(h, kind === 'text' ? `${k} == '' ? 0 : 1` : `${k} == ${k} ? 1 : 0`); sortKeys.push(h); }
     sortKeys.push(o.desc ? `-${k}` : k);
   }
   function finish(outs, kindFn, ctx) {

package/src/table.mjs

@@ -7,7 +7,7 @@
 
 import { parseKeys, labelRanks, sortIndex } from './sort.mjs';
 import { ZONE_ROWS, computeZones, zoneRange } from './zones.mjs';
-import { parse, toSource, walk, analyze, isMask } from './fuse.mjs';
+import { parse, toSource, walk, analyze, isMask, fold } from './fuse.mjs';
 import { generateGroupBy, generateCompact, generateHashGroups, generateMarkPairs, generateStridedKeys } from './table-kernels.mjs';
 import { decodeString } from './csv-kernel.mjs';
 import { maxUnits } from './kernels.mjs';
@@ -31,6 +31,10 @@ const IDENTITY = { sum: 0, count: 0, mean: 0, min: Infinity, max: -Infinity };
 const MAX_GROUP_SLOTS = 2 ** 28; // threads * aggregations * groups (2GB of float64)
 const PARALLEL_SORT_ROWS = 50_000; // smaller tables sort on the calling thread
 const TIME_STEPS = [86_400_000, 3_600_000, 60_000, 1_000]; // a day, an hour, a minute, a second (ms): see #directGroups
+// Column facts worked out once and remembered with the column's array
+// (arrays are never written once in a table):
+const MISSING = new WeakMap(); // { n: rows, missing: how many missing values }
+const KEY_RANGE = new WeakMap(); // { n: rows, range: [min, max, how many aren't whole] } (see #directGroups)
 
 export class Table {
   // cols: Map name -> { data, dict, codes, strings }
@@ -103,17 +107,18 @@ export class Table {
     for (const name of Object.keys(rows[0])) {
       checkName(name);
       const data = engine.alloc(type, n);
-      if (typeof rows[0][name] === 'string') {
+      const first = firstPresent(rows.map((r) => r[name]));
+      if (typeof first === 'string') {
         const codes = new Map(), dict = [];
         for (let i = 0; i < n; i++) {
-          const s = rows[i][name];
+          const s = rows[i][name] ?? ''; // (null / undefined: missing text)
           let c = codes.get(s);
           if (c === undefined) { c = dict.length; dict.push(s); codes.set(s, c); }
           data[i] = c;
         }
         cols.set(name, { data, dict, codes });
       } else {
-        const date = rows[0][name] instanceof Date;
+        const date = first instanceof Date;
         if (date && type === Float32Array) throw new Error(f32Dates(name));
         for (let i = 0; i < n; i++) { const v = rows[i][name]; data[i] = v === null || v === undefined ? NaN : date ? toMillis(v) : v; } // (null: missing)
         cols.set(name, { data, dict: null, codes: null, ...(date ? { date: true } : {}) });
@@ -310,7 +315,14 @@ export class Table {
   // Lazy filter: returns a view of the same columns with a condition attached.
   filter(expr, params = {}) {
     const merged = this.#mergeParams(params);
-    const { src } = this.#bind(expr, merged);
+    let { ast, src } = this.#bind(expr, merged);
+    walk(ast, (m) => { if (m.t === 'str') throw new SyntaxError(`vectro: text like ${JSON.stringify(m.v)} can only be compared with == or != to a text column, in "${expr}" (to compare dates, load the column as dates: { dates: ['name'] })`); });
+    const k = fold(ast);
+    if (k.t === 'num') { // no column in it (1 == 1, 0): every row, or none (a kernel needs a column to read)
+      if (k.v !== 0) return new Table(this.engine, this.cols, this.n, { pred: this.pred, params: merged, zones: this.zones });
+      const c = [...this.cols.keys()].find((x) => !this.cols.get(x).strings);
+      src = `${c} != ${c} && ${c} == ${c}`;
+    }
     const pred = this.pred ? `(${this.pred}) && (${src})` : src;
     return new Table(this.engine, this.cols, this.n, { pred, params: merged, zones: this.zones });
   }
@@ -578,6 +590,19 @@ export class Table {
     return Math.floor((engine._directSlots ?? DIRECT_SLOTS) / (engine.T * totals));
   }
 
+  // Is src a number column of this table with no missing values? Counted
+  // once per column (one pass on every thread) and remembered with its array.
+  *#noMissingG(src) {
+    const c = src && IDENT.test(src) ? this.cols.get(src) : null;
+    if (!c || c.dict || c.strings) return false;
+    let known = MISSING.get(c.data);
+    if (known?.n !== this.n) {
+      const k = c.enc ? { data: c.data, enc: c.enc, n: this.n, type: this.type === Float32Array ? 'f32' : 'f64' } : c.data;
+      MISSING.set(c.data, (known = { n: this.n, missing: (yield* this.engine.evalG(['count(k != k)'], { k }))[0] }));
+    }
+    return known.missing === 0;
+  }
+
   // Keys small enough to be group numbers as they are: a text column's codes
   // or whole numbers 0..2^24, several combined as k1 * size2 + k2..., while
   // the accumulators stay small (or no larger than the rows). Else null.
@@ -590,7 +615,9 @@ export class Table {
       if (!this.n) { sizes.push(0); offsets.push(0); steps.push(1); continue; }
       // whole numbers in a range under 2^24 (from their minimum: 202301..202412, IDs from 1e9, negatives)
       const k = c.enc ? { data: c.data, enc: c.enc, n: this.n, type: this.type === Float32Array ? 'f32' : 'f64' } : c.data;
-      const [lo, hi, frac] = yield* engine.evalG(['min(k)', 'max(k)', 'count(k != floor(k))'], { k });
+      let known = KEY_RANGE.get(c.data);
+      if (known?.n !== this.n) KEY_RANGE.set(c.data, (known = { n: this.n, range: yield* engine.evalG(['min(k)', 'max(k)', 'count(k != floor(k))'], { k }) }));
+      const [lo, hi, frac] = known.range;
       if (frac > 0 || !Number.isFinite(lo) || !Number.isFinite(hi)) return null;
       // or times a whole number of days (hours, minutes, seconds) apart: numbered by that unit
       let step = 1;
@@ -604,9 +631,16 @@ export class Table {
       offsets.push(lo);
       steps.push(step);
     }
+    const fits = (K) => !(K > 2 ** 24 || engine.T * J * K * (engine.jit && K <= 1024 ? 4 : 1) > (engine._maxGroupSlots ?? MAX_GROUP_SLOTS) || (K > 65_536 && K > 8 * this.n));
+    // One key from a small positive number (IDs from 1, years from 1990):
+    // numbered from 0 with a few groups left empty, so the key column itself
+    // is the group number and needs no copy.
+    if (keys.length === 1 && steps[0] === 1 && offsets[0] > 0 && offsets[0] <= Math.max(1024, sizes[0]) && offsets[0] + sizes[0] <= maxK && fits(offsets[0] + sizes[0])) {
+      sizes[0] += offsets[0];
+      offsets[0] = 0;
+    }
     const K = sizes.reduce((a, b) => a * b, 1);
-    const C = engine.jit && K <= 1024 ? 4 : 1;
-    if (K > 2 ** 24 || engine.T * J * K * C > (engine._maxGroupSlots ?? MAX_GROUP_SLOTS) || (K > 65_536 && K > 8 * this.n)) return null;
+    if (!fits(K)) return null;
     if (K > maxK) { if (info) info.K = K; return null; }
     const below = sizes.map((_, k) => sizes.slice(k + 1).reduce((a, b) => a * b, 1));
     const keyColumn = (k, keep) => directKeys(keep, below[k], sizes[k], steps[k], offsets[k]); // key k of each kept group (codes for text)
@@ -747,12 +781,20 @@ export class Table {
   *#runGroupsG(keys, groups, allItems, merged) {
     const { engine } = this;
     const { K } = groups;
-    const items = allItems.filter((it) => !DISTINCT.has(it.fn));
+    // Over a column with no missing values, count(x) is the row count (always
+    // counted) and sum / mean / min / max need no check for missing values:
+    // SQL's SUM(x) is sum(x) and count(x), so this takes a whole accumulator off.
+    const full = new Set();
+    for (const it of allItems) if (!DISTINCT.has(it.fn) && (yield* this.#noMissingG(it.raw))) full.add(it);
+    const rowCount = (it) => it.fn === 'count' && (!it.arg || full.has(it));
+    const items = allItems.filter((it) => !DISTINCT.has(it.fn) && !rowCount(it));
     // means need the count of values present: one more kernel item each
-    const kernelItems = [...items, ...items.filter((it) => it.present).map((it) => ({ fn: 'count', arg: it.present, of: it }))]; // means, mins, maxes
+    const kernelItems = [...items.map((it) => (full.has(it) ? { fn: it.fn, arg: it.raw } : it)),
+      ...items.filter((it) => it.present && !full.has(it)).map((it) => ({ fn: 'count', arg: it.present, of: it }))]; // means, mins, maxes
     const totals = yield* this.#groupTotalsG(groups, kernelItems, merged);
     if (!totals) return null; // (lookup groups the kernels can't take)
-    const presentOf = (it) => totals[1 + kernelItems.findIndex((k) => k.of === it)];
+    const totalOf = (it) => (rowCount(it) ? totals[0] : totals[1 + items.indexOf(it)]);
+    const presentOf = (it) => (full.has(it) ? totals[0] : totals[1 + kernelItems.findIndex((k) => k.of === it)]);
     // Keep groups that have rows, sorted by key (codes for text keys, so
     // dictionary order; NaN last), and build whole columns.
     let keep = keptGroups(totals[0], K);
@@ -768,7 +810,7 @@ export class Table {
     }
     // each item's value for every group number
     const perGroup = allItems.map((it) => {
-      const i = items.indexOf(it), v = i >= 0 ? totals[i + 1] : it.perGroup;
+      const v = DISTINCT.has(it.fn) ? it.perGroup : totalOf(it);
       return it.fn === 'mean' ? ratioOf(v, presentOf(it)) : it.fn === 'min' || it.fn === 'max' ? presentOnly(v, presentOf(it)) : v;
     });
     if (engine.jit && m >= PARALLEL_SORT_ROWS) return yield* this.#groupTableG(keys, groups, allItems, perGroup, keep);
@@ -963,29 +1005,34 @@ export class Table {
         interpretGroup(engine, t, analysis.items, predAst, cols, merged, groups.data, acc, K, this.n);
       }
 
-      // Combine threads (and copies) into one value per group and aggregation.
+      // Combine threads (and copies) into one value per group and aggregation:
+      // the thread blocks by fused passes added into thread 0's, 15 blocks a
+      // pass (kernels, so fast from the first query; small ones run on this
+      // thread), then the C copies of each group by a plain loop.
       totals = [];
       for (const [j, it] of analysis.items.entries()) {
-        const op = it.reduce === 'min' ? Math.min : it.reduce === 'max' ? Math.max : (a, b) => a + b;
-        if (C > 1 || T === 1 || !engine.jit) { // a plain loop is enough
-          const a = engine.live(acc);
-          const out = new Float64Array(K).fill(IDENTITY[it.reduce]);
-          for (let s = 0; s < T; s++) {
-            const [lo] = slots(s, j);
-            for (let g = 0; g < K; g++) for (let c = 0; c < C; c++) out[g] = op(out[g], a[lo + g * C + c]);
+        let from = T;
+        if (T > 1 && engine.jit) {
+          for (let s0 = 1; s0 < T; s0 += 15) {
+            const vars = { t0: acc.subarray(...slots(0, j)) };
+            for (let s = s0; s < Math.min(T, s0 + 15); s++) vars[`t${s}`] = acc.subarray(...slots(s, j));
+            const vnames = Object.keys(vars);
+            const expr = it.reduce === 'min' || it.reduce === 'max'
+              ? vnames.slice(1).reduce((e, nm) => `${it.reduce}(${e}, ${nm})`, vnames[0])
+              : vnames.join(' + ');
+            yield* engine.evalG(expr, vars, vars.t0);
           }
-          totals.push(out);
-          continue;
+          from = 1;
+        }
+        const a = engine.live(acc);
+        if (C === 1 && from === 1) { const [lo, hi] = slots(0, j); totals.push(a.slice(lo, hi)); continue; }
+        const op = it.reduce === 'min' ? Math.min : it.reduce === 'max' ? Math.max : (x, y) => x + y;
+        const out = new Float64Array(K).fill(IDENTITY[it.reduce]);
+        for (let s = 0; s < from; s++) {
+          const [lo] = slots(s, j);
+          for (let g = 0; g < K; g++) for (let c = 0; c < C; c++) out[g] = op(out[g], a[lo + g * C + c]);
         }
-        // many groups: one fused pass over all thread blocks, written into thread 0's
-        const vars = {};
-        for (let s = 0; s < T; s++) vars[`t${s}`] = acc.subarray(...slots(s, j));
-        const vnames = Object.keys(vars);
-        const expr = it.reduce === 'min' || it.reduce === 'max'
-          ? vnames.slice(1).reduce((e, nm) => `${it.reduce}(${e}, ${nm})`, vnames[0])
-          : vnames.join(' + ');
-        yield* engine.evalG(expr, vars, vars.t0);
-        totals.push(Float64Array.from(engine.live(vars.t0)));
+        totals.push(out);
       }
       engine.free(acc);
     }
@@ -1821,12 +1868,14 @@ function checkName(name) {
 // (mean divides by the values present), min and max skip them, count(x)
 // counts the values present (count(condition) still counts where it holds).
 // arg is the bound source; returns the aggregation item.
+// (raw: the argument as given; where it is a column with no missing values,
+// the group-by uses it as it is and takes counts from the row count)
 function missingRules(fn, arg) {
   const present = `(${arg}) == (${arg})`;
-  if (fn === 'count') return { fn, arg: isMask(parse(arg)) ? arg : present };
-  if (fn === 'min') return { fn, arg: `(${present} ? (${arg}) : Infinity)`, present }; // (present: NaN when nothing is)
-  if (fn === 'max') return { fn, arg: `(${present} ? (${arg}) : -Infinity)`, present };
-  return fn === 'mean' ? { fn, arg: `(${present} ? (${arg}) : 0)`, present } : { fn, arg: `(${present} ? (${arg}) : 0)` }; // sum, mean
+  if (fn === 'count') return isMask(parse(arg)) ? { fn, arg } : { fn, arg: present, raw: arg };
+  if (fn === 'min') return { fn, arg: `(${present} ? (${arg}) : Infinity)`, present, raw: arg }; // (present: NaN when nothing is)
+  if (fn === 'max') return { fn, arg: `(${present} ? (${arg}) : -Infinity)`, present, raw: arg };
+  return fn === 'mean' ? { fn, arg: `(${present} ? (${arg}) : 0)`, present, raw: arg } : { fn, arg: `(${present} ? (${arg}) : 0)`, raw: arg }; // sum, mean
 }
 
 // a category column given as codes into a labels array: { codes, labels }
@@ -1850,7 +1899,9 @@ function codedText(engine, type, { codes: src, labels }) {
 }
 
 const isDates = (values) => Array.isArray(values) && values.find((v) => v != null) instanceof Date;
-const isText = (values) => !ArrayBuffer.isView(values) && values.length > 0 && typeof values[0] === 'string';
+// the first value present decides (a column may start with null / undefined: missing)
+const firstPresent = (values) => { for (const v of values) if (v !== null && v !== undefined) return v; return undefined; };
+const isText = (values) => !ArrayBuffer.isView(values) && values.length > 0 && typeof firstPresent(values) === 'string';
 
 function copyNumbers(engine, type, values) {
   const data = engine.alloc(type, values.length);
@@ -1863,7 +1914,7 @@ function encodeText(engine, type, values) {
   const data = engine.alloc(type, values.length);
   const codes = new Map(), dict = [];
   for (let i = 0; i < values.length; i++) {
-    const s = values[i];
+    const s = values[i] ?? ''; // (null / undefined: missing text, as in JSON)
     let c = codes.get(s);
     if (c === undefined) { c = dict.length; dict.push(s); codes.set(s, c); }
     data[i] = c;