vectrojs 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,71 @@
+# Changelog
+
+## 0.1.0 (first release)
+
+The first public release. Everything below is new.
+
+**Engine**
+- Columnar tables in engine memory; expressions compiled into fused SIMD
+  WebAssembly loops and run on every CPU core (Node worker threads, Web
+  Workers on cross-origin-isolated pages), one thread elsewhere, and a
+  vectorized interpreter where code generation isn't allowed (Cloudflare Workers).
+- Filters, computed columns, aggregates, group by any keys (millions of
+  groups: partitioned by key range on every thread, sorted by key), exact and
+  approximate distinct counts, joins (inner, left, right, full), sort and
+  top-k, zone maps, compressed columns. Top-k over a file reads only the
+  blocks that can still hold a winning row.
+- A query planner: `table.lazy()` chains are optimized as a whole.
+- CSV (parallel parser), JSON lines, Apache Arrow (read and write).
+- Datasets bigger than memory (`.vectro` files): chunked, bit-packed, read in
+  parallel; group-by, joins and sorts spill to disk; in browsers too.
+- Streams (CSV, JSON lines, WebSocket feeds) with running windows.
+- Vector search (cosine / dot / L2, exact top-k).
+
+**SQL**
+- `sql(text, tables)`: joins, subqueries (IN, EXISTS, correlated), CTEs,
+  UNION / INTERSECT / EXCEPT, window functions, percentiles, text functions,
+  dates and intervals. 40 queries checked against DuckDB give the same answers.
+- Shortcuts for common shapes: only the rows that can rank in the first k for
+  `ROW_NUMBER() / RANK() ... <= k`; distinct counts of IDs with a byte map;
+  medians from bucket counts; CASE with quoted labels as label codes.
+
+**Serving** (beta)
+- `vectro serve`: SQL over HTTP (JSON, CSV, Arrow) and an OData v4 feed;
+  the PostgreSQL wire protocol (Power BI, Tableau, Metabase, Grafana, psql);
+  Arrow Flight SQL (pandas, Polars, DuckDB through ADBC). Checked with
+  pgJDBC, Npgsql, Microsoft's OData libraries and the ADBC driver.
+- An MCP server for AI assistants (`vectro-mcp`).
+
+**Connectors** (experimental)
+- `connectors/dist/Vectro.mez` for Power BI and `vectro.taco` for Tableau.
+  Not signed, not yet tried in the applications.
+
+**Missing values**
+- SQL uses three-valued logic for missing values: `NOT (x > 5)`, `x <> 5`,
+  `NOT IN`, `NOT BETWEEN`, `NOT LIKE` and `LIKE` drop rows where the value
+  is missing, as other databases do. Table expressions keep JavaScript's rules
+  (`x != 5` is true where x is missing).
+- Blank text (`''`, the missing value of a text column) is not counted by
+  `count(x)`, `countDistinct(x)`, `approxDistinct(x)` or SQL's `COUNT`, on
+  tables, datasets and split tables.
+- `null` in arrays and rows loads as missing (it used to load as 0); a `Date`
+  in `tableFromRows` makes a date column.
+- Clear errors for a text param, for date columns in a `Float32Array` table
+  (times don't fit in float32) and for SQL on a split table.
+- `readCSV(path, { compress: true })` packs the columns, as `parseCSV` does.
+
+**Testing**
+- `VECTRO_POISON=1` overwrites memory as it is freed, so code that reads an
+  array after freeing it gives wrong answers in tests (part of `test:all`).
+
+**Known gaps**
+- Slower than DuckDB on one shape measured: a join over a file on disk with
+  a GROUP BY on the joined table's columns (1.2-1.3x at 100M and 1B rows).
+  Faster on all 20 SQL queries measured, on every group-by, join and
+  distinct count measured in memory, and on 7 of 8 queries over a billion
+  rows on disk (see the README's numbers).
+- Free-text columns (text that rarely repeats) can be shown but not grouped
+  or filtered, except when served (`vectro serve` loads them as categories).
+- Window functions and percentiles keep the rows they read in memory.
+- An engine holds at most 4 GB in memory; tables are freed with `dispose()`.
+- Integers are exact up to 2^53; times are UTC.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hamza Masood
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.