quiverdb 0.12.6 → 0.12.8

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "quiverdb",
-  "version": "0.12.6",
+  "version": "0.12.8",
   "license": "MIT",
   "repository": {
     "type": "git",

package/src/lua-api.ts

@@ -52,7 +52,7 @@ Lua values map to Quiver column values as follows:
 | number (float)     | REAL         | A float is rejected for an INTEGER column.     |
 | string             | TEXT         | Also used for \`date_time\` columns (ISO 8601).  |
 | boolean            | INTEGER 1/0  | SQLite has no boolean type; \`true\` writes 1, \`false\` 0. |
-| \`nil\`              | NULL         | In query params, file paths, ts rows, relations.|
+| \`nil\`              | NULL         | In query params (not trailing), ts rows, group cells, relations. |
 | table (1-indexed)  | array        | Used for vectors/sets and column-oriented data.|
 
 A boolean is accepted **wherever an integer is** — element scalars and arrays, query parameters,
@@ -86,10 +86,18 @@ midnight.
 - **Type coercion.** An integer is accepted for a REAL column (coerced to real on insert); a float
   is rejected for an INTEGER column. A string bound to a \`date_*\` column must parse as ISO 8601
   (\`YYYY-MM-DD\`, optionally \`THH:MM:SS\` or \` HH:MM:SS\`). Other type mismatches raise a
-  validation error and roll the whole script back.
-- **Errors abort the script.** Any error thrown by a \`db:\` call stops the script and surfaces as
-  \`Failed to run Lua script: <message>\`. Validation failures roll back whatever the current
-  transaction covered.
+  validation error.
+- **An uncaught error aborts the script, and the script is not rolled back.** Any error thrown by
+  a \`db:\` call that you do not catch with \`pcall\` stops the script and surfaces as
+  \`Failed to run Lua script: <message>\`. Outside a transaction each \`db:\` write commits on its
+  own, so the failing call changes nothing in the database, but every write that finished before
+  it **stays**. Only \`db:transaction(fn)\` and \`db:dry_run(fn)\` undo their block's database
+  writes when \`fn\` errors; nothing undoes a file a \`db:\` call wrote. A
+  \`db:begin_transaction()\` (or \`db:begin_dry_run()\`) you opened and never closed stays open
+  after the error, and closing it is up to the host. A host that runs your script inside its own
+  transaction or dry run may undo the whole run; check \`db:in_transaction()\` if it matters. To
+  make several writes all-or-nothing, put them in one \`db:transaction(function(db) ... end)\`
+  and do not \`pcall\` inside it: a caught error lets the block commit whatever ran.
 - **Standard library.** Loaded standard libraries: base, string, table, math, coroutine, utf8.
   That is the pure-computation set — there is no \`os\`, \`io\`, \`debug\`, or \`package\`/\`require\`,
   and \`dofile\`/\`loadfile\` are removed (string-form \`load\` stays available). Integer division is
@@ -106,9 +114,12 @@ midnight.
   \`Cannot <op>: database is in-memory, file operations are unavailable\`.
 - **Output.** A script can \`return\` one value and the host receives it as JSON — prefer this over
   \`print()\` when you need structured data back (\`print()\` still works and is captured). Only the
-  **first** returned value is encoded. Arrays are 1-indexed (iterate with \`ipairs\`); reading a NULL
-  yields \`nil\`, writing \`nil\` stores NULL where NULL is accepted (query params, ts rows, file
-  columns, relation targets — but NOT element scalar attributes; see CRUD).
+  **first** returned value is encoded. Arrays are 1-indexed. Iterate with \`ipairs\` only where no
+  NULL can appear: a read of a nullable column (a bulk scalar read, a vector/set inner list, a
+  time-series value column, \`db:read_time_series_row\`) has \`nil\` holes, and \`ipairs\` stops at
+  the first one (see Scalar reads). Reading a NULL yields \`nil\`; writing \`nil\` stores NULL where
+  NULL is accepted (query params except a trailing one, ts rows, group cells, relation targets —
+  but NOT element scalar attributes; see CRUD).
 
   \`\`\`lua
   return { ids = db:read_element_ids("Collection"), total = 3 }
@@ -126,10 +137,10 @@ midnight.
   | any other table            | object, keys sorted; integer keys stringify              |
 
   **A table with holes is an object, not an array.** A bulk read of a nullable column returns
-  \`nil\` holes (see Reading), so \`return db:read_scalar_integers(c, a)\` encodes as
+  \`nil\` holes (see Scalar reads), so \`return db:read_scalar_integers(c, a)\` encodes as
   \`{"1":10,"3":30}\` — not \`[10,null,30]\` — and the keys sort as text (\`"1","11","2"\`). When the
   host needs positional scalar data, return the ids alongside and fill the holes yourself:
-  \`local v = db:read_scalar_integers(c, a); local out = {}; for i in ipairs(ids) do out[i] = v[i] or false end\`.
+  \`local ids = db:read_element_ids(c); local v = db:read_scalar_integers(c, a); local out = {}; for i = 1, #ids do out[i] = v[i] or false end\`.
   **Inside** a vector/set read an inner list with an interior NULL cell encodes as an object nested
   in the outer array (\`[{"1":10,"3":30},[]]\`), but a trailing NULL cell leaves no trace at all:
   \`[10, NULL]\` encodes as the plain \`[10]\`. The ids count elements, not rows, so they cannot
@@ -142,8 +153,10 @@ midnight.
   self-referencing table); a result over 64 MB raises
   \`Cannot run: script return value exceeds ... bytes of JSON\`; a string holding bytes that are not
   valid UTF-8 raises \`Cannot run: script return value contains a string that is not valid UTF-8\`;
-  and a table where an integer key and a string key spell the same thing (\`{[1] = 'a', ['1'] = 'b'}\`)
-  raises \`Cannot run: script returned a table with duplicate key '1'\`.
+  a table where an integer key and a string key spell the same thing (\`{[1] = 'a', ['1'] = 'b'}\`)
+  raises \`Cannot run: script returned a table with duplicate key '1'\`; and a table with a key that
+  is neither an integer nor a string (a boolean, a float such as \`1.5\`, a table) raises
+  \`Cannot run: script returned a table with an unsupported key type\`.
 - **Embedding harness may restrict further.** The library itself allows transactions and the
   (sandboxed) CSV/file operations below. A host that runs your script (e.g. a hosted \`run_lua\`
   tool) may disable some of them and report \`disabled in the run_lua sandbox\` — that limit comes
@@ -193,10 +206,12 @@ end)   -- both writes commit together; if either throws, both roll back
 \`\`\`
 
 **Caveat:** if the host already runs your script inside a plain transaction (not a dry run), an
-explicit \`db:begin_transaction()\` will error (\`cannot start a transaction within a transaction\`)
-and a mid-script \`db:commit()\` would prematurely end the host's transaction. When unsure whether
-a transaction is already open, check \`db:in_transaction()\` first, or just issue writes directly —
-each \`db:\` write is durable on its own.
+explicit \`db:begin_transaction()\` will error
+(\`Cannot begin_transaction: transaction already active\`) and a mid-script \`db:commit()\` would
+prematurely end the host's transaction. \`db:transaction(fn)\` begins one too, so it fails the same
+way. When unsure whether a transaction is already open, check \`db:in_transaction()\` first, or just issue writes directly:
+outside a transaction each \`db:\` write commits on its own, while inside the host's transaction it
+commits or rolls back with the host's.
 
 ---
 
@@ -235,6 +250,11 @@ Rules worth knowing:
   \`Cannot begin_dry_run: dry run already active\` — check \`db:in_dry_run()\` first and skip the
   wrapper when one is already active. Do **not** call \`db:end_dry_run()\` to get around it: that
   ends the host's dry run, and everything you write afterwards is committed for real.
+- **Nor inside a plain transaction.** If the host (or a \`db:transaction\` block) already holds a
+  plain transaction, \`db:begin_dry_run()\` and \`db:dry_run(fn)\` throw
+  \`Cannot begin_dry_run: transaction already active\`. \`db:in_transaction()\` is true under a host
+  dry run *and* under a host transaction, so it is the one check that covers both;
+  \`db:in_dry_run()\` covers only the first.
 - For a rough sense of how much a run touched, \`db:query_integer("SELECT total_changes()")\` gives
   the number of rows inserted, updated or deleted on this connection.
 
@@ -293,8 +313,10 @@ Notes:
   \`{ x = nil }\` is identical to \`{}\`; an update/create table that ends up with no attributes
   **throws** (\`...must have at least one scalar attribute\` on create, \`...at least one attribute
   to update\` on update). To leave a column unchanged, omit the key — you cannot set a scalar to
-  NULL via the element table. (\`nil\` → NULL is only accepted by
-  \`upsert_time_series_row\`, \`update_time_series_files\` and \`update_relation\`.)
+  NULL via the element table. (\`nil\` → NULL is accepted by \`update_relation\` and in query params
+  (not a trailing one), and as a cell in the group writers; \`upsert_time_series_row\` and
+  \`update_time_series_files\` replace the whole row, so a column you leave out (or set to \`nil\`,
+  the same thing) is cleared — see Time series and Time series files.)
 - **\`update_relation\` points one scalar foreign-key relation at another element**, named by the
   target's label. The column is derived from the naming convention —
   \`lowercase(collection_to) .. "_" .. relation_type\`, so
@@ -311,7 +333,10 @@ Notes:
 
 ## Scalar reads (bulk, across all elements)
 
-Each returns a flat array (1-indexed table), one value per element, in id order.
+Each returns a flat array (1-indexed table), one value per element, in id order. A NULL is a
+\`nil\` hole at that position, so \`ipairs\` stops at the first one and \`#\` is unreliable. Loop
+\`for i = 1, #ids\` over \`local ids = db:read_element_ids(collection)\` and index the result with
+\`i\`.
 
 \`\`\`lua
 db:read_scalar_integers(collection, attribute)   -- { 42, 37, ... }
@@ -432,9 +457,10 @@ local values = db:read_time_series_row(collection, group, attribute, date_time)
 \`\`\`
 
 One value per element using **last non-null value at or before \`date_time\`** semantics. Elements
-with no matching data yield \`nil\` in the array. \`date_time\` is an ISO 8601 string. A group with
-more than one dimension column (e.g. \`date_time\` + \`block\`) throws — read it with
-\`read_time_series_group\`.
+with no matching data yield \`nil\` in the array, so the result can hold \`nil\` holes: loop over
+\`db:read_element_ids(collection)\` rather than using \`ipairs\`. \`date_time\` is an ISO 8601
+string. A group with more than one dimension column (e.g. \`date_time\` + \`block\`) throws — read
+it with \`read_time_series_group\`.
 
 ### Replace a whole group (column-oriented — SAME shape as the read)
 
@@ -463,10 +489,10 @@ db:update_time_series_group("Items", "data", id, ts)   -- write the whole group
 
 **DO NOT pass an array of row tables** (\`{ { date_time = ..., value = ... }, ... }\`) — that is the
 \`upsert_time_series_row\` shape, not this one. Doing so raises
-\`stack index -1, expected string, received number\` (the integer array indices 1, 2, 3 are not
-column names). Each value of the top-level table must be an **array**, not a scalar.
+\`Cannot update_time_series_group: column names must be strings; pass { column = { values... } }, not an array of row tables\`.
+Each value of the top-level table must be an **array**, not a scalar.
 
-**Rules** (validation throws, rolling the script back):
+**Rules** (a violation throws; see "Errors abort the script" for what is kept):
 - Every column value must be an array — a bare scalar throws \`column '...' must be an array of values\`.
 - The **dimension column(s) set the row count** and must be present and fully populated: the
   \`date_*\` ordering column, plus any extra primary-key columns in a multi-dimensional group (e.g.
@@ -478,8 +504,9 @@ column names). Each value of the top-level table must be an **array**, not a sca
   column **longer** than the dimension column throws \`column '...' has length N but expected M\`.
 - Named columns whose dimension transposes to zero rows throw (\`contain no rows; pass an empty
   table {} to clear the group\`) — only a bare \`{}\` clears.
-- Integer values are accepted for REAL columns (converted on insert). Booleans, functions, and
-  other unsupported Lua types throw \`column '...' has unsupported Lua type\`.
+- Integer values are accepted for REAL columns (converted on insert), and a boolean is written as
+  1/0. A function, a table or another unsupported Lua type throws
+  \`column '...' has unsupported Lua type\`.
 - The element id must exist; the \`_by_label\` form takes a label in its place, with
   \`update_element_by_label\`'s resolution and miss semantics. Every rule above applies to both.
 
@@ -503,6 +530,10 @@ db:upsert_time_series_row_by_label("Items", "data", "Item 1", {
 The element id must exist; the \`_by_label\` form takes a label in its place, with
 \`update_element_by_label\`'s resolution and miss semantics.
 
+An existing row with the same dimension key is **replaced whole**: every value column you leave
+out (or set to \`nil\`) is reset to NULL (or its DEFAULT). To change one value, pass every value
+column of the row.
+
 ---
 
 ## Time series files
@@ -514,10 +545,15 @@ singleton table):
 db:has_time_series_files(collection)              -- boolean
 db:list_time_series_files_columns(collection)     -- { "data_file", "metadata_file", ... }
 db:read_time_series_files(collection)             -- { data_file = "path", metadata_file = nil, ... }
-db:update_time_series_files(collection, { data_file = "path/to/data.bin", metadata_file = nil })
+db:update_time_series_files(collection, { data_file = "path/to/data.bin" })  -- metadata_file is cleared
 \`\`\`
 
-In \`update_time_series_files\`, a \`nil\` value clears that column.
+\`update_time_series_files\` **replaces the whole row**: every column you do not give a string is
+set to NULL (or to its DEFAULT, if the schema declares one), and in Lua a \`nil\` value and a
+missing key are the same thing. A table with no string values (\`{}\`, or only \`nil\` values)
+changes nothing and clears nothing. A value that is not a string throws. To change one path, read
+the row with \`db:read_time_series_files(collection)\`, change that one entry, and pass the whole
+table back.
 
 ---
 
@@ -575,9 +611,15 @@ entirely.
 ## Query (parameterized SQL)
 
 Positional \`?\` placeholders; \`params\` is an optional 1-indexed array. Each returns the first
-column of the first row as the requested type, or \`nil\` if there is no result. The number of
+column of the first row, or \`nil\` when there is no row, the value is NULL, or it is not already
+the requested type. Only \`query_float\` converts (it widens an INTEGER), so
+\`db:query_string("SELECT COUNT(*) ...")\` and \`db:query_integer("SELECT AVG(x) ...")\` return
+\`nil\`; \`CAST\` in the SQL when unsure (\`SELECT CAST(AVG(x) AS INTEGER)\`). The number of
 \`params\` must match the number of \`?\` placeholders exactly — a mismatch (too few or too many)
-throws rather than binding NULL or ignoring extras.
+throws rather than binding NULL or ignoring extras. A \`nil\` param binds NULL only in a table
+constructor whose last entry is not \`nil\` (\`{ nil, 5 }\`): Lua stores no key for a \`nil\`, so
+\`{ 5, nil }\` or \`{ nil }\` comes up short and throws that mismatch. To test for NULL, write
+\`IS NULL\` in the SQL.
 
 \`\`\`lua
 db:query_string(sql, params)    -- string or nil
@@ -599,9 +641,22 @@ local count = db:query_integer("SELECT COUNT(*) FROM Collection")
 
 ## CSV import / export
 
-Export a time-series group to a CSV file, or import one from a CSV file. \`path\` is sandboxed:
-relative paths resolve against the database file's directory and must stay inside it (see
-Critical rules); \`options\` is optional.
+Export one table of \`collection\` to a CSV file, or import one from a CSV file. \`group\` names a
+vector, set or time-series group of \`collection\`; pass \`""\` for the collection's own scalar table
+(the CSV then holds exactly the table's columns except \`id\`, \`label\` included). \`path\` is
+sandboxed: relative paths resolve against the database file's directory and must stay inside it
+(see Critical rules); \`options\` is optional.
+
+**\`db:import_csv\` replaces, it does not merge.** A group import deletes every existing row of
+that group, for every element, not only the elements named in the file, and then inserts the
+file's rows. A scalar import (\`group = ""\`) matches elements by exact label. An element whose
+label is in the file keeps its id and has every column overwritten from the file (a blank cell
+writes NULL). **Every element whose label is not in the file is deleted** as
+\`db:delete_element\` would delete it: its group rows go with it, and every relation pointing at
+it from another table follows its \`ON DELETE\` action (\`SET NULL\` clears it, \`CASCADE\`
+deletes the row that holds it, a group row or an element of another collection). So editing a
+label in the file deletes that element and creates a new one. A header-only CSV empties the
+table. To add or change a few elements, use \`db:create_element\` / \`db:update_element\` instead.
 
 \`\`\`lua
 db:export_csv(collection, group, path, options)
@@ -612,7 +667,7 @@ The optional \`options\` table has two keys:
 
 \`\`\`lua
 {
-    date_time_format = "%Y-%m-%d",   -- strftime-style format for the dimension column
+    date_time_format = "%Y-%m-%d",   -- strftime-style format for every date-time column
 
     -- enum_labels: write/read integer codes as human labels. Three nested levels:
     --   attribute name -> locale -> { label = integer_id }
@@ -625,6 +680,9 @@ The optional \`options\` table has two keys:
 }
 \`\`\`
 
+An options value that is not a table, an unknown key or a wrong-typed value throws
+(\`Cannot export_csv: unknown option '...'\`).
+
 **Precondition:** \`db:import_csv\` cannot run inside an open transaction (it opens and commits its
 own) — it throws \`Cannot import_csv: transaction already active\`. Call it outside any
 \`db:transaction\` / \`db:begin_transaction\` block.
@@ -812,7 +870,9 @@ File I/O is db-scoped (\`db:open_file\` / \`db:bin_to_csv\` / \`db:csv_to_bin\`)
 base paths, sandboxed to the database directory (see Critical rules), and \`get_file_path()\` returns
 the resolved absolute path. The pure-metadata builders and expression constructors live under the
 global \`quiver\` table. Mirrors the Julia surface; aggregation ops are strings (Lua has no enums);
-operators are \`+ - * /\` and unary \`-\`, with scalars allowed on either side.
+operators are \`+ - * /\` and unary \`-\`, with scalars allowed on either side. A file handle does not
+outlive the \`run()\` that opened it: any handle still open when the script returns is closed (and a
+writer flushed), so reopen the file in each script.
 
 \`\`\`lua
 local md = quiver.metadata{
@@ -847,7 +907,8 @@ e:save(out_path); e:metadata()                       -- save path is sandboxed l
 
 **\`quiver.metadata{...}\` kwargs and defaults:** \`version\` defaults to \`"1"\`; \`initial_datetime\`
 and \`unit\` default to \`""\`; \`labels\`, \`dimensions\`, \`dimension_sizes\`, \`time_dimensions\`, and
-\`frequencies\` default to empty arrays.
+\`frequencies\` default to empty arrays. Only these eight keys are accepted; an unknown key or a
+value of the wrong type throws.
 
 **\`get_dimensions()\` / \`get_metadata()\` dimension shape** — returns an array of dimension tables: