w-orm-reladb 1.1.12 → 1.1.13

package/README.md

@@ -77,7 +77,7 @@ Differences from MongoDB you should know: `NULL` follows SQL three valued logic,
 
 Column names in a `find` must match the model exactly: the attribute name, or the database column name (`field`) of exactly one attribute. Any other spelling is rejected with `unknown column name`: a different letter case, full width forms, a column the model does not list (sqlite's hidden `rowid` included) or a virtual attribute. Databases resolve identifiers with their own collation rules (the default collation of mssql treats `id`, `ID`, `ＩＤ`, `id ` and `ɪd` as the same column) and the ORM rewrites names before they reach the database, so a spelling that is not an exact match could slip past the type checks above (`{ VALUE: { $lt: 'abc' } }` used to delete every non `NULL` row on sqlite); the error message suggests the closest name. Names that the ORM rewrites or re-interprets are rejected with `unsupported column name`: names outside the grammar above, names that are members of `Object.prototype` (such as `constructor`, `valueOf`), an attribute whose `field` is set but is not a string, or whose database column name is empty, contains quote characters, brackets or dots, or starts and ends with `$`; an attribute whose column is also the name of another attribute, and an attribute that shares its column with an attribute defined later (only the last one defined of a shared column is accepted), are rejected as `ambiguous column name`, because the ORM would serialize the value with the type of the other attribute, or the type of the column is decided by another one. The `attrs` you hand to `cvFind` must be the `rawAttributes` of the model that runs the query (a plain object without symbol keys and with enumerable keys only), otherwise `cvFind` throws `invalid ctx`. `delAll` is also refused for a model whose `defaultScope` has `limit`, `offset`, `group`, `having`, a non-empty `include`, or a `where` that uses `Op.and` (`select` and `delete` would not touch the same rows). The model must therefore list every column you query (a model generated by `genModelsByTabs` does). A name that matches nothing is rejected before the database is reached, an empty `$in` included (`delAll({ misspelled: { $in: [] } })` rejects instead of matching nothing silently). A `DATE` value (a `Date`, an epoch number, truncated to whole milliseconds, or an ISO string with a time zone) must fall in the years 1 to 9999 (UTC) and is always sent as a `Date`.
 
-A primary key value whose type does not fit the primary key column (for example `'abc'` or `12.4` for an integer key; a number for a string key is converted to its string form instead) is an invalid key: `selectByPk` returns `null`, `del` reports that row with `ok: 0`, and `insert`, `insertBulk` and `save` reject without writing anything.
+A primary key value whose type does not fit the primary key column (for example `'abc'` or `12.4` for an integer key; a number for a string key is converted to its string form instead) is an invalid key: `selectByPk` returns `null`, `del` reports that row with `ok: 0`, and `insert`, `insertBulk` and `save` reject without writing anything. A string key with a lone surrogate or a NUL is refused as well, with a different outcome for `save` (that row only gets `ok: 0`), see "Well formed strings" below.
 
 Model files are only read, never rewritten; an `id` column that lacks `primaryKey: true` is treated as the primary key in memory.
 
@@ -149,6 +149,66 @@ Checked by running the previous release (1.1.10) on sqlite.
 
 `inDateRange` and `genDateChecker` are new exports of `src/cvFind.mjs`; the condition side (`select` / `delAll`) and the write side use the same range test.
 
+### Well formed strings: primary keys, written data and names
+
+A string with a **lone surrogate** (half of a UTF-16 surrogate pair; `s.isWellFormed()` is `false`) cannot be stored as it is: the database turns the lone half into U+FFFD (sqlite stores text as UTF-8), so different keys become one key. On sqlite `'a\uD800'`, `'a\uDC00'` and `'a\uFFFD'` are the same primary key: `insert` skipped the "duplicate", `save` overwrote the other row, `selectByPk` and `del` hit a row that was never meant (checked on the previous release, 1.1.12). The conditions of `select` and `delAll` have refused such strings for a long time; the write side and the primary key functions now apply the same standard, series wide (shared with the other `w-orm-*` packages through `chkOrmStr` and `chkOrmData` of `wsemi`). Everything below happens **before the model files are loaded, before a connection is made and before anything is read, written or deleted**, on the copy of your data that is used for writing (made when you call the function), and it does not depend on `opt.autoGenPk`: a bad key you supplied is never replaced by a generated one. U+FFFD itself and valid pairs (emoji, rare CJK characters) are well formed and accepted as before.
+
+| What | Refused |
+|---|---|
+| the primary key value (also a `new String(…)` wrapper) | a lone surrogate, and also `NUL` (`\u0000`), the same standard as a `find` |
+| every string value and key name inside the data of `insert`, `insertBulk` and `save` (nested objects, array items, `Map` / `Set` content, `Error` / `RegExp` text …), **whether or not the model has that column** | a lone surrogate (a `NUL` inside a value is not refused by this check) |
+| `opt.url`, `opt.db`, `opt.cl`, `opt.storage` | a lone surrogate: **the constructor throws** `invalid opt.<name>: string contains lone surrogate` (two different names would otherwise point to the same storage, for example two sqlite files) |
+
+| Function | What happens with a bad primary key or bad data |
+|---|---|
+| `insert`, `insertBulk` | the whole call rejects and **nothing is written**, the error is emitted once as the `error` event before the promise rejects, no `change` event |
+| `save` | **only that row** reports `{ n: 1, nInserted: 0, nModified: 0, ok: 0, err }` (also with `autoInsert: false`), the other rows are processed as usual; nothing is read or written for the bad row, and when every row is bad no connection is made at all |
+| `del` | only that row reports `{ n: 0, nDeleted: 0, ok: 0, err }`, the other rows are deleted as usual; only the primary key is checked, the other fields of the row you pass are not; when every row is bad no connection is made at all |
+| `selectByPk` | `null` (a closed instance still rejects first) |
+| `select`, `delAll` | unchanged: the condition check rejects with `invalid find: …` |
+
+The messages are fixed words and **never quote the value**; the position uses the index of the row in your array (`data[0]` when you pass a single object):
+
+```alias
+invalid id: string contains lone surrogate at [data[1].id]
+invalid id: string contains NUL at [data[1].id]
+invalid data: string contains lone surrogate at [data[1].meta.tags[0]]
+invalid data: string contains lone surrogate at [data[1].<invalid key>]
+invalid opt.cl: string contains lone surrogate
+```
+
+Per row the primary key comes before the data and a key name before its value; for `insert` / `insertBulk` the row with the smallest index is reported. The other error messages that used to quote the primary key do not any more (`invalid id[…]` and `type does not match pk column` now read `invalid data[<index>].<pk>…`; the `save` message after three failed tries names the position instead of the key).
+
+Known limits: a `new String('…')` inside the data that holds a valid pair is neither unwrapped nor refused (the check follows the characters you wrote; nobody should store wrapper objects); output of a custom `toJSON` / `toString` / `valueOf` is not inspected; a custom `set` function of the model that rewrites a value afterwards is not checked either.
+
+**Rows written before this version are not migrated or read differently.** On sqlite a lone surrogate written earlier was already replaced by U+FFFD by the database: the original text is gone, and the stored text is well formed, so nothing changes for those rows. mssql (`nvarchar`) kept the lone surrogate. To find such rows and remove them:
+
+```alias
+let bad = (await w.select()).filter((v) => typeof v.id === 'string' && !v.id.isWellFormed())
+// selectByPk(bad[0].id) is null and del reports it with ok: 0 from now on (invalid id); address the row through another column,
+// for example await w.delAll({ name: bad[0].name }) (a condition with a lone surrogate is refused as well), or clear the table with await w.delAll()
+```
+
+**Equal text on mssql.** On mssql the string comparison follows the database collation, and the default collation (for example `SQL_Latin1_General_CP1_CI_AS`) treats **well formed** strings such as `'a'`, `'a'` followed by an emoji and `'a'` followed by U+FFFD as equal (checked on SQL Server 2022: all three compare equal under the default collation and different under `Latin1_General_100_BIN2`). That is a property of the collation, like the case insensitivity already described above, not a malformed string, so this check cannot and does not change it, and `createStorage` still creates the tables with the database default collation. If keys or values must be compared exactly, create the table (or the column) with a binary collation such as `Latin1_General_100_BIN2`; be aware that this also makes the comparison case sensitive. sqlite compares binary and is not affected.
+
+### Breaking changes: well formed strings
+
+Checked by running the previous release (1.1.12) on sqlite (and on mssql where stated); there is no transition period.
+
+| Input | before (1.1.12) | now |
+|---|---|---|
+| `insert` of a row whose key has a lone surrogate (`'a\uD800'`) | sqlite: stored as `'a\uFFFD'`, or skipped as a duplicate when `'a\uFFFD'` already exists (`nInserted` short by one, `ok: 1`); mssql: stored with the lone surrogate, or skipped as a duplicate when a key that compares equal under the collation (such as `'a\uFFFD'`) exists | the whole call rejects, nothing written: `invalid id: string contains lone surrogate at [data[i].id]` |
+| `insertBulk` of such a row | sqlite: stored as `'a\uFFFD'`, or rejects as a duplicate key (`Validation error`) when it exists; mssql: stored with the lone surrogate, or `Validation error` when an equal key exists | rejects, nothing written (same message) |
+| `save` of such a row | sqlite: overwrote the row `'a\uFFFD'`; mssql: updated the row that compares equal (and replaced its stored key by the lone surrogate), or inserted a new row | that row `ok: 0` with the same message, the other rows are saved |
+| `del` of such a key | sqlite: deleted the row `'a\uFFFD'` (`ok: 1`); mssql: deleted the row that compares equal | that row `ok: 0` with the same message |
+| `selectByPk` of such a key | sqlite and mssql: returned the row `'a\uFFFD'` | `null` |
+| a key containing NUL | sqlite: `insert` stored it, while `insertBulk` rejected, `save` and `del` reported `ok: 0` and `selectByPk` rejected, all with a database error (`SQLITE_ERROR: unrecognized token`); mssql: `insert`, `insertBulk` and `save` stored the key with the NUL, `del` found nothing (`n: 0`) and `selectByPk` returned `null` | the same refusal as a lone surrogate (`invalid id: string contains NUL …`); a NUL inside a data value is not refused |
+| a lone surrogate in a string value or key name of the data, at any depth (also columns the model does not list, which sequelize drops silently) | stored as `U+FFFD` text (sqlite) or as it is (mssql), or dropped | `insert` / `insertBulk` reject as a whole, `save` marks that row `ok: 0` (`invalid data: … at [data[i].path]`) |
+| `opt.url`, `opt.db`, `opt.cl` or `opt.storage` with a lone surrogate | accepted (the storage path was converted, so two names could share one file) | the constructor throws `invalid opt.<name>: string contains lone surrogate` |
+| error messages of `insert`, `insertBulk`, `save`, `del` for a primary key of the wrong type | quoted the key value (`invalid data[0].id[12.4], type does not match pk column [id]`, `invalid id[NaN]`) | no value: `invalid data[0].id, type does not match pk column [id]` (`insert`, `insertBulk`, `save`), `invalid data[0].id` and `invalid data[0].id, type does not match pk column` (`del`) |
+
+`chkPkWf` and `chkWf` are new exports of `src/cvFind.mjs`; `normPk` keeps its signature and result shape (`{ kind, v }`) and now answers `{ kind: 'invalid' }` for a string with a lone surrogate or NUL.
+
 ### Concurrency
 
 The scope of the atomicity guarantee, by range: