arn-db 0.1.0 → 0.2.1

package/GUIDE.md

@@ -30,6 +30,7 @@ Contents:
 [Writing related rows](#writing-related-rows) ·
 [Transactions](#transactions) ·
 [Read replicas](#read-replicas) ·
+[Raw SQL fragments](#raw-sql-fragments) ·
 [Errors](#errors) ·
 [Not supported yet](#not-supported-yet)
 
@@ -54,10 +55,12 @@ await db.from('users').update({ scores: db.ops.append(4) }).eq('id', id)
 await db.from('posts').insert({ title, user_id, comments: db.rel.create([{ body, author_id }]) })
 ```
 
-`db` reads the variables on its first query, not on import, so loading dotenv
-later in startup still works. If they aren't set yet, it loads `./.env` from the
-working directory first (Node 20.12+). Variables that are already set always win.
-A missing variable throws on that first query.
+Importing arn-db loads `./.env` from the working directory into `process.env`
+(Node 20.12+), like `import "dotenv/config"`: the app's other variables are set
+too, so files need no dotenv import. Variables that are already set always win,
+and a missing `.env` is fine. `db` reads its two variables on the first query (it
+tries `./.env` once more then), so setting them later in startup still works. A
+missing variable throws on that first query.
 
 Inside `transaction()` the same helpers are on `tx`: `tx.from()`, `tx.ref()`,
 `tx.ops.*`, `tx.rel.*`. (`import { ops, rel } from 'arn-db'` still works too.)
@@ -483,7 +486,8 @@ await db.from('posts').select('user_id').distinct()       // SELECT DISTINCT
 ```
 
 Aggregates, `groupBy()` and `distinct()` don't combine with related tables in the
-same select. Not yet: `having`.
+same select. For `HAVING`, use `.having(db.raw('count(*) >= ?', [2]))` (see
+[Raw SQL fragments](#raw-sql-fragments)).
 
 ---
 
@@ -700,6 +704,98 @@ await db.from('users').select().eq('email', email).primary()
 
 ---
 
+## Raw SQL fragments
+
+When the builder has no method for it, `db.raw(sql, bindings?)` adds one SQL
+expression to the query (same as `import { raw }`, and `tx.raw` in
+transactions). It is knex's `whereRaw` / `orderByRaw` / `havingRaw` /
+`knex.raw`: a piece of a query, never a whole statement. `test/raw.test.ts`
+checks these against the test database.
+
+```ts
+const n = db.raw('substring(name from 6)::int')   // user_42 -> 42
+
+// where(): AND (<fragment>)
+await db.from('users').select('name').where(db.raw('substring(name from 6)::int <= ?', [3])).order('name')
+// [{ name: 'user_1' }, { name: 'user_2' }, { name: 'user_3' }]
+
+// inside or() / and() / not() it is one condition of the group
+await db.from('users').select('name')
+  .or((q) => q.where(db.raw('substring(name from 6)::int = ?', [1])).eq('name', 'user_2'))
+// user_1, user_2
+
+// array binding: = any(?)
+await db.from('users').select('name').where(db.raw('lower(trim(email)) = any(?)', [['user_1@example.com', 'user_7@example.com']]))
+
+// jsonb ? operator: \? (written '\\?' in JS) is a literal ?
+await db.from('users').where(db.raw('meta \\? ?', ['vip'])).count()          // 100
+
+// order()
+await db.from('users').select('name').order(n, { ascending: false }).limit(3)
+// user_100, user_99, user_98
+
+// select(): extra arguments are raw columns, each needs .as(name); options go last
+await db.from('users').select('name', db.raw('char_length(name)').as('name_len')).eq('name', 'user_42').single()
+// { name: 'user_42', name_len: 7 }
+
+// having(), with groupBy()
+await db.from('posts').select('user_id, n:count()').groupBy('user_id').having(db.raw('count(*) >= ?', [2]))
+// 33 groups, each n: 2
+```
+
+As values in `insert()` / `update()` / `upsert()` rows, and in upsert `merge`,
+where `excluded.col` is the incoming value and `table.col` the stored one:
+
+```ts
+await db.from('users').update({ age: db.raw('age + ?', [5]) }).eq('email', email).returning('age')   // 10 -> [{ age: 15 }]
+
+await db.from('users').upsert({ email, name: 'second', age: 5 }, {
+  onConflict: 'email',
+  merge: { age: db.raw('CASE WHEN excluded.age IS NOT NULL AND users.age IS NOT NULL THEN users.age + excluded.age ELSE ? END', [100]) },
+})
+// stored age 10 -> 15; name stays (only merge columns change); with age: null -> 100
+```
+
+Rules:
+
+- Each `?` is one binding, passed as a query parameter, so values are never
+  pasted into the SQL. The count must match `bindings`, else a 400 error
+  (`raw fragment: 2 placeholders but 1 binding`). Strings, numbers, booleans and
+  `null` pass through; an array becomes a Postgres array (cast it if needed:
+  `?::text[]`); an object becomes jsonb (`?::jsonb`). A `?` inside `'quoted
+  text'` is not a binding.
+- `\?` (in a JS string: `'\\?'`) is a literal `?`, for the jsonb `?` operator.
+- Fragments see the main table only: its columns, plain or qualified with the
+  table's name (`users.age`, `"users".age`), and `excluded.col` in `merge`.
+  Related tables are not visible, and `order(raw)` with `referencedTable` is an error.
+- The API key must be allowed to use raw fragments. Otherwise every request
+  (or transaction) that contains one gets 403, code `RAW_NOT_ALLOWED`.
+- The server rejects with 400, code `RAW_FRAGMENT`: `;`, `--` and `/*`
+  comments, dollar quoting (`$$`), the keywords `select`, `insert`, `update`,
+  `delete`, `with`, `union`, `into`, `copy`, `grant`, `drop`, `alter`,
+  `create`, `truncate`, `execute`, `do`, `call` outside quoted text, an empty
+  fragment or one over 2,000 characters, and more than 100 bindings.
+  Functions, `CASE WHEN`, casts, `interval '1 hour'`, operators, `count(*)`
+  and `excluded.x` are fine.
+- In the client, `raw()` throws when `sql` is not a non-empty string or
+  `bindings` not an array. `select()` with a raw column lacking `.as()`, and
+  `where()` / `having()` with something other than `raw()`, resolve to `{ error }`
+  without sending.
+
+Moving from knex:
+
+| knex | arn-db |
+|---|---|
+| `.whereRaw(sql, b)`, `.andWhereRaw(sql, b)` | `.where(db.raw(sql, b))` |
+| `.orWhereRaw(sql, b)` | `.or((q) => q.eq(...).where(db.raw(sql, b)))` |
+| `.orderByRaw(sql, b)` | `.order(db.raw(sql, b), { ascending, nullsFirst })` |
+| `.select('id', knex.raw('char_length(title) as n'))` | `.select('id', db.raw('char_length(title)').as('n'))` |
+| `{ total: knex.raw('total + ?', [1]) }` | `{ total: db.raw('total + ?', [1]) }` |
+| `.onConflict('email').merge({ total: knex.raw('…') })` | `.upsert(rows, { onConflict: 'email', merge: { total: db.raw('…') } })` |
+| `.havingRaw(sql, b)` | `.having(db.raw(sql, b))` |
+
+---
+
 ## Errors
 
 `error` has `message`, and usually `code` and `details`:
@@ -712,11 +808,13 @@ await db.from('users').select().eq('email', email).primary()
 | `22P02` | value has the wrong format | `.eq('id', 'abc')` on a uuid |
 | `42703` | column doesn't exist | `select('nope')` → `column _t0.nope does not exist` (`_t0` is the main table) |
 | `42P10` | `onConflict` columns have no unique constraint | `upsert(…, { onConflict: 'title' })` |
+| `RAW_FRAGMENT` | (400) a `db.raw()` fragment the server refuses | `db.raw('1; drop table x')` |
+| `RAW_NOT_ALLOWED` | (403) the API key may not use raw fragments | |
 | `401` | missing or wrong API key | |
 | `403` | request came from a browser | |
 
-Only setup mistakes throw: `createClient()` without a key, and `db`'s first
-query when `ARN_DB_URL` or `ARN_DB_KEY` is missing.
+Only setup mistakes throw: `createClient()` without a key, `db.raw()` without a
+SQL string, and `db`'s first query when `ARN_DB_URL` or `ARN_DB_KEY` is missing.
 
 Errors without a code are checks done before the database runs anything, for
 example: `more than one relationship between "posts" and "users", use one of: users!reviewer_id, users!user_id`.
@@ -727,6 +825,7 @@ example: `more than one relationship between "posts" and "users", use one of: us
 
 - Sorting main rows by a related table's column (e.g. posts by author name). Sorting *inside* a relation works.
 - OR across tables in one group (e.g. `name = 'x' OR posts.published`): each `or()` group is on one table.
-- Full-text search, calling Postgres functions (`rpc`), `having` for groups, column
-  functions like `lower(trim(email))`.
+- Full-text search and calling Postgres functions (`rpc`) as methods. Column
+  functions like `lower(trim(email))` and `having` work through
+  [raw fragments](#raw-sql-fragments).
 - `db.col()` across tables, and as a value in `insert()` / `update()`.

package/README.md

@@ -34,15 +34,16 @@ import { db } from 'arn-db'          // or: const { db } = require('arn-db')
 const { data, error } = await db.from('users').select().eq('id', 1).single()
 ```
 
-That's all: `db` reads the two variables on its first query. If they aren't set,
-it loads `./.env` first (Node 20.12+), as arn-knexjs did with dotenv. ESM, CommonJS and
-TypeScript types are included.
+That's all. Importing arn-db loads `./.env` into `process.env` (Node 20.12+;
+variables that are already set win), like arn-knexjs's `dotenv/config`, so your
+files need no dotenv import. `db` reads the two variables on its first query.
+ESM, CommonJS and TypeScript types are included.
 
 ## Quick tour
 
 Import the ready `db` everywhere: `db.from()`, `db.transaction()`, `db.ops.*`,
-`db.rel.*`. Inside transactions the same helpers are on `tx` (`tx.ops.*`,
-`tx.rel.*`, `tx.ref()`). For another key or options, make your own client with
+`db.rel.*`, `db.raw()`. Inside transactions the same helpers are on `tx` (`tx.ops.*`,
+`tx.rel.*`, `tx.raw()`, `tx.ref()`). For another key or options, make your own client with
 `createClient(url, key, options)`.
 
 ```ts
@@ -90,8 +91,8 @@ await db.from('users').update({ meta: db.ops.append('x').at('address', 'lines')
 
 Errors never throw: every call (and every transaction) resolves to
 `{ data, error, count, status }`, like Supabase. Only setup mistakes throw:
-`createClient()` without an API key or fetch, and `db`'s first query when
-`ARN_DB_URL` / `ARN_DB_KEY` is missing.
+`createClient()` without an API key or fetch, `db.raw()` without a SQL string,
+and `db`'s first query when `ARN_DB_URL` / `ARN_DB_KEY` is missing.
 
 ## Searching arrays (jsonb arrays and Postgres arrays)
 
@@ -214,3 +215,53 @@ await db.transaction((tx) => [
   returned an array, pick one: `tx.ref('posts.0.id')`.
 - Step names: words without dots. Number-like keys (`'1'`) are rejected because
   JavaScript would reorder them.
+
+## Raw SQL fragments
+
+For the bits the builder has no method for, `db.raw(sql, bindings?)` (also
+`import { raw }` and `tx.raw`) adds one SQL expression to the query the builder
+makes, like knex's `whereRaw` / `orderByRaw` / `knex.raw`. It is a fragment,
+never a whole statement.
+
+```ts
+// filters: AND (<fragment>), or one member of an or() group
+await db.from('users').select('id').where(db.raw('lower(trim(email)) = any(?)', [emails]))
+await db.from('accounts').select().or((q) => q.where(db.raw('sent_per_hour < limit_per_hour')).is('limit_per_hour', null))
+
+// order, computed select columns (.as() is required), having
+await db.from('accounts').select().order(db.raw("coalesce(last_read_time, '1970-01-01')"), { ascending: true })
+await db.from('posts').select('id, title', db.raw('char_length(title)').as('title_len'))
+await db.from('posts').select('user_id, n:count()').groupBy('user_id').having(db.raw('count(*) >= ?', [2]))
+
+// values in insert / update / upsert rows, and in upsert merge (excluded.col = incoming value)
+await db.from('accounts').update({ sent_total: db.raw('sent_total + ?', [1]) }).eq('id', id)
+await db.from('accounts').upsert(rows, {
+  onConflict: 'email',
+  merge: { total: db.raw('CASE WHEN excluded.total IS NULL THEN accounts.total ELSE excluded.total END') },
+})
+```
+
+- Each `?` is a binding, sent as a query parameter (never pasted into the SQL).
+  Their number must match `bindings`. An array binding is a Postgres array
+  (`= any(?)`), an object is jsonb (`?::jsonb`). `?` inside `'quoted text'` is
+  not a binding.
+- `\?` (in a JS string: `'\\?'`) is a literal `?`, for the jsonb `?` operator:
+  `db.raw('meta \\? ?', ['vip'])`.
+- A fragment sees the main table's columns, plain or qualified with the table
+  name (`accounts.total`); not related tables.
+- The API key must be allowed to use raw fragments; otherwise the request gets
+  403 `RAW_NOT_ALLOWED`.
+- The server rejects (400 `RAW_FRAGMENT`): `;`, `--` and `/* */` comments,
+  dollar quoting, statement keywords (`select`, `insert`, `update`, `delete`,
+  `with`, `union`, `into`, `drop`, `alter`, `create`, `truncate`, …) outside
+  quoted text, fragments over 2,000 characters and more than 100 bindings.
+
+| knex | arn-db |
+|---|---|
+| `whereRaw(sql, b)` / `andWhereRaw(sql, b)` | `.where(db.raw(sql, b))` |
+| `orWhereRaw(sql, b)` | `.or((q) => q.where(db.raw(sql, b)).eq(...))` |
+| `orderByRaw(sql, b)` | `.order(db.raw(sql, b), { ascending })` |
+| `select('id', knex.raw('… as x'))` | `.select('id', db.raw('…').as('x'))` |
+| `{ col: knex.raw('col + ?', [1]) }` value | `{ col: db.raw('col + ?', [1]) }` |
+| `merge({ col: knex.raw(...) })` | `upsert(rows, { merge: { col: db.raw(...) } })` |
+| `havingRaw(sql, b)` | `.having(db.raw(sql, b))` |

package/dist/index.cjs

@@ -152,10 +152,66 @@ var Now = class Now {
 };
 const col = (name) => new Col(name);
 const now = () => new Now();
+/**
+* A raw SQL fragment: one expression inside the query the builder makes, with
+* `?` for each binding (`\\?` is a literal `?`, e.g. the jsonb `?` operator).
+* Bindings are sent as query parameters, never pasted into the SQL. Use it in
+* `where()`, `order()`, `having()`, as a `select()` column with `.as()`, and as
+* an insert/update/upsert value or upsert `merge` value. The API key must be
+* allowed to use raw fragments.
+*
+*   .where(db.raw('lower(email) = any(?)', [emails]))
+*   .update({ sent_total: db.raw('sent_total + ?', [1]) })
+*/
+var Raw = class Raw {
+	sql;
+	bindings;
+	#as;
+	constructor(sql, bindings = [], as) {
+		if (typeof sql !== "string" || sql.trim() === "") throw new Error("arn-db: raw(sql) needs a non-empty SQL string");
+		if (!Array.isArray(bindings)) throw new Error("arn-db: raw(sql, bindings) takes the bindings as an array");
+		this.sql = sql;
+		this.bindings = bindings;
+		this.#as = as;
+	}
+	/** Name of this fragment as a `select()` column: `raw('char_length(title)').as('title_len')`. */
+	as(name) {
+		return new Raw(this.sql, this.bindings, name);
+	}
+	/** @internal The `.as()` name, if any. */
+	alias() {
+		return this.#as;
+	}
+	/** @internal The fragment without its name, for filters, order and having. */
+	fragment() {
+		return {
+			$raw: this.sql,
+			bindings: this.bindings
+		};
+	}
+	/** @internal Wire format. */
+	toJSON() {
+		return this.#as === void 0 ? this.fragment() : {
+			...this.fragment(),
+			as: this.#as
+		};
+	}
+};
+/**
+* A raw SQL fragment with `?` bindings: knex's `whereRaw` / `orderByRaw` /
+* `havingRaw` / `knex.raw` value. Throws if `sql` is not a non-empty string
+* or `bindings` not an array; the server checks the rest.
+*
+* @example
+* db.from('users').select().where(raw('lower(trim(email)) = any(?)', [emails]))
+*/
+const raw = (sql, bindings) => new Raw(sql, bindings);
 // src/builder.ts
 const embedOf = (o) => o.referencedTable ?? o.foreignTable;
 /** Filter methods, shared by builders and groups. */
 var Filters = class {
+	/** @internal A problem found while building, reported instead of sending. */
+	problem;
 	#filter(column, op, value) {
 		if (column instanceof Col) {
 			const ifNull = column.nullValue();
@@ -288,6 +344,24 @@ var Filters = class {
 		return this;
 	}
 	/**
+	* A raw SQL condition, knex's `whereRaw`: `AND (<fragment>)`, or one member
+	* of an `or()` group. Only takes `raw()`; for a column use `eq()` and friends.
+	* The fragment sees the main table's columns, not related tables.
+	*
+	* @example
+	* .where(db.raw('lower(trim(email)) = any(?)', [emails]))
+	* .where(db.raw('sent_per_hour < limit_per_hour'))
+	* .or((q) => q.where(db.raw('meta \\? ?', ['vip'])).eq('age', 30))   // jsonb ? operator
+	*/
+	where(fragment) {
+		if (fragment instanceof Raw) this.push({
+			op: "raw",
+			value: fragment.fragment()
+		});
+		else this.problem ??= "where() takes a raw() fragment; for a column use eq(), gt(), in() and friends";
+		return this;
+	}
+	/**
 	* Any of the conditions.
 	*
 	* @example
@@ -314,6 +388,7 @@ var Filters = class {
 	#group(group, conditions) {
 		const q = new FilterGroup();
 		conditions(q);
+		if (q.problem) this.problem ??= q.problem;
 		this.push({
 			group,
 			filters: q.filters
@@ -340,8 +415,6 @@ var FilterGroup = class extends Filters {
 var Builder = class extends Filters {
 	/** @internal */ transport;
 	/** @internal */ req;
-	/** @internal A problem found while building, reported instead of sending. */
-	problem;
 	#name;
 	/** @internal */
 	constructor(transport, req, name, problem) {
@@ -445,11 +518,30 @@ var QueryBuilder = class extends Builder {
 	* db.from('users').select('name, posts!user_id!inner(title)')  // only users that have posts
 	* db.from('users').select('name, posts!user_id(count)')         // posts: [{ count: 2 }]
 	* db.from('posts').select('*', { count: 'exact' }).range(0, 9)  // rows + total
+	*
+	* Extra `raw(...).as(name)` arguments add SQL expressions of the main table
+	* as columns (options, if any, go last).
+	*
+	* @example
+	* db.from('posts').select('id, title', db.raw('char_length(title)').as('title_len'))
 	*/
-	select(columns = "*", options = {}) {
+	select(columns = "*", ...extra) {
 		this.req.columns = columns;
-		if (options.count) this.req.count = true;
-		if (options.head) this.req.head = true;
+		const rawColumns = [];
+		for (const e of extra) if (e instanceof Raw) {
+			const as = e.alias();
+			if (as === void 0) this.problem ??= `raw("${e.sql}") in select() needs a column name: add .as('name')`;
+			else if (!/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(as)) this.problem ??= `raw(...).as("${as}"): the name must be letters, digits and _`;
+			else rawColumns.push({
+				...e.fragment(),
+				as
+			});
+		} else if (e) {
+			if (e.count) this.req.count = true;
+			if (e.head) this.req.head = true;
+		}
+		if (rawColumns.length) this.req.rawColumns = rawColumns;
+		else delete this.req.rawColumns;
 		return this;
 	}
 	/**
@@ -511,6 +603,18 @@ var QueryBuilder = class extends Builder {
 		return this;
 	}
 	/**
+	* Keep only the groups where the raw condition holds, knex's `havingRaw`.
+	* Use with `groupBy()`; several calls are joined with AND.
+	*
+	* @example
+	* db.from('posts').select('user_id, n:count()').groupBy('user_id').having(db.raw('count(*) >= ?', [2]))
+	*/
+	having(fragment) {
+		if (fragment instanceof Raw) (this.req.having ??= []).push(fragment.fragment());
+		else this.problem ??= "having() takes a raw() fragment: having(db.raw(\"count(*) >= ?\", [2]))";
+		return this;
+	}
+	/**
 	* Leave out repeated rows: `SELECT DISTINCT`.
 	*
 	* @example
@@ -637,9 +741,19 @@ var QueryBuilder = class extends Builder {
 	* .order('active', { ascending: false }).order('age', { ascending: false }).order('name')
 	* .order('meta->>score::int', { ascending: false, nullsFirst: false })
 	* .order('title', { referencedTable: 'posts' })   // posts inside each user
+	* .order(db.raw("coalesce(last_read_time, '1970-01-01')"))   // raw, main table only
 	*/
 	order(column, options = {}) {
 		const { ascending = true, nullsFirst } = options;
+		if (column instanceof Raw) {
+			if (embedOf(options)) this.problem ??= "order(raw(...)) sorts the main table only: drop referencedTable";
+			else this.req.order.push({
+				raw: column.fragment(),
+				ascending,
+				nullsFirst
+			});
+			return this;
+		}
 		this.req.order.push({
 			column,
 			ascending,
@@ -921,6 +1035,7 @@ function createClient(url, apiKey, options = {}) {
 		rel,
 		col,
 		now,
+		raw,
 		async transaction(build) {
 			const steps = toSteps(build({
 				from: (table) => new QueryBuilder(null, table),
@@ -928,7 +1043,8 @@ function createClient(url, apiKey, options = {}) {
 				ops,
 				rel,
 				col,
-				now
+				now,
+				raw
 			}));
 			if (typeof steps === "string") return fail(steps);
 			const res = await transport.post("/transaction", { steps: steps.wire });
@@ -944,14 +1060,20 @@ function createClient(url, apiKey, options = {}) {
 }
 // src/env.ts
 let client;
+const proc = globalThis.process;
+/** Reads ./.env into process.env (Node 20.12+; never overrides set vars). */
+function loadDotEnv() {
+	if (!proc?.loadEnvFile) return;
+	try {
+		proc.loadEnvFile();
+	} catch {}
+}
+loadDotEnv();
 /** The ARN_DB_URL / ARN_DB_KEY client, made on the first query (so a later dotenv still counts). */
 function envClient() {
 	if (client) return client;
-	const proc = globalThis.process;
 	if (!proc) throw new Error("arn-db: `db` reads ARN_DB_URL / ARN_DB_KEY from process.env; use createClient(url, key) here");
-	if ((!proc.env.ARN_DB_URL || !proc.env.ARN_DB_KEY) && proc.loadEnvFile) try {
-		proc.loadEnvFile();
-	} catch {}
+	if (!proc.env.ARN_DB_URL || !proc.env.ARN_DB_KEY) loadDotEnv();
 	const url = proc.env.ARN_DB_URL;
 	const key = proc.env.ARN_DB_KEY;
 	if (!url) throw new Error("arn-db: set ARN_DB_URL (e.g. https://your-api.example.com/v2/raw) or use createClient(url, key)");
@@ -975,19 +1097,22 @@ const db = {
 	ops,
 	rel,
 	col,
-	now
+	now,
+	raw
 };
 exports.Col = Col;
 exports.FilterGroup = FilterGroup;
 exports.Now = Now;
 exports.Op = Op;
 exports.QueryBuilder = QueryBuilder;
+exports.Raw = Raw;
 exports.WriteBuilder = WriteBuilder;
 exports.col = col;
 exports.createClient = createClient;
 exports.db = db;
 exports.now = now;
 exports.ops = ops;
+exports.raw = raw;
 exports.rel = rel;
 
 //# sourceMappingURL=index.cjs.map