@osqd/jql 0.1.1

package/docs/course/06-typed-queries.md

@@ -0,0 +1,240 @@
+# Lesson 6 — Typed queries
+
+**Goal:** make a typo in a field name a compile error instead of an empty result.
+
+← [Course](index.md) · Previous: [Dates and windows](05-dates.md) · Next: [Requests](07-requests.md)
+
+---
+
+## The most expensive mistake
+
+```json
+{ "statsu": "paid" }
+```
+
+That is a perfectly valid query. It asks about a field nothing has, so it matches nothing, so
+the page comes back empty, so somebody reports "there were no paid orders this week" — and
+they are wrong.
+
+Nothing at run time can catch this. `statsu` might genuinely be a field on some other
+collection; the engine has no way to know it is not one of yours. The compiler does.
+
+`Query<T>` knows the shape of `T` and checks three things: the **paths**, the **values**, and
+which **operators** make sense for each field.
+
+This is the one lesson that needs TypeScript.
+
+## Setting up
+
+```bash
+npm install --save-dev typescript
+```
+
+`harbour/tsconfig.json`:
+
+```json
+{
+  "compilerOptions": {
+    "target": "es2022",
+    "module": "node16",
+    "moduleResolution": "node16",
+    "strict": true,
+    "noEmit": true
+  },
+  "files": ["lesson-06.ts"]
+}
+```
+
+And the shape of an order, which the rest of the lesson refers to. Put this at the top of
+`harbour/lesson-06.ts`:
+
+```ts
+import { filter, type Query } from "@osqd/jql";
+
+interface Order {
+  id: string;
+  placed: string;
+  status: "open" | "paid" | "refunded" | "cancelled";
+  channel: "web" | "app" | "phone";
+  customer: { name: string; country: string; email: string };
+  lines: { sku: string; title: string; quantity: number; price: number }[];
+  total: number;
+  paid: number;
+  tags: string[];
+  note?: string | null;
+}
+
+declare const orders: Order[];
+```
+
+## What a typed query looks like
+
+Everything you learned in lessons 1 to 5 still applies — paths, operators, `$elemMatch`,
+`$date`. The type adds nothing to write:
+
+```ts
+const good: Query<Order> = {
+  status: "paid",
+  "customer.country": "GB",
+  "lines.sku": "pen-fine",
+  total: { $gte: 30 },
+  tags: "repeat",
+  lines: { $elemMatch: { quantity: { $gt: 5 } } },
+  placed: { $gte: { $date: "2026-09-01" } },
+};
+
+filter(orders, good);
+```
+
+That compiles cleanly. Run `npx tsc -p tsconfig.json` and you will get nothing back, which is
+what "nothing wrong" looks like.
+
+## Six mistakes, six errors
+
+Now add these one at a time and compile after each.
+
+### A misspelt field
+
+```ts
+const q: Query<Order> = { statsu: "paid" };
+```
+
+```
+error TS2561: Object literal may only specify known properties, but 'statsu' does not exist in type 'TypedQuery<Order, {}>'. Did you mean to write 'status'?
+```
+
+The suggestion is TypeScript's own spell-checker, working from the key list JQL built out of
+your interface.
+
+### The wrong kind of value
+
+```ts
+const q: Query<Order> = { total: "30" };
+```
+
+```
+error TS2322: Type 'string' is not assignable to type 'FieldQuery<number> | undefined'.
+```
+
+`total` is a number, so a string is refused — which matters because lesson 2's comparisons
+never coerce, so `"30"` really would have matched nothing.
+
+### A value outside a union
+
+```ts
+const q: Query<Order> = { status: "posted" };
+```
+
+```
+error TS2322: Type '"posted"' is not assignable to type 'FieldQuery<"open" | "paid" | "refunded" | "cancelled"> | undefined'.
+```
+
+A union type is checked member by member, so a status that does not exist is caught by name.
+
+### A misspelt path
+
+```ts
+const q: Query<Order> = { "customer.contry": "GB" };
+```
+
+```
+error TS2353: Object literal may only specify known properties, and '"customer.contry"' does not exist in type 'TypedQuery<Order, {}>'.
+```
+
+Paths are enumerated five levels deep, through arrays as well as objects. Deeper ones still
+work at run time; they are simply not offered as suggestions.
+
+### An operator the field cannot serve
+
+```ts
+const q: Query<Order> = { status: { $size: 3 } };
+```
+
+```
+error TS2322: Type 'number' is not assignable to type 'undefined'.
+```
+
+This is the one message worth memorising, because it reads so oddly. `$size` counts array
+elements and `status` is a string, so `$size`'s type on this field is `undefined` — and
+anything at all you pass to it is an error. **`… is not assignable to type 'undefined'` always
+means "this operator does not apply to this field".**
+
+### A typo inside `$elemMatch`
+
+```ts
+const q: Query<Order> = { lines: { $elemMatch: { quantiy: 5 } } };
+```
+
+```
+error TS2322: Type '{ $elemMatch: { quantiy: number; }; }' is not assignable to type 'FieldQuery<{ sku: string; title: string; quantity: number; price: number; }[]> | undefined'.
+  Types of property '$elemMatch' are incompatible.
+    Object literal may only specify known properties, but 'quantiy' does not exist in type 'TypedQuery<{ sku: string; title: string; quantity: number; price: number; }, {}>'. Did you mean to write 'quantity'?
+```
+
+Inside `$elemMatch` is a full query over the **element** type, checked just as thoroughly as
+the outer one. Read that message from the bottom up; the last line is the actual problem.
+
+Every one of those six would otherwise have compiled, run, and matched nothing.
+
+## When the shape is not known
+
+A query parsed out of a request body, read from a database or typed into a search box has no
+`T` to check against — its shape only exists at run time. Say so rather than fighting the
+compiler:
+
+```ts
+import { untyped, validate } from "@osqd/jql";
+
+// A query that arrived as text: check it, then run the thing you checked.
+const checked = validate<Order>(JSON.parse(body));
+if (!checked.valid) return respond(400, { error: checked.error.message });
+const matching = orders.filter(checked.test);
+
+// A query built at run time, used where a typed one is expected.
+filter(orders, untyped(somethingBuiltAtRunTime));
+```
+
+- **`untyped(value)`** marks a query as one to check at run time instead of at compile time,
+  and is accepted anywhere a typed query is. It is what makes `orders.jqlFilter(parseText(input))`
+  compile in lesson 11.
+- **`validate`** is lesson 12's subject. Note that it hands back the predicate it compiled, so
+  the query you checked is guaranteed to be the query you run.
+- **`Query<unknown>`** accepts any field, for the same reason.
+
+In all three cases the engine still refuses anything that is not valid JQL. What you give up
+is only the compiler's help with names.
+
+## Exercise
+
+Add `outstanding: number` to `Order` and write a typed query for orders on the app with more
+than £50 outstanding. Then misspell `outstanding` and see which of the six errors you get.
+
+<details>
+<summary>Answer</summary>
+
+```ts
+interface Order { /* … as before … */ outstanding: number }
+
+const chasing: Query<Order> = { channel: "app", outstanding: { $gt: 50 } };
+```
+
+Misspell it —
+
+```ts
+const chasing: Query<Order> = { channel: "app", outstandng: { $gt: 50 } };
+```
+
+— and it is the first of the six errors again:
+
+```
+error TS2561: Object literal may only specify known properties, but 'outstandng' does not exist in type 'TypedQuery<Order, {}>'. Did you mean to write 'outstanding'?
+```
+
+In lesson 10 this same field becomes a **computed** one that is not on the interface at all —
+and the types still know about it.
+</details>
+
+## Related
+
+- [TypeScript guide](../guides/typescript.md) — typed queries, the array methods, vocabularies
+- [Library API](../reference/api.md) — `Query`, `UntypedQuery`, `validate`

package/docs/course/07-requests.md

@@ -0,0 +1,261 @@
+# Lesson 7 — Requests
+
+**Goal:** ask a whole question — which rows, in what order, which page, which fields — as one
+document.
+
+← [Course](index.md) · Previous: [Typed queries](06-typed-queries.md) · Next: [Counting by a field](08-grouping.md)
+
+---
+
+## A table on a screen is four questions
+
+It is a filter, an order, a page, and a decision about which columns are allowed to leave the
+server. A **request** is all four in one JSON document:
+
+```json
+{
+  "where":  { "status": "paid" },
+  "sort":   { "total": -1 },
+  "skip":   0,
+  "limit":  20,
+  "fields": ["id", "total"],
+  "omit":   []
+}
+```
+
+Every key is optional, and `where` takes a query — everything from lessons 1 to 5. Because the
+whole thing is still one JSON document, the whole question can travel in a URL, be saved, or
+be validated in one go.
+
+`search(collection, request)` runs it.
+
+## Filter, sort, and take a page
+
+```js
+import { search } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const page = search(orders, {
+  where: { status: { $ne: "cancelled" } },
+  sort: { total: -1 },
+  limit: 3,
+  fields: ["id", "total"],
+});
+
+for (const row of page) console.log(row);
+```
+
+```
+{ id: 'o-1005', total: 170 }
+{ id: 'o-1004', total: 36 }
+{ id: 'o-1006', total: 33 }
+```
+
+## The next page
+
+Add `skip`. Nothing else changes:
+
+```js
+import { search } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const request = {
+  where: { status: { $ne: "cancelled" } },
+  sort: { total: -1 },
+  limit: 3,
+  fields: ["id", "total"],
+};
+
+console.log("page 1:", search(orders, { ...request, skip: 0 }).map((row) => row.id));
+console.log("page 2:", search(orders, { ...request, skip: 3 }).map((row) => row.id));
+console.log("page 3:", search(orders, { ...request, skip: 6 }).map((row) => row.id));
+```
+
+```
+page 1: [ 'o-1005', 'o-1004', 'o-1006' ]
+page 2: [ 'o-1002', 'o-1001', 'o-1008' ]
+page 3: [ 'o-1003' ]
+```
+
+## Sorting by more than one key
+
+The keys are applied in order: the first sorts, the next breaks its ties. `1` or `"asc"` is
+ascending, `-1` or `"desc"` descending.
+
+```js
+import { search } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const line = (row) => `${row.status.padEnd(10)}${String(row.total).padStart(6)}  ${row.id}`;
+
+console.log("by status alone:");
+for (const row of search(orders, { sort: { status: "asc" } })) console.log("  ", line(row));
+
+console.log("by status, then total, descending:");
+for (const row of search(orders, { sort: { status: "asc", total: -1 } })) console.log("  ", line(row));
+```
+
+```
+by status alone:
+   cancelled    280  o-1007
+   open          30  o-1002
+   open         170  o-1005
+   paid        22.5  o-1001
+   paid         4.5  o-1003
+   paid          33  o-1006
+   paid          12  o-1008
+   refunded      36  o-1004
+by status, then total, descending:
+   cancelled    280  o-1007
+   open         170  o-1005
+   open          30  o-1002
+   paid          33  o-1006
+   paid        22.5  o-1001
+   paid          12  o-1008
+   paid         4.5  o-1003
+   refunded      36  o-1004
+```
+
+### Why this paging can be trusted
+
+**The sort is stable.** In the first list the four paid orders tie on the only key there is,
+and they come back in the order they arrived: `o-1001`, `o-1003`, `o-1006`, `o-1008`. An
+unstable sort is free to return them in any order at all, and a *different* order each time —
+so a reader paging through would see one of them twice and never see another. Adding `total`
+as a second key, as the second list does, settles them deliberately instead.
+
+**Missing values have a defined place.** The full order is written down in the specification —
+missing and `null` first, then numbers, strings, objects, booleans, dates. It is not the order
+you would guess. It is the order you can *rely* on, which is the property that matters when
+two systems sort the same result.
+
+## `fields` — keeping some of the document
+
+```js
+import { search } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const [row] = search(orders, {
+  where: { id: "o-1001" },
+  fields: ["id", "customer.name", "lines.sku"],
+});
+
+console.log(JSON.stringify(row, null, 2));
+```
+
+```
+{
+  "id": "o-1001",
+  "customer": {
+    "name": "Ada Lovelace"
+  },
+  "lines": [
+    {
+      "sku": "pen-fine"
+    },
+    {
+      "sku": "ink-blue"
+    }
+  ]
+}
+```
+
+`fields` keeps the paths you name **in the document's own shape**. `customer.name` comes back
+as `{ customer: { name } }`, and an array on the way stays an array of projected elements. A
+caller reads a projected row exactly as it reads a whole one — no flattening, no `"customer.name"`
+keys to unpick.
+
+## `omit` — dropping some of it
+
+```js
+import { search } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const [row] = search(orders, {
+  where: { id: "o-1001" },
+  omit: ["customer.email", "lines", "note"],
+});
+
+console.log(JSON.stringify(row, null, 2));
+```
+
+```
+{
+  "id": "o-1001",
+  "placed": "2026-09-01T09:15:00Z",
+  "status": "paid",
+  "channel": "web",
+  "customer": {
+    "name": "Ada Lovelace",
+    "country": "GB"
+  },
+  "total": 22.5,
+  "paid": 22.5,
+  "tags": [
+    "gift"
+  ]
+}
+```
+
+`omit` runs **after** `fields`, and that order is what redaction needs:
+
+```json
+{ "fields": ["request"], "omit": ["request.headers.cookie"] }
+```
+
+Keep the whole request object, except the one header that must not leave the process. Naming
+every header you *do* want would mean a code change every time a new one appears; naming the
+one you do not is a rule that stays true.
+
+The row is copied rather than changed, and only the objects on the way to a dropped field are
+copied — so redacting one field of a large record does not duplicate the record.
+
+## Two things `search` does that a hand-written version would not
+
+**Without a sort, it stops early.** The first page of matches is the first `skip + limit` of
+them, so the walk ends as soon as it has them rather than testing the rest of the collection.
+
+**With a sort and a limit, it keeps only the best `skip + limit`** in a bounded heap rather
+than sorting everything to keep ten. "The ten largest orders out of a million" sorts ten. This
+is the one case where `search` is genuinely *faster* than filter-sort-slice by hand, rather
+than merely equal to it.
+
+## Exercise
+
+The console shows "the five newest orders that still owe money", with the id, the customer's
+name, and enough to work out what is outstanding. Write the request.
+
+<details>
+<summary>Answer</summary>
+
+```js
+import { search } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const chase = search(orders, {
+  where: { paid: 0, status: { $nin: ["cancelled", "refunded"] } },
+  sort: { placed: -1 },
+  limit: 5,
+  fields: ["id", "customer.name", "total", "paid"],
+});
+
+for (const row of chase) console.log(row.id, row.customer.name, "owes", row.total - row.paid);
+```
+
+```
+o-1005 Tim Berners-Lee owes 170
+o-1002 Grace Hopper owes 30
+```
+
+`paid: 0` is the filter. The `$nin` is what stops a cancelled or refunded order appearing on a
+list of people to chase — worth writing down, because "unpaid" and "owes money" are not the
+same set.
+
+In lesson 10 `outstanding` becomes a computed field, and this request gets to name it
+directly instead of handing back two numbers for the screen to subtract.
+</details>
+
+## Related
+
+- [Specification §8](../reference/specification.md#8-requests) — the request envelope
+- [Specification §8.1](../reference/specification.md#81-sort-order) — the order, in full

package/docs/course/08-grouping.md

@@ -0,0 +1,210 @@
+# Lesson 8 — Counting by a field
+
+**Goal:** the number above the table, without writing the tally loop again.
+
+← [Course](index.md) · Previous: [Requests](07-requests.md) · Next: [Why did this not match?](09-explaining.md)
+
+---
+
+## Everybody writes this loop
+
+Orders per status, requests per verdict, errors per service. It is five lines of `Map` and a
+`sort`, which is why every screen writes its own — and why no two of them agree. Three
+questions decide the answer, and each project answers them differently by accident:
+
+- what does an item with **two** values count as?
+- where do the items with **no** value go?
+- what order do the groups come back in?
+
+`group` answers all three, the same way every time.
+
+## The simplest tally
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+for (const held of group(orders, "status")) console.log(held.count, held.key);
+```
+
+```
+4 paid
+2 open
+1 cancelled
+1 refunded
+```
+
+Each group is `{ key, count }`, biggest first. The second argument is a **path**, so anything
+lessons 1 to 3 taught you to reach is something you can count by:
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+for (const held of group(orders, "customer.country")) console.log(held.count, held.key);
+```
+
+```
+3 GB
+3 US
+1 DE
+1 FR
+```
+
+## Counting only some of the rows
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const big = group(orders, "status", { where: { total: { $gte: 30 } } });
+
+console.log(big.map((held) => `${held.key}=${held.count}`));
+```
+
+```
+[ 'open=2', 'cancelled=1', 'paid=1', 'refunded=1' ]
+```
+
+`where` takes an ordinary query. Put the filter **here**, not around the result: grouping
+everything and then presenting it beside a filtered table is a wrong number with nothing on
+the screen to say so.
+
+## Grouping by something that holds several values
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+console.log("by tag:");
+for (const held of group(orders, "tags")) console.log("  ", held.count, JSON.stringify(held.key));
+
+console.log("by sku:");
+for (const held of group(orders, "lines.sku")) console.log("  ", held.count, held.key);
+```
+
+```
+by tag:
+   3 "repeat"
+   2 null
+   2 "fragile"
+   1 "damaged"
+   1 "gift"
+by sku:
+   4 pad-a5
+   3 pen-fine
+   2 desk-lamp
+   2 ink-blue
+   2 ink-red
+```
+
+Two things are happening here, and both are answers to the questions at the top.
+
+**An item counts once in each group it belongs to.** `o-1004` is tagged
+`["damaged", "repeat"]` and appears under both. A value repeated *within* one item still
+counts once, because the question is "how many orders involved this sku", not "how many
+lines".
+
+So **the counts do not add up to the number of rows**, and should not: the sku tally comes to
+thirteen over eight orders. If you want the counts to add up, group by something each row has
+exactly one of.
+
+**Everything that cannot name a group shares `null`.** That is the `null` in the tag list: the
+two orders with no tags at all. A missing field, a `null`, a value that is an object, and an
+empty list all land there.
+
+That is the "unlabelled" heading on a chart, and it is deliberately a real answer rather than
+a row that quietly vanished. A count that silently omits rows is worse than no count, because
+it still looks like a count.
+
+## Order and size
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const line = (groups) => groups.map((held) => `${held.key}=${held.count}`).join("  ");
+
+console.log("biggest first (default):", line(group(orders, "status")));
+console.log("alphabetically by key  :", line(group(orders, "status", { sort: "key" })));
+console.log("the top two            :", line(group(orders, "status", { limit: 2 })));
+```
+
+```
+biggest first (default): paid=4  open=2  cancelled=1  refunded=1
+alphabetically by key  : cancelled=1  open=2  paid=4  refunded=1
+the top two            : paid=4  open=2
+```
+
+Ties break by the key, so the same data always comes back in the same order and a chart does
+not reshuffle itself between two refreshes that found nothing new.
+
+## Keeping the rows behind each group
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+for (const held of group(orders, "channel", { items: 2 })) {
+  console.log(String(held.key).padEnd(6), held.count, "→", held.items.map((order) => order.id).join(" "));
+}
+```
+
+```
+web    4 → o-1001 o-1002
+app    2 → o-1005 o-1006
+phone  2 → o-1003 o-1008
+```
+
+`items: true` keeps every row; `items: 2` keeps the first two. This is what turns a chart into
+a chart you can click — and the cap is why a group of ten thousand rows does not arrive whole
+in a dashboard payload.
+
+## The options
+
+| | |
+| --- | --- |
+| `where` | which items to count |
+| `sort` | `"count"` (the default) or `"key"` |
+| `limit` | how many groups, largest first |
+| `items` | `true` to keep every item, or a number to keep the first few |
+
+## Exercise
+
+Harbour wants "money outstanding, by country" — only orders that still owe, largest first.
+`group` counts rows rather than summing a field, so this one needs a little thought.
+
+<details>
+<summary>Answer</summary>
+
+`group` deliberately counts rather than sums. A general aggregation language is a much bigger
+thing than a query language, and this one stops at the edge on purpose. Keep the rows and sum
+them yourself:
+
+```js
+import { group } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+const owing = group(orders, "customer.country", {
+  where: { paid: 0, status: "open" },
+  items: true,
+});
+
+for (const held of owing) {
+  const total = held.items.reduce((sum, order) => sum + order.total, 0);
+  console.log(held.key, held.count, "order(s),", total, "outstanding");
+}
+```
+
+```
+GB 1 order(s), 170 outstanding
+US 1 order(s), 30 outstanding
+```
+
+If you find yourself really wanting a `$sum` here, that is the signal that the *store* should
+be answering this question rather than your process — which is lesson 15.
+</details>
+
+## Related
+
+- [Library API](../reference/api.md) — `group` and its options