@osqd/jql 0.1.1

package/docs/course/13-saved-filters.md

@@ -0,0 +1,199 @@
+# Lesson 13 — Saved filters
+
+**Goal:** give every question one shape and one short name, so a list of saved filters does
+not quietly hold the same filter three times.
+
+← [Course](index.md) · Previous: [Queries from outside](12-untrusted.md) · Next: [Logs and streams](14-streams-and-cli.md)
+
+---
+
+## The same question, four spellings
+
+Once people can save a filter, they save the same one repeatedly. One was typed into the
+search box, one was written by hand, one was built by a form, one came back from an API with
+its keys in a different order.
+
+They are one question and four rows in a table — and a cache keyed on the query text misses on
+every one of them.
+
+Two functions fix that:
+
+- **`canonical(query)`** rewrites a query into one fixed shape;
+- **`fingerprint(query)`** names that shape in sixteen characters.
+
+## One filter, one name
+
+```js
+import { fingerprint } from "@osqd/jql";
+
+const written = [
+  { status: "open", total: { $gte: 100 } },
+  { total: { $gte: 100 }, status: "open" },
+  { status: { $eq: "open" }, total: { $gte: 100 } },
+  { $and: [{ status: "open" }, { total: { $gte: 100 } }], $comment: "chasing" },
+];
+
+for (const query of written) console.log(fingerprint(query), JSON.stringify(query));
+
+console.log("distinct filters here:", new Set(written.map(fingerprint)).size);
+```
+
+```
+03646d4ac8b6def3 {"status":"open","total":{"$gte":100}}
+03646d4ac8b6def3 {"total":{"$gte":100},"status":"open"}
+03646d4ac8b6def3 {"status":{"$eq":"open"},"total":{"$gte":100}}
+03646d4ac8b6def3 {"$and":[{"status":"open"},{"total":{"$gte":100}}],"$comment":"chasing"}
+distinct filters here: 1
+```
+
+Four spellings, one name. Note the fourth in particular: a `$and` of two clauses, and a
+comment, and it still lands on the same fingerprint as the other three.
+
+## The shape behind the name
+
+```js
+import { canonical } from "@osqd/jql";
+
+const typed = { $and: [{ status: "open" }, { total: { $gte: 100 } }], $comment: "chasing" };
+
+console.log(JSON.stringify(canonical(typed), null, 2));
+```
+
+```
+{
+  "status": {
+    "$eq": "open"
+  },
+  "total": {
+    "$gte": 100
+  }
+}
+```
+
+What comes back **is still a query** — you can run it, store it, or put it in a URL — and it
+matches exactly what the original matched. The test suite holds it to that on thousands of
+generated queries by running both and comparing answers, rather than by comparing shapes.
+
+| Written | Becomes |
+| --- | --- |
+| `{ a: 1 }` | `{ a: { $eq: 1 } }` — a bare value is a condition |
+| keys in any order | keys sorted |
+| `$comment: "…"` | dropped: a note decides nothing |
+| `{ $and: [x, y] }` | `{ …x, …y }` — the box writes the first, a person writes the second |
+| `$and` inside `$and` | one `$and` |
+| `{ $in: [3, 1, 3] }` | `{ $in: [1, 3] }` — a list that is a set is sorted and deduplicated |
+| `$options: "ui"` | `$options: "iu"` |
+| the branches of an `$or` | sorted, so branch order stops mattering |
+
+## Different questions, different names
+
+```js
+import { fingerprint } from "@osqd/jql";
+
+const questions = [
+  { status: "open" },
+  { status: "paid" },
+  { status: { $ne: "open" } },
+  { $or: [{ status: "open" }, { status: "paid" }] },
+  { $or: [{ status: "paid" }, { status: "open" }] },
+];
+
+for (const query of questions) console.log(fingerprint(query), JSON.stringify(query));
+```
+
+```
+d4d13aede48bfdad {"status":"open"}
+0c24bc335f3c3269 {"status":"paid"}
+a5ba405e8cd835c8 {"status":{"$ne":"open"}}
+bc66ac7eefa26bc0 {"$or":[{"status":"open"},{"status":"paid"}]}
+bc66ac7eefa26bc0 {"$or":[{"status":"paid"},{"status":"open"}]}
+```
+
+The last two are the same question with its branches the other way round, so they share a
+name.
+
+## What it deliberately will not do
+
+```js
+import { fingerprint } from "@osqd/jql";
+
+console.log(fingerprint({ total: { $gt: 3 } }), "{ total: { $gt: 3 } }");
+console.log(fingerprint({ total: { $gte: 4 } }), "{ total: { $gte: 4 } }");
+```
+
+```
+e7a73371277f4960 { total: { $gt: 3 } }
+a27bb5f54416fc83 { total: { $gte: 4 } }
+```
+
+Those two agree on every integer, and disagree on 3.5. They get different names, and that is
+the honest limit of the whole idea:
+
+> **Equal fingerprints mean the same query. Different fingerprints mean only that the queries
+> are written differently.**
+
+One direction is a cheap, decidable property. The other — deciding whether two arbitrary
+queries select the same rows — is not, and a tool that claimed to do it would be wrong in ways
+you would find out about much later.
+
+## About the name itself
+
+A fingerprint is a 64-bit checksum in hexadecimal. It is **not** a cryptographic hash: two
+different queries can collide, rarely, and never in a way that matters for a cache you are
+allowed to miss.
+
+It **is** stable across processes and across versions of this library, so it is safe as a
+stored key. A set of fingerprints is written into the test suite precisely so that changing
+what a query is called takes a deliberate act rather than happening as a side effect.
+
+## Where this earns its place
+
+- **A saved-filter list.** Fingerprint before inserting, and say "you already have this one,
+  saved as *Chasing large orders*" instead of adding a fifth copy.
+- **A cache key.** `` `orders:${fingerprint(query)}` ``.
+- **A log line.** Sixteen characters that say which filter ran, without printing a query that
+  may be long and may contain a customer's email address.
+- **Telling two dashboards apart** when both claim to show "open orders" and one of them is
+  wrong.
+
+## Exercise
+
+Harbour's filter list stores `{ name, query }`. Write `save(list, name, query)` so that saving
+a duplicate gives back the existing entry instead of adding a row.
+
+<details>
+<summary>Answer</summary>
+
+```js
+import { canonical, fingerprint } from "@osqd/jql";
+
+const save = (list, name, query) => {
+  const key = fingerprint(query);
+  const already = list.find((entry) => entry.key === key);
+  if (already !== undefined) return { saved: false, as: already.name };
+  list.push({ key, name, query: canonical(query) });
+  return { saved: true, as: name };
+};
+
+const list = [];
+
+console.log(save(list, "Chasing large orders", { status: "open", total: { $gte: 100 } }));
+console.log(save(list, "Big open ones", { total: { $gte: 100 }, status: { $eq: "open" } }));
+console.log("rows in the list:", list.length);
+```
+
+```
+{ saved: true, as: 'Chasing large orders' }
+{ saved: false, as: 'Chasing large orders' }
+rows in the list: 1
+```
+
+Store `canonical(query)` rather than whatever was typed, so the row you read back tomorrow has
+one shape instead of whichever spelling happened to arrive first. Keep the fingerprint beside
+it, so the lookup is an index rather than a scan.
+</details>
+
+## Related
+
+- [Library API](../reference/api.md) — `canonical` and `fingerprint`
+- [Specification §13](../reference/specification.md#13-canonical-form-optional) — the rules, so another implementation agrees

package/docs/course/14-streams-and-cli.md

@@ -0,0 +1,276 @@
+# Lesson 14 — Logs and streams
+
+**Goal:** run a query over something that arrives over time, and over a file, from a shell.
+
+← [Course](index.md) · Previous: [Saved filters](13-saved-filters.md) · Next: [Pushing into a store](15-pushdown.md)
+
+---
+
+## Data that arrives over time
+
+Harbour exports its order book nightly as JSON lines. That is the shape most real data arrives
+in — a log, a `kubectl` dump, a store's export — and by the time it matters it is too big to
+hold in memory.
+
+The asynchronous helpers are the same engine with an `await` in the loop. The query is
+compiled once, before the first item, and **they stop reading as soon as the answer cannot
+change** — which is what makes a source that never ends answerable at all.
+
+First, write the export:
+
+```bash
+node -e "import('./orders.mjs').then(({orders}) => require('node:fs').writeFileSync('orders.jsonl', orders.map(o => JSON.stringify(o)).join('\n') + '\n'))"
+```
+
+Then save this as `harbour/exported.mjs`, so the examples can read it back a line at a time:
+
+```js
+import { createReadStream } from "node:fs";
+import { createInterface } from "node:readline";
+
+/** Every line of the nightly export, parsed, as it arrives. */
+export async function* exported() {
+  const reader = createInterface({
+    input: createReadStream("orders.jsonl", "utf8"),
+    crlfDelay: Number.POSITIVE_INFINITY,
+  });
+  for await (const line of reader) if (line.trim() !== "") yield JSON.parse(line);
+}
+```
+
+## Asking a source the usual questions
+
+```js
+import { countAsync, findAsync, someAsync } from "@osqd/jql";
+import { exported } from "./exported.mjs";
+
+console.log("the first open one:", (await findAsync(exported(), { status: "open" })).id);
+console.log("how many paid     :", await countAsync(exported(), { status: "paid" }));
+console.log("anything over £200:", await someAsync(exported(), { total: { $gt: 200 } }));
+```
+
+```
+the first open one: o-1002
+how many paid     : 4
+anything over £200: true
+```
+
+Same names as lesson 1, same queries, `await` in front. `findAsync` and `someAsync` stop at
+the first match — the rest of the file is never read.
+
+## A page from a stream
+
+```js
+import { searchAsync } from "@osqd/jql";
+import { exported } from "./exported.mjs";
+
+const page = await searchAsync(exported(), {
+  where: { status: { $ne: "cancelled" } },
+  sort: { total: -1 },
+  limit: 2,
+  fields: ["id", "total"],
+});
+
+console.log(page);
+```
+
+```
+[ { id: 'o-1005', total: 170 }, { id: 'o-1004', total: 36 } ]
+```
+
+## A stream you can walk away from
+
+```js
+import { filterStream } from "@osqd/jql";
+import { exported } from "./exported.mjs";
+
+for await (const order of filterStream(exported(), { "customer.country": "GB" })) {
+  console.log(order.id, order.customer.name);
+  if (order.id === "o-1003") break;
+}
+
+console.log("stopped early, and the file stopped being read");
+```
+
+```
+o-1001 Ada Lovelace
+o-1003 Alan Turing
+stopped early, and the file stopped being read
+```
+
+`filterStream` yields matches as it finds them. Stop consuming and the source stops being
+pulled — which is the whole point of the next example.
+
+## A source that never ends
+
+```js
+import { filterStream } from "@osqd/jql";
+
+/** A feed with no end, like a socket or a tail. */
+async function* forever() {
+  for (let i = 0; ; i++) yield { id: i, status: i % 3 === 0 ? "open" : "paid" };
+}
+
+const pulled = [];
+for await (const item of filterStream(forever(), { status: "open" })) {
+  pulled.push(item.id);
+  if (pulled.length === 3) break;
+}
+
+console.log(pulled);
+```
+
+```
+[ 0, 3, 6 ]
+```
+
+That program terminates. Nothing pulls from `forever()` once the loop breaks, so an endless
+feed is an ordinary source as long as the question has a stopping point.
+
+## What stops early, and what cannot
+
+| | |
+| --- | --- |
+| `findAsync`, `someAsync` | stop at the first match |
+| `filterStream` | yields as it goes; stop consuming and the source stops |
+| `searchAsync` **without** a sort | stops once the page is full |
+| `searchAsync` **with** a sort | reads everything — it has to — but holds only `skip + limit` items while it does |
+
+That last row is the one that surprises people, and it is worth being explicit: an order over
+items you have not seen yet is not an order. A sorted page needs the whole source. What it
+does *not* need is the whole source in memory, and the bounded heap from lesson 7 keeps only
+the page you asked for.
+
+## The same thing from a shell
+
+The library ships the command that does all this over a file, which is often quicker than
+writing the script:
+
+```bash
+jql '{"status":"open"}' orders.jsonl | head -1
+```
+
+```
+{"id":"o-1002","placed":"2026-09-03T14:02:00Z","status":"open","channel":"web","customer":{"name":"Grace Hopper","country":"US","email":"grace@example.com"},"lines":[{"sku":"pad-a5","title":"A5 pad","quantity":10,"price":3}],"total":30,"paid":0,"tags":[]}
+```
+
+```bash
+jql --count '{"customer.country":"GB"}' orders.jsonl
+```
+
+```
+3
+```
+
+```bash
+jql --text 'status:paid total:>20' --sort total:desc --fields id,total orders.jsonl
+```
+
+```
+{"id":"o-1006","total":33}
+{"id":"o-1001","total":22.5}
+```
+
+```bash
+jql --group status '{}' orders.jsonl
+```
+
+```
+{"key":"paid","count":4}
+{"key":"open","count":2}
+{"key":"cancelled","count":1}
+{"key":"refunded","count":1}
+```
+
+```bash
+cat orders.jsonl | jql --text 'customer.country:gb' --fields id,customer.name -
+```
+
+```
+{"id":"o-1001","customer":{"name":"Ada Lovelace"}}
+{"id":"o-1003","customer":{"name":"Alan Turing"}}
+{"id":"o-1005","customer":{"name":"Tim Berners-Lee"}}
+```
+
+Two rules shape the command, and both are worth copying into tools of your own.
+
+### stdout is the artifact
+
+Matching lines and nothing else, so `jql … > kept.jsonl` produces a file worth having. Counts,
+warnings and explanations go to stderr, which is why `--explain` combines with anything:
+
+```bash
+jql --explain --count '{"status":"paid"}' orders.jsonl
+```
+
+```
+4
+```
+
+— on stdout, with the explanation beside it on stderr.
+
+A matching line is written back **exactly as it arrived**, byte for byte, so filtering a file
+cannot change the data in it. That is not fussiness: re-serialising turns a thirty-digit
+identifier into `1.2345678901234568e+29`, and `1e400` into `null`, silently. Ask for the
+result to be reshaped — `--fields`, `--omit`, `--group`, `--pretty` — and you get JSON,
+because then you have asked for something other than the line.
+
+### A value is validated, never coerced
+
+`--limit all` is an error rather than a limit that quietly stops limiting, and a query that is
+not JQL is refused with the reason and exit code 2:
+
+```bash
+jql '{"total":{"$gtt":1}}' orders.jsonl; echo "exit $?"
+```
+
+```
+jql: at total.$gtt: "$gtt" is not an operator; did you mean "$gt"?
+exit 2
+```
+
+Note what `--text` did in the examples above: `customer.country:gb` uses the **storage path**,
+because there is no vocabulary on the command line to translate a short name. Everything in
+lesson 10 still applies in your own code; the command is the generic tool.
+
+## Exercise
+
+Find the three largest unpaid orders in the export, printing only the id and the total, from
+the shell alone.
+
+<details>
+<summary>Answer</summary>
+
+```bash
+jql '{"paid":0}' --sort total:desc --limit 3 --fields id,total orders.jsonl
+```
+
+```
+{"id":"o-1007","total":280}
+{"id":"o-1005","total":170}
+{"id":"o-1004","total":36}
+```
+
+Now try the spelling you probably reached for first:
+
+```bash
+jql --text 'paid:0' --sort total:desc --limit 3 --fields id,total orders.jsonl
+```
+
+```
+```
+
+Nothing at all. **There is no vocabulary on the command line**, so every field is `text` — and
+`paid:0` becomes `{ paid: { $contains: "0", $options: "i" } }`, which asks whether a *string*
+contains a zero. `paid` holds a number, and `$contains` does not match numbers.
+
+That is the argument for lesson 10 in a single line. A `kind` is what turns `paid:0` into a
+comparison; without one the text syntax can only guess from the shape of what was typed, and
+it guesses conservatively rather than coercing. On the command line, prefer JSON for anything
+that is not prose.
+</details>
+
+## Related
+
+- [The command](../reference/cli.md) — every option, the exit codes, the two rules
+- [Library API](../reference/api.md) — the `*Async` helpers