@osqd/jql 0.1.1

package/docs/course/11-the-search-box.md

@@ -0,0 +1,349 @@
+# Lesson 11 — The search box
+
+**Goal:** let people type, and get back the same JQL you would have written yourself.
+
+← [Course](index.md) · Previous: [A vocabulary](10-vocabulary.md) · Next: [Queries from outside](12-untrusted.md)
+
+---
+
+## Nobody types JSON into a search box
+
+`@osqd/jql/text` is the syntax people *do* type:
+
+```
+status:open total:>100 -has:note
+```
+
+It has no matcher of its own. It **compiles into a JQL document**, and the engine you have
+been using for ten lessons runs that. There is one set of semantics, and the text is a second
+way of writing it.
+
+That matters more than it sounds. A search box with its own matcher drifts from the JSON one
+within a week, and then a saved filter means one thing on the screen and another in the
+export.
+
+## Typing into it
+
+```js
+import { filter } from "@osqd/jql";
+import { parseText } from "@osqd/jql/text";
+import { orders } from "./orders.mjs";
+import { ORDERS } from "./vocabulary.mjs";
+
+// A fixed clock, so "the last week" means the same thing every time this runs.
+const options = { vocabulary: ORDERS, now: () => Date.parse("2026-09-25T12:00:00Z") };
+
+const box = (typed) => {
+  const query = parseText(typed, { vocabulary: ORDERS });
+  const found = filter(orders, query, options).map((o) => o.id).join(" ") || "(none)";
+  console.log(typed.padEnd(28), found);
+};
+
+box("status:open");
+box("status:open country:gb");
+box("-status:paid");
+box("status:$in(open, cancelled)");
+```
+
+```
+status:open                  o-1002 o-1005
+status:open country:gb       o-1005
+-status:paid                 o-1002 o-1004 o-1005 o-1007
+status:$in(open, cancelled)  o-1002 o-1005 o-1007
+```
+
+Adjacent terms are **and**, which is what narrowing means to the person typing. `-` is not,
+and `$in(…)` is one of a set.
+
+### Numbers and dates
+
+```js
+import { filter } from "@osqd/jql";
+import { parseText } from "@osqd/jql/text";
+import { orders } from "./orders.mjs";
+import { ORDERS } from "./vocabulary.mjs";
+
+const options = { vocabulary: ORDERS, now: () => Date.parse("2026-09-25T12:00:00Z") };
+
+const box = (typed) => {
+  const found = filter(orders, parseText(typed, { vocabulary: ORDERS }), options);
+  console.log(typed.padEnd(28), found.map((o) => o.id).join(" ") || "(none)");
+};
+
+box("total:>100");
+box("total:10..40");
+box("at:>2026-09-18");
+box("at:>-7d");
+```
+
+```
+total:>100                   o-1005 o-1007
+total:10..40                 o-1001 o-1002 o-1004 o-1006 o-1008
+at:>2026-09-18               o-1006 o-1007 o-1008
+at:>-7d                      o-1006 o-1007 o-1008
+```
+
+`total` and `placed` were declared `number` and `date` in the vocabulary, which is what lets
+`>100` and `>-7d` be read as comparisons rather than searched for as text. That is the whole
+job of `kind`.
+
+### Presence, alternatives and bare words
+
+```js
+import { filter } from "@osqd/jql";
+import { parseText } from "@osqd/jql/text";
+import { orders } from "./orders.mjs";
+import { ORDERS } from "./vocabulary.mjs";
+
+const box = (typed) => {
+  const found = filter(orders, parseText(typed, { vocabulary: ORDERS }), { vocabulary: ORDERS });
+  console.log(typed.padEnd(28), found.map((o) => o.id).join(" ") || "(none)");
+};
+
+box("has:note");
+box("-has:note");
+box("sku:desk-lamp $or tag:gift");
+box("ada");
+box('"gift wrap"');
+```
+
+```
+has:note                     o-1001 o-1003 o-1004 o-1006 o-1007
+-has:note                    o-1002 o-1005 o-1008
+sku:desk-lamp $or tag:gift   o-1001 o-1005 o-1007
+ada                          o-1001 o-1008
+"gift wrap"                  o-1006
+```
+
+A bare word searches the `text` fields the vocabulary listed — `id`, `who`, `email`, `sku` and
+`note` — so `ada` finds both Adas without anybody naming a field. A quoted phrase is one term.
+
+## The syntax, in one table
+
+| Typed | Means |
+| --- | --- |
+| `status:open` | the field, read according to its `kind` |
+| `status:open country:gb` | adjacent terms are **and** |
+| `-status:paid`, `!status:paid` | not |
+| `status:$in(open, cancelled)` | one of a set; `$notin(…)` for none of them |
+| `total:>100`, `total:>=100`, `total:10..20` | comparisons and ranges, on a `number` or `date` field |
+| `at:>2026-09-18`, `at:>-7d` | an instant, or a signed duration from now |
+| `has:note`, `-has:note` | the field is set, or is not |
+| `a $or b`, `$not a`, `(a $or b) c` | precedence: `$not` tightest, then and, then `$or` |
+| `ada`, `"gift wrap"` | a bare word or phrase, in the vocabulary's `text` fields |
+
+Operators carry a `$` on purpose. A bare `or` is a word that appears in notes and addresses,
+and a language where an ordinary search word silently becomes an operator is a language that
+lies about what it matched.
+
+## What it compiles to
+
+```js
+import { parseText } from "@osqd/jql/text";
+import { ORDERS } from "./vocabulary.mjs";
+
+console.log(JSON.stringify(parseText("status:open total:>100", { vocabulary: ORDERS }), null, 2));
+```
+
+```
+{
+  "$and": [
+    {
+      "status": {
+        "$eq": "open",
+        "$options": "i"
+      }
+    },
+    {
+      "total": {
+        "$gt": 100
+      }
+    }
+  ]
+}
+```
+
+Nothing exotic: it is the query you would have written by hand. `$options: "i"` is there
+because `status` was declared `exact`, which means "equals it, ignoring case".
+
+## The parser never throws
+
+It runs on **every keystroke**, so half-typed input is the normal state rather than an error:
+
+```js
+import { count } from "@osqd/jql";
+import { parseText } from "@osqd/jql/text";
+import { orders } from "./orders.mjs";
+import { ORDERS } from "./vocabulary.mjs";
+
+const halfTyped = parseText("status:open $and (total:>", { vocabulary: ORDERS });
+
+console.log(JSON.stringify(halfTyped));
+console.log("matches", count(orders, halfTyped, { vocabulary: ORDERS }), "of 8");
+```
+
+```
+{"$and":[{"status":{"$eq":"open","$options":"i"}},{"total":{"$in":[]}}]}
+matches 0 of 8
+```
+
+An unclosed bracket, a dangling `$or`, a `$in(` with nothing after it: each parses to the best
+reading available. Look at what the unfinished half became — `{ "$in": [] }`, a term that
+matches **nothing**.
+
+That direction is chosen deliberately. A half-typed filter that quietly *widened* to everything
+would show the operator a screen full of rows that are not the ones they are looking for, and
+nothing on the screen would say the filter was incomplete.
+
+## Putting a saved filter back in the box
+
+A dashboard that stores filters as JSON has to show one in the box when somebody opens it to
+edit. That is `toText`:
+
+```js
+import { toText } from "@osqd/jql/text";
+import { ORDERS } from "./vocabulary.mjs";
+
+const stored = [
+  { status: { $eq: "open", $options: "i" } },
+  { $and: [{ country: { $eq: "GB", $options: "i" } }, { total: { $gt: 100 } }] },
+  { placed: { $gte: { $date: { $ago: "7d" } } } },
+  { $not: { tag: { $eq: "gift", $options: "i" } } },
+  { total: { $gte: 10, $lte: 40 } },
+];
+
+for (const query of stored) console.log(JSON.stringify(toText(query, { vocabulary: ORDERS }).text));
+```
+
+```
+"status:open"
+"country:GB total:>100"
+"placed:>=-7d"
+"-tag:gift"
+"total:10..40"
+```
+
+Parsed back with the same vocabulary, each of those means precisely the clauses it carries —
+a property the test suite holds the writer to on thousands of generated queries.
+
+### When the box cannot say it
+
+`toText` is deliberately **not** total. The text syntax is smaller than JQL, and some queries
+have no spelling in it:
+
+```js
+import { toText } from "@osqd/jql/text";
+import { ORDERS } from "./vocabulary.mjs";
+
+const hard = { status: { $eq: "open", $options: "i" }, tag: { $size: 2 } };
+const form = toText(hard, { vocabulary: ORDERS });
+
+console.log("text       :", JSON.stringify(form.text));
+console.log("complete   :", form.complete);
+console.log("unexpressed:", JSON.stringify(form.unexpressed, null, 2));
+```
+
+```
+text       : "status:open"
+complete   : false
+unexpressed: [
+  {
+    "at": "tag.$size",
+    "clause": {
+      "$size": 2
+    },
+    "why": "\"$size\" has no spelling in the text syntax"
+  }
+]
+```
+
+This is the contract, and it is worth reading twice. The text holds only the part the box
+*could* express — `status:open` — and says nothing at all about the `$size` constraint that is
+still in force. So `complete` is `false`, and `unexpressed` names the clause that got left
+behind and why.
+
+A caller that shows `form.text` and ignores `form.complete` has just shown somebody a filter
+**wider than the one that is running**. The signature returns all three so that the check is
+one line:
+
+```js
+import { toText } from "@osqd/jql/text";
+import { ORDERS } from "./vocabulary.mjs";
+
+const show = (query) => {
+  const form = toText(query, { vocabulary: ORDERS });
+  console.log(form.complete ? "shows it all " : "⚠ incomplete  ", JSON.stringify(form.text));
+};
+
+show({ status: { $eq: "open", $options: "i" } });
+show({ status: { $eq: "open", $options: "i" }, tag: { $size: 2 } });
+```
+
+```
+shows it all  "status:open"
+⚠ incomplete   "status:open"
+```
+
+## Completions
+
+```js
+import { suggest } from "@osqd/jql/text";
+import { ORDERS } from "./vocabulary.mjs";
+
+const caretAt = [["stat", 4], ["status:", 7], ["status:op", 9], ["-co", 3], ["status:open c", 13], ["$", 1]];
+
+for (const [typed, caret] of caretAt) {
+  const { options, from, to } = suggest(typed, caret, ORDERS);
+  console.log(`${JSON.stringify(typed).padEnd(16)} replace [${from},${to}] with: ${options.join(" ") || "(nothing)"}`);
+}
+```
+
+```
+"stat"           replace [0,4] with: status:
+"status:"        replace [0,7] with: status:open status:paid status:refunded status:cancelled
+"status:op"      replace [0,9] with: status:open
+"-co"            replace [0,3] with: -country:
+"status:open c"  replace [12,13] with: channel: country:
+"$"              replace [0,1] with: $and $or $not $in $notin
+```
+
+`from` and `to` are the span to replace, so inserting a completion does not disturb the rest
+of the input.
+
+The values come from the `values` you listed in the vocabulary. A field that can hold anything
+offers none, because guessing there would be inventing options rather than completing them.
+
+## Exercise
+
+Harbour's console has a saved filter: `{ status: { $eq: "open", $options: "i" }, outstanding: { $gt: 100 } }`.
+Put it in the box and say what the operator sees.
+
+<details>
+<summary>Answer</summary>
+
+```js
+import { toText } from "@osqd/jql/text";
+import { ORDERS } from "./vocabulary.mjs";
+
+const form = toText({ status: { $eq: "open", $options: "i" }, outstanding: { $gt: 100 } }, { vocabulary: ORDERS });
+
+console.log(form);
+```
+
+```
+{
+  text: 'status:open outstanding:>100',
+  complete: true,
+  unexpressed: []
+}
+```
+
+A computed field is a name like any other here — the box neither knows nor cares that
+`outstanding` is worked out rather than stored. Type that text back in and you get the same
+query.
+</details>
+
+## Related
+
+- [Text syntax](../reference/text-syntax.md) — every term, and what each `kind` does with it
+- [Library API](../reference/api.md) — `parseText`, `toText`, `suggest`

package/docs/course/12-untrusted.md

@@ -0,0 +1,257 @@
+# Lesson 12 — Queries from outside
+
+**Goal:** accept a query from a URL or a request body without accepting *every* query.
+
+← [Course](index.md) · Previous: [The search box](11-the-search-box.md) · Next: [Saved filters](13-saved-filters.md)
+
+---
+
+## A filter in a URL is a lovely API
+
+…right up until it is a way to read every field of every row, and spend a minute of CPU doing
+it. A query is data, which is what lets it travel — and what lets one arrive from somebody you
+have never met.
+
+Three independent things make that safe, and a fourth ties them together:
+
+| | |
+| --- | --- |
+| **the untrusted limits** | bound **cost** — size, depth, patterns |
+| **an allowlist** | bounds **capability** — the operators this endpoint answers |
+| **a strict vocabulary** | bounds **surface** — the fields it will talk about |
+| **`validate`** | one call that answers 400 with a sentence instead of 500 with a stack |
+
+## The endpoint
+
+Save this as `harbour/public-api.mjs`; the examples below use it.
+
+```js
+import { UNTRUSTED_LIMITS, defineVocabulary, validate } from "@osqd/jql";
+import { orders } from "./orders.mjs";
+
+/** The fields a stranger may ask about — and, because of `strict`, no others. */
+const PUBLIC = defineVocabulary()({
+  fields: {
+    status: { kind: "exact", values: ["open", "paid", "refunded", "cancelled"] },
+    country: { path: "customer.country", kind: "exact" },
+    total: { kind: "number" },
+    placed: { kind: "date" },
+  },
+  strict: true,
+});
+
+/** The untrusted caps, plus the short menu of operators this endpoint answers. */
+const limits = {
+  ...UNTRUSTED_LIMITS,
+  allowOperators: ["$eq", "$ne", "$in", "$gt", "$gte", "$lt", "$lte", "$and", "$or"],
+};
+
+/** One request: JSON in, a status line out. */
+export function endpoint(body) {
+  const checked = validate(JSON.parse(body), { vocabulary: PUBLIC, limits });
+  if (!checked.valid) return `400 ${checked.error.message}`;
+  return `200 ${orders.filter(checked.test).map((o) => o.id).join(" ") || "(no matches)"}`;
+}
+```
+
+## What it accepts
+
+```js
+import { endpoint } from "./public-api.mjs";
+
+console.log(endpoint('{"status":"open"}'));
+console.log(endpoint('{"country":"GB","total":{"$gte":20}}'));
+console.log(endpoint('{"placed":{"$gte":{"$date":"2026-09-18"}}}'));
+console.log(endpoint('{"status":{"$in":[]}}'));
+```
+
+```
+200 o-1002 o-1005
+200 o-1001 o-1005
+200 o-1006 o-1007 o-1008
+200 (no matches)
+```
+
+Ordinary queries, named through the vocabulary. The last one matches nothing and is still a
+perfectly valid request — "no results" is an answer, not an error.
+
+## What it refuses, and what it says
+
+```js
+import { endpoint } from "./public-api.mjs";
+
+for (const body of [
+  '{"customer.email":"ada@example.com"}',
+  '{"note":{"$contains":"gift"}}',
+  '{"total":{"$gtt":20}}',
+  '{"status":{"$regex":"^o"}}',
+  '{"$text":"ada"}',
+]) {
+  console.log(body);
+  console.log("  ", endpoint(body));
+}
+```
+
+```
+{"customer.email":"ada@example.com"}
+   400 at customer.email: "customer.email" is not a field here
+{"note":{"$contains":"gift"}}
+   400 at note: "note" is not a field here
+{"total":{"$gtt":20}}
+   400 at total.$gtt: "$gtt" is not an operator this query may use ($eq, $ne, $in, $gt, $gte, $lt, $lte, $and, …)
+{"status":{"$regex":"^o"}}
+   400 at status.$regex: "$regex" is not an operator this query may use ($eq, $ne, $in, $gt, $gte, $lt, $lte, $and, …)
+{"$text":"ada"}
+   400 at $text: "$text" is not an operator this query may use ($eq, $ne, $in, $gt, $gte, $lt, $lte, $and, …)
+```
+
+Every refusal is a sentence you can hand straight back to the caller. It quotes only the query
+they sent, and it names the place in it — so a partner integrating against your API can fix
+their request without opening a support ticket.
+
+## Why `validate` and not `try`/`compile`
+
+`validate` **hands back the predicate it compiled**. That is not a convenience; it closes a
+real hole. The obvious alternative has a gap in the middle of it:
+
+```ts
+function handler(body: string) {
+  const query = JSON.parse(body);
+  if (!isValid(query, { limits: UNTRUSTED_LIMITS })) return respond(400);
+  const test = compile(query);            // ← the untrusted limits are gone
+  return respond(200, orders.filter(test));
+}
+```
+
+```
+!! THREW
+file:///tmp/claude-1000/-home-micha-Documents-OSQD/b2e0526c-49d7-4fed-9959-41234adfc826/scratchpad/harbour/authoring-3.mjs:1
+if (!isValid(query, { limits: UNTRUSTED_LIMITS })) return respond(400);
+                                                   ^^^^^^
+
+SyntaxError: Illegal return statement
+    at compileSourceTextModule (node:internal/modules/esm/utils:346:16)
+```
+
+Two compiles, two sets of options, and the check no longer describes the thing that runs.
+Nothing in that code looks wrong. With `validate`, the query you checked *is* the query you
+run.
+
+## The limits
+
+```js
+import { DEFAULT_LIMITS, UNTRUSTED_LIMITS } from "@osqd/jql";
+
+for (const [name, limits] of [["default", DEFAULT_LIMITS], ["untrusted", UNTRUSTED_LIMITS]]) {
+  console.log(name);
+  for (const [key, value] of Object.entries(limits)) console.log("  ", key.padEnd(18), JSON.stringify(value));
+}
+```
+
+```
+default
+   maxDepth           32
+   maxNodes           10000
+   maxPatternLength   1024
+   maxGlobLength      1024
+   maxTextDepth       16
+   allowRegex         true
+   allowOperators     "all"
+untrusted
+   maxDepth           16
+   maxNodes           512
+   maxPatternLength   0
+   maxGlobLength      256
+   maxTextDepth       8
+   allowRegex         false
+   allowOperators     "all"
+```
+
+`UNTRUSTED_LIMITS` halves the depth, caps the query at 512 fields and operators, and **turns
+`$regex` off** — `maxPatternLength: 0`.
+
+That last one is the important one. No engine can tell a pattern that backtracks for a minute
+(`(a+)+$` against a long run of `a`s) from one that returns immediately, without running it.
+The string operators from lesson 2 answer almost everything a pattern would, in linear time,
+and they stay available.
+
+Every cap **refuses** rather than truncates. A query quietly cut short answers a different
+question, and nobody downstream can tell that it did.
+
+## The allowlist
+
+`$regex` being off is a decision this library makes for you. Which of the *rest* your endpoint
+wants is a decision only you can make: `$text` walks a whole document, `$elemMatch` carries a
+query of its own. An endpoint that needs neither says so once, in `allowOperators`, rather
+than finding out later which of them somebody used.
+
+```js
+import { validate } from "@osqd/jql";
+
+const typo = validate({ a: 1 }, { limits: { allowOperators: ["$eq", "$gte", "$grater"] } });
+
+console.log(typo.valid ? "accepted" : typo.error.message);
+```
+
+```
+at limits.allowOperators: "$grater" is not an operator, so allowing it allows nothing
+```
+
+A typo in the allowlist is refused too. An allowlist that allows nothing while looking like it
+allows something is precisely the failure this library exists to prevent.
+
+## The strict vocabulary
+
+This is the defence people skip and regret. Without it a query can reach **every field of
+every row you run it over** — including the ones you added last week and never meant to
+publish. `customer.email` and `note` were refused above not because they are secret, but
+because they were never promised.
+
+## What this is not
+
+**Not an access control layer.** A query runs over the rows you hand it. If a caller must not
+see a row, do not put the row in the collection — filter first, then run their query over what
+is left.
+
+**Not a bound on the request.** A `$in` of a million values is a legitimate query, and `$in`
+lists deliberately do not count towards `maxNodes`, because looking one item up in a set costs
+the same however long the list is. Cap the body where you read it.
+
+## Exercise
+
+Harbour's public API should let a partner ask about their own orders by status and date, but
+never about money. Write the vocabulary, and prove it by showing what happens to
+`{"total":{"$gte":100}}`.
+
+<details>
+<summary>Answer</summary>
+
+Leave `total` out of the vocabulary entirely. `strict` does the rest:
+
+```js
+import { UNTRUSTED_LIMITS, defineVocabulary, validate } from "@osqd/jql";
+
+const PARTNER = defineVocabulary()({
+  fields: { status: { kind: "exact" }, placed: { kind: "date" } },
+  strict: true,
+});
+
+const asked = validate({ total: { $gte: 100 } }, { vocabulary: PARTNER, limits: UNTRUSTED_LIMITS });
+
+console.log(asked.valid ? "accepted" : asked.error.message);
+```
+
+```
+at total: "total" is not a field here
+```
+
+A field that is not in the vocabulary cannot be filtered on — *and* cannot be sorted by or
+projected with, because `fields`, `omit` and `sort` resolve their names through the same
+vocabulary. One list, one contract.
+</details>
+
+## Related
+
+- [Queries from outside](../guides/untrusted-input.md) — the same ground as a reference page
+- [Security](../../SECURITY.md) — what this does and does not promise
+- [Specification §10](../reference/specification.md#10-limits) — what the standard says about caps