@osqd/jql 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (115) hide show
  1. package/CHANGELOG.md +220 -0
  2. package/LICENSE +102 -0
  3. package/README.md +84 -0
  4. package/bin/jql.mjs +5 -0
  5. package/conformance/cases.json +290 -0
  6. package/dist/array.d.ts +11 -0
  7. package/dist/async.d.ts +44 -0
  8. package/dist/canonical.d.ts +18 -0
  9. package/dist/cjs/array.d.ts +11 -0
  10. package/dist/cjs/async.d.ts +44 -0
  11. package/dist/cjs/canonical.d.ts +18 -0
  12. package/dist/cjs/cli.d.ts +15 -0
  13. package/dist/cjs/collections.d.ts +34 -0
  14. package/dist/cjs/core.d.ts +117 -0
  15. package/dist/cjs/errors.d.ts +17 -0
  16. package/dist/cjs/explain.d.ts +30 -0
  17. package/dist/cjs/global.d.ts +90 -0
  18. package/dist/cjs/group.d.ts +47 -0
  19. package/dist/cjs/index.d.ts +25 -0
  20. package/dist/cjs/internal/closest.d.ts +9 -0
  21. package/dist/cjs/internal/duration.d.ts +15 -0
  22. package/dist/cjs/internal/equal.d.ts +34 -0
  23. package/dist/cjs/internal/glob.d.ts +30 -0
  24. package/dist/cjs/internal/order.d.ts +24 -0
  25. package/dist/cjs/internal/path.d.ts +109 -0
  26. package/dist/cjs/internal/record.d.ts +18 -0
  27. package/dist/cjs/internal/values.d.ts +41 -0
  28. package/dist/cjs/limits.d.ts +64 -0
  29. package/dist/cjs/operators.d.ts +81 -0
  30. package/dist/cjs/package.json +3 -0
  31. package/dist/cjs/plan.d.ts +79 -0
  32. package/dist/cjs/search.d.ts +42 -0
  33. package/dist/cjs/targets/mongo.d.ts +55 -0
  34. package/dist/cjs/text/index.d.ts +12 -0
  35. package/dist/cjs/text/parse.d.ts +91 -0
  36. package/dist/cjs/text/suggest.d.ts +16 -0
  37. package/dist/cjs/text/write.d.ts +34 -0
  38. package/dist/cjs/types.d.ts +236 -0
  39. package/dist/cjs/vocabulary.d.ts +106 -0
  40. package/dist/cli.d.ts +15 -0
  41. package/dist/cli.js +2729 -0
  42. package/dist/cli.js.map +1 -0
  43. package/dist/collections.d.ts +34 -0
  44. package/dist/core.d.ts +117 -0
  45. package/dist/errors.d.ts +17 -0
  46. package/dist/explain.d.ts +30 -0
  47. package/dist/global.cjs +1953 -0
  48. package/dist/global.cjs.map +1 -0
  49. package/dist/global.d.ts +90 -0
  50. package/dist/global.js +1950 -0
  51. package/dist/global.js.map +1 -0
  52. package/dist/group.d.ts +47 -0
  53. package/dist/index.cjs +2529 -0
  54. package/dist/index.cjs.map +1 -0
  55. package/dist/index.d.ts +25 -0
  56. package/dist/index.js +2495 -0
  57. package/dist/index.js.map +1 -0
  58. package/dist/internal/closest.d.ts +9 -0
  59. package/dist/internal/duration.d.ts +15 -0
  60. package/dist/internal/equal.d.ts +34 -0
  61. package/dist/internal/glob.d.ts +30 -0
  62. package/dist/internal/order.d.ts +24 -0
  63. package/dist/internal/path.d.ts +109 -0
  64. package/dist/internal/record.d.ts +18 -0
  65. package/dist/internal/values.d.ts +41 -0
  66. package/dist/limits.d.ts +64 -0
  67. package/dist/mongo.cjs +357 -0
  68. package/dist/mongo.cjs.map +1 -0
  69. package/dist/mongo.js +354 -0
  70. package/dist/mongo.js.map +1 -0
  71. package/dist/operators.d.ts +81 -0
  72. package/dist/plan.d.ts +79 -0
  73. package/dist/search.d.ts +42 -0
  74. package/dist/targets/mongo.d.ts +55 -0
  75. package/dist/text/index.d.ts +12 -0
  76. package/dist/text/parse.d.ts +91 -0
  77. package/dist/text/suggest.d.ts +16 -0
  78. package/dist/text/write.d.ts +34 -0
  79. package/dist/text.cjs +674 -0
  80. package/dist/text.cjs.map +1 -0
  81. package/dist/text.js +667 -0
  82. package/dist/text.js.map +1 -0
  83. package/dist/types.d.ts +236 -0
  84. package/dist/vocabulary.d.ts +106 -0
  85. package/docs/course/01-first-query.md +217 -0
  86. package/docs/course/02-operators.md +285 -0
  87. package/docs/course/03-arrays-and-paths.md +239 -0
  88. package/docs/course/04-combining.md +221 -0
  89. package/docs/course/05-dates.md +214 -0
  90. package/docs/course/06-typed-queries.md +240 -0
  91. package/docs/course/07-requests.md +261 -0
  92. package/docs/course/08-grouping.md +210 -0
  93. package/docs/course/09-explaining.md +171 -0
  94. package/docs/course/10-vocabulary.md +276 -0
  95. package/docs/course/11-the-search-box.md +349 -0
  96. package/docs/course/12-untrusted.md +257 -0
  97. package/docs/course/13-saved-filters.md +199 -0
  98. package/docs/course/14-streams-and-cli.md +276 -0
  99. package/docs/course/15-pushdown.md +240 -0
  100. package/docs/course/16-extending.md +199 -0
  101. package/docs/course/index.md +185 -0
  102. package/docs/design/decisions.md +198 -0
  103. package/docs/design/performance.md +102 -0
  104. package/docs/guides/adopting.md +81 -0
  105. package/docs/guides/pushdown.md +147 -0
  106. package/docs/guides/typescript.md +115 -0
  107. package/docs/guides/untrusted-input.md +86 -0
  108. package/docs/index.md +102 -0
  109. package/docs/reference/api.md +266 -0
  110. package/docs/reference/cli.md +103 -0
  111. package/docs/reference/index.md +12 -0
  112. package/docs/reference/specification.md +549 -0
  113. package/docs/reference/text-syntax.md +152 -0
  114. package/docs/start/quick-start.md +84 -0
  115. package/package.json +136 -0
@@ -0,0 +1,240 @@
1
+ # Lesson 15 — Pushing a query into a store
2
+
3
+ **Goal:** let a database answer the part of a query it can, and answer the rest here — without
4
+ either half quietly changing the question.
5
+
6
+ ← [Course](index.md) · Previous: [Logs and streams](14-streams-and-cli.md) · Next: [Extending it, and proving it](16-extending.md)
7
+
8
+ ---
9
+
10
+ ## Half a query is still useful
11
+
12
+ Every query so far has run over an array in memory. Harbour's orders really live in a
13
+ database, and that database can answer *some* of what a query asks — the indexed fields, the
14
+ operators it happens to have — and none of the rest.
15
+
16
+ `plan(query, capabilities)` splits a query in two:
17
+
18
+ ```
19
+ pushed ∧ remaining ≡ query
20
+ ```
21
+
22
+ The store filters with `pushed`; this engine filters what comes back with `remaining`. That
23
+ identity is the contract, and it has a direction: **neither half is ever wider than the
24
+ query**, so a caller that runs only `pushed` gets too many rows rather than too few.
25
+
26
+ It splits only on **conjunctions**, because those are the only parts that can go to either
27
+ side independently. A branch of an `$or` cannot be split without changing what is asked, so a
28
+ conjunct goes whole or not at all.
29
+
30
+ ## Describing a store
31
+
32
+ Save this as `harbour/store.mjs`. It is what a modest store can do: a few indexed columns,
33
+ equality and ranges, nothing clever.
34
+
35
+ ```js
36
+ /** What Harbour's database can answer for itself. */
37
+ export const INDEXED = {
38
+ fields: ["status", "customer.country", "placed", "total"],
39
+ operators: ["$eq", "$in", "$gt", "$gte", "$lt", "$lte", "$and"],
40
+ or: false,
41
+ not: false,
42
+ };
43
+
44
+ /** Print a plan the way a person reads one. */
45
+ export function show(result) {
46
+ console.log(" pushed ", JSON.stringify(result.pushed));
47
+ console.log(" remaining", JSON.stringify(result.remaining));
48
+ console.log(" complete ", result.complete);
49
+ for (const kept of result.kept) console.log(` kept ${kept.at}: ${kept.why}`);
50
+ }
51
+ ```
52
+
53
+ | | |
54
+ | --- | --- |
55
+ | `fields` | the paths it can filter on, as **it** names them, or `"all"` |
56
+ | `operators` | the operators it can answer, in JQL's names, or `"all"` |
57
+ | `or` | whether it can answer an `$or`. Default `false` — plenty of key-value stores cannot |
58
+ | `not` | whether it can answer a negation. Default `false` |
59
+ | `accepts` | the last word on one field's condition, for a store whose limits depend on the *values* |
60
+
61
+ ## Splitting a query
62
+
63
+ ```js
64
+ import { plan } from "@osqd/jql";
65
+ import { INDEXED, show } from "./store.mjs";
66
+
67
+ const query = {
68
+ status: "paid",
69
+ "customer.country": "GB",
70
+ note: { $contains: "neighbour" },
71
+ "lines.quantity": { $gte: 3 },
72
+ };
73
+
74
+ show(plan(query, INDEXED));
75
+ ```
76
+
77
+ ```
78
+ pushed {"$and":[{"status":"paid"},{"customer.country":"GB"}]}
79
+ remaining {"$and":[{"note":{"$contains":"neighbour"}},{"lines.quantity":{"$gte":3}}]}
80
+ complete false
81
+ kept note: the store cannot filter on "note"
82
+ kept lines.quantity: the store cannot filter on "lines.quantity"
83
+ ```
84
+
85
+ Two conjuncts went to the store, two stayed here, and `kept` says why each one stayed.
86
+
87
+ That sentence — `the store cannot filter on "note"` — is something you can act on. Either
88
+ index `note` and add it to `fields`, or decide you do not care. A pushdown layer that silently
89
+ held things back would leave you guessing why a query is slow.
90
+
91
+ ## The halves add up
92
+
93
+ ```js
94
+ import { filter, plan } from "@osqd/jql";
95
+ import { orders } from "./orders.mjs";
96
+ import { INDEXED } from "./store.mjs";
97
+
98
+ const query = {
99
+ status: "paid",
100
+ "customer.country": "GB",
101
+ note: { $contains: "neighbour" },
102
+ "lines.quantity": { $gte: 3 },
103
+ };
104
+
105
+ const { pushed, remaining } = plan(query, INDEXED);
106
+
107
+ // `orders` stands in for the database here; in real life this is a round trip.
108
+ const fromStore = filter(orders, pushed ?? {});
109
+ const answer = filter(fromStore, remaining ?? {});
110
+
111
+ console.log("the store returned:", fromStore.map((o) => o.id));
112
+ console.log("the engine kept :", answer.map((o) => o.id));
113
+ console.log("whole query, here :", filter(orders, query).map((o) => o.id));
114
+ ```
115
+
116
+ ```
117
+ the store returned: [ 'o-1001', 'o-1003' ]
118
+ the engine kept : [ 'o-1001' ]
119
+ whole query, here : [ 'o-1001' ]
120
+ ```
121
+
122
+ The store narrowed eight orders to two, using only indexed columns; the engine took one of
123
+ those away. The answer is the one the whole query gives.
124
+
125
+ ## What cannot be split at all
126
+
127
+ ```js
128
+ import { plan } from "@osqd/jql";
129
+ import { INDEXED, show } from "./store.mjs";
130
+
131
+ show(plan({ $or: [{ status: "open" }, { status: "paid" }] }, INDEXED));
132
+ ```
133
+
134
+ ```
135
+ pushed undefined
136
+ remaining {"$or":[{"status":"open"},{"status":"paid"}]}
137
+ complete false
138
+ kept $or: the store cannot answer an $or
139
+ ```
140
+
141
+ `pushed` is `undefined` — there is nothing to send. Both branches are about an indexed field,
142
+ but this store cannot answer an `$or` at all, and half an `$or` is a different question. So
143
+ the whole thing stays here, and `kept` says so in one line.
144
+
145
+ ## A store that can answer more
146
+
147
+ A store target is a capabilities object plus a translator. One ships with the library, for
148
+ document databases that speak the conventional query-document dialect:
149
+
150
+ ```js
151
+ import { plan } from "@osqd/jql";
152
+ import { MONGO_CAPABILITIES, toMongoFilter } from "@osqd/jql/mongo";
153
+
154
+ const query = {
155
+ status: "paid",
156
+ "customer.country": "GB",
157
+ note: { $contains: "neighbour" },
158
+ "lines.quantity": { $gte: 3 },
159
+ };
160
+
161
+ const planned = plan(query, MONGO_CAPABILITIES);
162
+
163
+ console.log("complete:", planned.complete);
164
+ // A driver is handed real RegExp and Date objects, which JSON.stringify cannot show.
165
+ const readable = (value) => JSON.stringify(value, (_, held) => (held instanceof RegExp ? `/${held.source}/${held.flags}` : held));
166
+ console.log("filter :", readable(toMongoFilter(planned.pushed)));
167
+ ```
168
+
169
+ ```
170
+ complete: true
171
+ filter : {"$and":[{"status":"paid"},{"customer.country":"GB"},{"note":"/neighbour/"},{"lines.quantity":{"$gte":3}}]}
172
+ ```
173
+
174
+ All four conjuncts went, so `complete` is `true` and there is no second pass to run:
175
+
176
+ ```ts
177
+ const { pushed, remaining, complete } = plan(query, MONGO_CAPABILITIES);
178
+ const rows = await collection.find(toMongoFilter(pushed)).toArray();
179
+ const answer = complete ? rows : filter(rows, remaining);
180
+ ```
181
+
182
+ Look at what happened to `$contains`. The store has no such operator, so the target wrote an
183
+ **escaped pattern** that the store runs exactly as JQL would.
184
+
185
+ What a target will not do is guess. Any operator the store cannot answer *exactly* is left out
186
+ of its capabilities, so `plan` keeps those clauses here and the translator never sees them. A
187
+ translation that was *nearly* right would be the worst of both worlds: fewer rows than the
188
+ query asked for, from a store that looked as though it had answered.
189
+
190
+ ## The limit of `complete`
191
+
192
+ Capabilities describe what a store can be *asked*. They cannot describe how it resolves a path
193
+ through your data, and those can differ.
194
+
195
+ The case to know about is an array held directly inside another array. JQL sees an array
196
+ through at every level (lesson 3); a store may apply a path segment to an array's elements but
197
+ not to the elements of *those* arrays. For such a document the two give different answers, and
198
+ `plan` cannot know, because nothing in the query says what shape the rows are.
199
+
200
+ If your collection stores arrays of arrays of documents, do not read `complete` as permission
201
+ to skip the second pass over those fields.
202
+
203
+ ## Exercise
204
+
205
+ Harbour's store gains an index on `tags`. Add it to `INDEXED` and see what changes for
206
+ `{ status: "paid", tags: "repeat", note: { $contains: "same" } }`.
207
+
208
+ <details>
209
+ <summary>Answer</summary>
210
+
211
+ ```js
212
+ import { plan } from "@osqd/jql";
213
+ import { INDEXED, show } from "./store.mjs";
214
+
215
+ const WITH_TAGS = { ...INDEXED, fields: [...INDEXED.fields, "tags"] };
216
+
217
+ show(plan({ status: "paid", tags: "repeat", note: { $contains: "same" } }, WITH_TAGS));
218
+ ```
219
+
220
+ ```
221
+ pushed {"$and":[{"status":"paid"},{"tags":"repeat"}]}
222
+ remaining {"note":{"$contains":"same"}}
223
+ complete false
224
+ kept note: the store cannot filter on "note"
225
+ ```
226
+
227
+ `tags: "repeat"` pushed as an equality even though `tags` is a list, because *a value matches
228
+ any element* is a rule both sides share — and shared rules are exactly what makes a clause
229
+ safe to push.
230
+
231
+ `note` stayed, and the reason is worth reading closely: `the store cannot filter on "note"`.
232
+ Not "cannot answer `$contains`" — `note` is not in `fields` at all, so the field is refused
233
+ before the operator is ever considered. Two different fixes hide behind those two sentences
234
+ (index the column, or teach the store an operator), which is why `kept` says which one it is.
235
+ </details>
236
+
237
+ ## Related
238
+
239
+ - [Pushing a query into a store](../guides/pushdown.md) — the guide, including writing your own target
240
+ - [Specification §14](../reference/specification.md#14-splitting-a-query) — the split, as a rule other implementations can follow
@@ -0,0 +1,199 @@
1
+ # Lesson 16 — Extending it, and proving it
2
+
3
+ **Goal:** add an operator this project needs, check a filter before it meets anybody, and
4
+ finish Harbour.
5
+
6
+ ← [Course](index.md) · Previous: [Pushing into a store](15-pushdown.md)
7
+
8
+ ---
9
+
10
+ ## Operators of your own
11
+
12
+ Some questions are not JSON. *"Is this address inside that network"*, *"does this score beat
13
+ the model's threshold"* — no amount of `$gt` expresses them, and the honest answer is to let a
14
+ project add an operator.
15
+
16
+ The cost of doing that is portability, so the language makes the cost **visible**: an added
17
+ operator's name must begin with `$x` and a capital letter. Somebody reading a stored filter
18
+ can see at a glance that it needs more than a standard engine, and a future version of JQL
19
+ cannot collide with a name you chose.
20
+
21
+ Save this as `harbour/within-days.mjs`:
22
+
23
+ ```js
24
+ import { JqlError } from "@osqd/jql";
25
+
26
+ /** How recently an order was placed — the sort of question a project adds for itself. */
27
+ export const withinDays = {
28
+ name: "$xWithinDays",
29
+ cost: 2,
30
+ compile: (operand, at) => {
31
+ if (typeof operand !== "number" || !Number.isFinite(operand)) {
32
+ throw new JqlError(`takes a number of days, not ${JSON.stringify(operand)}`, at);
33
+ }
34
+ const since = Date.parse("2026-09-25T12:00:00Z") - operand * 86_400_000;
35
+ return (value) => typeof value === "string" && Date.parse(value) >= since;
36
+ },
37
+ };
38
+ ```
39
+
40
+ ## Using it
41
+
42
+ ```js
43
+ import { filter } from "@osqd/jql";
44
+ import { orders } from "./orders.mjs";
45
+ import { withinDays } from "./within-days.mjs";
46
+
47
+ const operators = [withinDays];
48
+ const ids = (query) => filter(orders, query, { operators }).map((o) => o.id);
49
+
50
+ console.log("within 7 days :", ids({ placed: { $xWithinDays: 7 } }));
51
+ console.log("within 20 days:", ids({ placed: { $xWithinDays: 20 } }));
52
+ ```
53
+
54
+ ```
55
+ within 7 days : [ 'o-1006', 'o-1007', 'o-1008' ]
56
+ within 20 days: [ 'o-1003', 'o-1004', 'o-1005', 'o-1006', 'o-1007', 'o-1008' ]
57
+ ```
58
+
59
+ From the query's side it is an ordinary operator: it composes with everything else, it can sit
60
+ inside `$or`, and `explain` describes it like any other clause.
61
+
62
+ ## Three refusals that make it safe to have
63
+
64
+ ```js
65
+ import { filter, validate } from "@osqd/jql";
66
+ import { orders } from "./orders.mjs";
67
+ import { withinDays } from "./within-days.mjs";
68
+
69
+ const say = (verdict) => console.log(" ", verdict.valid ? "accepted" : verdict.error.message);
70
+
71
+ console.log("an operand that is not a number:");
72
+ say(validate({ placed: { $xWithinDays: "seven" } }, { operators: [withinDays] }));
73
+
74
+ console.log("an engine that was not given it:");
75
+ say(validate({ placed: { $xWithinDays: 7 } }));
76
+
77
+ console.log("a name without the $x:");
78
+ try {
79
+ filter(orders, { placed: { $withinDays: 7 } }, { operators: [{ ...withinDays, name: "$withinDays" }] });
80
+ console.log(" accepted");
81
+ } catch (error) {
82
+ console.log(" ", error.message);
83
+ }
84
+ ```
85
+
86
+ ```
87
+ an operand that is not a number:
88
+ at placed.$xWithinDays: takes a number of days, not "seven"
89
+ an engine that was not given it:
90
+ at placed.$xWithinDays: "$xWithinDays" is an added operator, and this query was compiled without it; pass it in `operators`
91
+ a name without the $x:
92
+ at operators: "$withinDays" cannot name an added operator: the name is $x and a capital, such as "$xCidr", so a query that needs more than standard JQL says so
93
+ ```
94
+
95
+ Each of those is a property worth having:
96
+
97
+ - **`compile` runs once**, when the query is compiled, and returns the test. Validate the
98
+ operand there and throw `JqlError` — the sentence reaches whoever wrote the query, with the
99
+ place in it.
100
+ - **A query that needs an added operator says so.** The second refusal is an engine *without*
101
+ the operator, naming it and saying what to do, rather than treating it as a typo and
102
+ matching nothing.
103
+ - **The naming rule is enforced**, so the visibility it buys is real.
104
+
105
+ There is a fourth property with no error message: **`cost`** is a hint for ordering an `$and`,
106
+ so a cheap equality runs before your expensive test. It changes the speed and never the
107
+ answer.
108
+
109
+ Before reaching for any of this, look at the field operators once more. `$word`, `$glob`,
110
+ `$length` and `{ "$field": … }` exist precisely because they are the four things people most
111
+ often added an escape hatch to do — and a query using them is still portable JQL.
112
+
113
+ ## Proving a filter before it meets anybody
114
+
115
+ The language ships its own test suite as **data**: `conformance/cases.json`, 156 matching
116
+ cases and 27 request cases. Each is a document set, a query, and the answer any implementation
117
+ must give.
118
+
119
+ ```json
120
+ {"group": "equality", "name": "an empty query matches everything", "documents": "people", "query": {}, "matches": [0, 1, 2, 3]}
121
+ ```
122
+
123
+ You need it mostly if you implement JQL in another language. But it is also the clearest
124
+ statement of what an operator means, in twelve groups — equality, arrays and paths, existence
125
+ and type, ordering, strings, logic, text, references, length, glob, relative dates, refusals.
126
+ When you are unsure what something does, the case is faster to read than the prose.
127
+
128
+ For your *own* filters, the tools are ones you already have:
129
+
130
+ | | |
131
+ | --- | --- |
132
+ | `validate` | a query from outside, refused with a sentence (lesson 12) |
133
+ | `explain` | why one row did or did not match (lesson 9) |
134
+ | `jql --explain` | the same from a shell, over the first line of a real file (lesson 14) |
135
+ | `fingerprint` | whether the filter in the test is the filter in production (lesson 13) |
136
+ | `toText(…).complete` | whether the box is showing the whole filter (lesson 11) |
137
+
138
+ ## Harbour, finished
139
+
140
+ Everything the course built, in one place:
141
+
142
+ ```ts
143
+ // vocabulary.mjs — the names, once, for both front ends (lesson 10)
144
+ export const ORDERS = defineVocabulary()({ fields: { … }, text: [ … ] });
145
+
146
+ // the console's search box (lesson 11)
147
+ const query = parseText(typed, { vocabulary: ORDERS });
148
+
149
+ // the table (lessons 7, 15)
150
+ const { pushed, remaining, complete } = plan(query, STORE_CAPABILITIES);
151
+ const rows = await collection.find(toStoreFilter(pushed)).toArray();
152
+ const page = search(complete ? rows : filter(rows, remaining), {
153
+ sort: { placed: -1 }, skip, limit, omit: ["customer.email"],
154
+ }, { vocabulary: ORDERS });
155
+
156
+ // the number above it (lesson 8)
157
+ const byStatus = group(rows, "status", { vocabulary: ORDERS, where: query });
158
+
159
+ // "why isn't this here?" (lesson 9)
160
+ const why = explain(query, row, { vocabulary: ORDERS });
161
+
162
+ // saving it (lesson 13)
163
+ await filters.insert({ key: fingerprint(query), name, query: canonical(query) });
164
+
165
+ // and the public API, which trusts none of the above (lesson 12)
166
+ const checked = validate(JSON.parse(body), { vocabulary: PUBLIC, limits });
167
+ if (!checked.valid) return respond(400, { error: checked.error.message });
168
+ ```
169
+
170
+ One query document runs in the console, in the API, in the store and in the nightly export,
171
+ and means the same thing in all four. That is the whole argument for a query being data.
172
+
173
+ ## Where to go next
174
+
175
+ | | |
176
+ | --- | --- |
177
+ | [The specification](../reference/specification.md) | what every operator means, precisely — the thing to read when you disagree with an answer |
178
+ | [Adopting JQL in a project](../guides/adopting.md) | moving an existing project's filtering onto it |
179
+ | [Pushing a query into a store](../guides/pushdown.md) | writing a target for a store of your own |
180
+ | [Design decisions](../design/decisions.md) | what this library refuses to do, and what that costs |
181
+ | [Performance](../design/performance.md) | how fast it is, how that is measured, and what makes it so |
182
+
183
+ ## A last exercise
184
+
185
+ Take the filter your own project uses most — the one behind a dashboard or an endpoint — and
186
+ write it as a JQL document. Two things usually come out of that:
187
+
188
+ 1. a clause you cannot express, which is worth knowing about; and
189
+ 2. a clause you *can* express that you had been doing in application code, which is worth
190
+ moving.
191
+
192
+ If it is the first, the [specification](../reference/specification.md) will tell you whether
193
+ it is a gap in the language or a gap in this course. If it is the second, you have just made a
194
+ filter storable, sendable and explainable — which is what the whole thing is for.
195
+
196
+ ## Related
197
+
198
+ - [Course index](index.md) — all sixteen lessons
199
+ - [Library API](../reference/api.md) — every export, grouped by what it is for
@@ -0,0 +1,185 @@
1
+ # The JQL course
2
+
3
+ Sixteen lessons that build one real integration, from a first query to a filter you can put
4
+ behind a public endpoint.
5
+
6
+ ← [Documentation](../index.md)
7
+
8
+ ---
9
+
10
+ ## What this is
11
+
12
+ The [reference documentation](../index.md) answers *"how does X work?"*. This answers *"what
13
+ do I do, and in what order?"* — every capability of the library, taught in the order that
14
+ makes each one make sense, with something to run at every step.
15
+
16
+ It is written to be worked through rather than read. Every lesson is a handful of short,
17
+ self-contained programs, each followed by what it prints — and **every one of those outputs
18
+ was produced by running the code**, not by imagining what it would print. A script in this
19
+ repository runs all of them on the packed package before a release, so a page that drifts
20
+ from the library fails the build.
21
+
22
+ **Time:** about three hours to do properly. Lessons 1–5 are the language itself and are worth
23
+ slowing down for; everything after them assumes you have those.
24
+
25
+ ## Who it is for
26
+
27
+ A TypeScript or JavaScript developer with a collection of things and a filter to write — rows
28
+ in a dashboard, records behind an API, lines in a log. You need no background in query
29
+ languages: the operator names are the conventional ones, so if you have written a filter in
30
+ almost any document store you will recognise most of them, and
31
+ [the specification](../reference/specification.md) says precisely what each one means here.
32
+
33
+ ## The running example
34
+
35
+ You are building **Harbour**, the order desk of a small shop. It has an order book worth
36
+ searching, a support console worth giving a search box, and an API worth not letting strangers
37
+ run arbitrary queries against. Every lesson adds one thing to Harbour, and by lesson 16 you
38
+ have a complete, tested, production-shaped setup.
39
+
40
+ ## Set up once
41
+
42
+ ```bash
43
+ mkdir harbour && cd harbour
44
+ npm init -y && npm pkg set type=module
45
+ npm install @osqd/jql
46
+ ```
47
+
48
+ Then save the order book as `harbour/orders.mjs`. Every lesson imports it, and it is small
49
+ enough that you can check any answer by hand:
50
+
51
+ ```js
52
+ /** The Harbour order book: eight orders, small enough to check an answer by hand. */
53
+ export const orders = [
54
+ {
55
+ id: "o-1001", placed: "2026-09-01T09:15:00Z", status: "paid", channel: "web",
56
+ customer: { name: "Ada Lovelace", country: "GB", email: "ada@example.com" },
57
+ lines: [{ sku: "pen-fine", title: "Fine liner", quantity: 3, price: 4.5 }, { sku: "ink-blue", title: "Blue ink", quantity: 1, price: 9 }],
58
+ total: 22.5, paid: 22.5, tags: ["gift"], note: "leave with the neighbour",
59
+ },
60
+ {
61
+ id: "o-1002", placed: "2026-09-03T14:02:00Z", status: "open", channel: "web",
62
+ customer: { name: "Grace Hopper", country: "US", email: "grace@example.com" },
63
+ lines: [{ sku: "pad-a5", title: "A5 pad", quantity: 10, price: 3 }],
64
+ total: 30, paid: 0, tags: [],
65
+ },
66
+ {
67
+ id: "o-1003", placed: "2026-09-07T08:40:00Z", status: "paid", channel: "phone",
68
+ customer: { name: "Alan Turing", country: "GB", email: "alan@example.com" },
69
+ lines: [{ sku: "pen-fine", title: "Fine liner", quantity: 1, price: 4.5 }],
70
+ total: 4.5, paid: 4.5, tags: ["repeat"], note: "same as last time",
71
+ },
72
+ {
73
+ id: "o-1004", placed: "2026-09-11T19:30:00Z", status: "refunded", channel: "web",
74
+ customer: { name: "Katherine Johnson", country: "US", email: "kj@example.com" },
75
+ lines: [{ sku: "ink-blue", title: "Blue ink", quantity: 2, price: 9 }, { sku: "ink-red", title: "Red ink", quantity: 2, price: 9 }],
76
+ total: 36, paid: 0, tags: ["damaged", "repeat"], note: "arrived leaking",
77
+ },
78
+ {
79
+ id: "o-1005", placed: "2026-09-14T11:05:00Z", status: "open", channel: "app",
80
+ customer: { name: "Tim Berners-Lee", country: "GB", email: "tim@example.com" },
81
+ lines: [{ sku: "desk-lamp", title: "Desk lamp", quantity: 1, price: 140 }, { sku: "pad-a5", title: "A5 pad", quantity: 10, price: 3 }],
82
+ total: 170, paid: 0, tags: ["fragile"],
83
+ },
84
+ {
85
+ id: "o-1006", placed: "2026-09-18T16:45:00Z", status: "paid", channel: "app",
86
+ customer: { name: "Barbara Liskov", country: "US", email: "barbara@example.com" },
87
+ lines: [{ sku: "pad-a5", title: "A5 pad", quantity: 2, price: 3 }, { sku: "pen-fine", title: "Fine liner", quantity: 6, price: 4.5 }],
88
+ total: 33, paid: 33, tags: [], note: "gift wrap, no receipt",
89
+ },
90
+ {
91
+ id: "o-1007", placed: "2026-09-21T07:20:00Z", status: "cancelled", channel: "web",
92
+ customer: { name: "Margaret Hamilton", country: "DE", email: "margaret@example.com" },
93
+ lines: [{ sku: "desk-lamp", title: "Desk lamp", quantity: 2, price: 140 }],
94
+ total: 280, paid: 0, tags: ["fragile"], note: null,
95
+ },
96
+ {
97
+ id: "o-1008", placed: "2026-09-24T13:10:00Z", status: "paid", channel: "phone",
98
+ customer: { name: "Ada Byron", country: "FR", email: "ada.b@example.com" },
99
+ lines: [{ sku: "ink-red", title: "Red ink", quantity: 1, price: 9 }, { sku: "pad-a5", title: "A5 pad", quantity: 1, price: 3 }],
100
+ total: 12, paid: 12, tags: ["repeat"],
101
+ },
102
+ ];
103
+ ```
104
+
105
+ Look at it for a moment before you start. Three orders have no `note` at all, one has a `note`
106
+ of `null`, two have no tags, and one has two lines that are nothing like each other. Every one
107
+ of those is there to catch a lesson out.
108
+
109
+ Each example is a whole program: put it in a file next to `orders.mjs` and run it with
110
+ `node`. A few lessons introduce a second module — a vocabulary, a store description — and say
111
+ so where they do. Lesson 6 is the one that wants TypeScript; nothing here needs a server or a
112
+ database at all.
113
+
114
+ ---
115
+
116
+ ## Part 1 — The language
117
+
118
+ | | | |
119
+ |-|-|-|
120
+ | 1 | [Your first query](01-first-query.md) | A query is data. Four helpers, and a refusal. |
121
+ | 2 | [Asking precisely](02-operators.md) | Comparison, membership, existence, strings, patterns. |
122
+ | 3 | [Arrays and paths](03-arrays-and-paths.md) | How a path meets an array. **The most important lesson here.** |
123
+ | 4 | [Combining and negating](04-combining.md) | `$or`, `$nor`, `$not` — and what "not" means over a list. |
124
+ | 5 | [Dates and windows](05-dates.md) | An instant in JSON, and a window that stays true tomorrow. |
125
+
126
+ ## Part 2 — Letting the compiler help
127
+
128
+ | | | |
129
+ |-|-|-|
130
+ | 6 | [Typed queries](06-typed-queries.md) | A typo in a field name as a compile error rather than an empty result. |
131
+
132
+ ## Part 3 — Whole questions about a collection
133
+
134
+ | | | |
135
+ |-|-|-|
136
+ | 7 | [Requests](07-requests.md) | Order, pages, and which fields you hand back. |
137
+ | 8 | [Counting by a field](08-grouping.md) | The number above the table. |
138
+ | 9 | [Why did this not match?](09-explaining.md) | The clause that decided it, with the values it read. |
139
+
140
+ ## Part 4 — People typing
141
+
142
+ | | | |
143
+ |-|-|-|
144
+ | 10 | [A vocabulary](10-vocabulary.md) | The names your data has, in one place, for both front ends. |
145
+ | 11 | [The search box](11-the-search-box.md) | Text into JQL, JQL back into text, and completions. |
146
+
147
+ ## Part 5 — Production
148
+
149
+ | | | |
150
+ |-|-|-|
151
+ | 12 | [Queries from outside](12-untrusted.md) | Accepting one from a URL without accepting everything. |
152
+ | 13 | [Saved filters](13-saved-filters.md) | One shape and one short name per question. |
153
+ | 14 | [Logs and streams](14-streams-and-cli.md) | The async helpers, and the `jql` command. |
154
+ | 15 | [Pushing into a store](15-pushdown.md) | Letting a backend answer the part it can. |
155
+ | 16 | [Extending it, and proving it](16-extending.md) | Operators of your own, the conformance suite, and a finished Harbour. |
156
+
157
+ ---
158
+
159
+ ## How to get the most out of it
160
+
161
+ **Type the code, do not paste it.** The examples are short on purpose.
162
+
163
+ **Predict the answer before you run it.** Eight orders is few enough to work out in your head,
164
+ and the times you are wrong are the times you learn where this language differs from the one
165
+ in your head.
166
+
167
+ **When something surprises you, chase it.** Every lesson ends with links into the reference
168
+ documentation and the specification for the thing you just used.
169
+
170
+ ## One sentence to keep
171
+
172
+ > **A query that is not understood is refused, never run as something that looks like it
173
+ > worked.**
174
+
175
+ You will meet it in lesson 1, thirty seconds in, when a misspelt operator throws instead of
176
+ quietly matching nothing. Every other refusal in this course is that same rule: the failure
177
+ this language exists to prevent is a filter that looks applied and is not.
178
+
179
+ Start with [lesson 1](01-first-query.md).
180
+
181
+ ## Related
182
+
183
+ - [Quick start](../start/quick-start.md) — if you want five minutes rather than three hours
184
+ - [The specification](../reference/specification.md) — what every operator means, precisely
185
+ - [Library API](../reference/api.md) — every export, grouped by what it is for