@osqd/jql 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +220 -0
- package/LICENSE +102 -0
- package/README.md +84 -0
- package/bin/jql.mjs +5 -0
- package/conformance/cases.json +290 -0
- package/dist/array.d.ts +11 -0
- package/dist/async.d.ts +44 -0
- package/dist/canonical.d.ts +18 -0
- package/dist/cjs/array.d.ts +11 -0
- package/dist/cjs/async.d.ts +44 -0
- package/dist/cjs/canonical.d.ts +18 -0
- package/dist/cjs/cli.d.ts +15 -0
- package/dist/cjs/collections.d.ts +34 -0
- package/dist/cjs/core.d.ts +117 -0
- package/dist/cjs/errors.d.ts +17 -0
- package/dist/cjs/explain.d.ts +30 -0
- package/dist/cjs/global.d.ts +90 -0
- package/dist/cjs/group.d.ts +47 -0
- package/dist/cjs/index.d.ts +25 -0
- package/dist/cjs/internal/closest.d.ts +9 -0
- package/dist/cjs/internal/duration.d.ts +15 -0
- package/dist/cjs/internal/equal.d.ts +34 -0
- package/dist/cjs/internal/glob.d.ts +30 -0
- package/dist/cjs/internal/order.d.ts +24 -0
- package/dist/cjs/internal/path.d.ts +109 -0
- package/dist/cjs/internal/record.d.ts +18 -0
- package/dist/cjs/internal/values.d.ts +41 -0
- package/dist/cjs/limits.d.ts +64 -0
- package/dist/cjs/operators.d.ts +81 -0
- package/dist/cjs/package.json +3 -0
- package/dist/cjs/plan.d.ts +79 -0
- package/dist/cjs/search.d.ts +42 -0
- package/dist/cjs/targets/mongo.d.ts +55 -0
- package/dist/cjs/text/index.d.ts +12 -0
- package/dist/cjs/text/parse.d.ts +91 -0
- package/dist/cjs/text/suggest.d.ts +16 -0
- package/dist/cjs/text/write.d.ts +34 -0
- package/dist/cjs/types.d.ts +236 -0
- package/dist/cjs/vocabulary.d.ts +106 -0
- package/dist/cli.d.ts +15 -0
- package/dist/cli.js +2729 -0
- package/dist/cli.js.map +1 -0
- package/dist/collections.d.ts +34 -0
- package/dist/core.d.ts +117 -0
- package/dist/errors.d.ts +17 -0
- package/dist/explain.d.ts +30 -0
- package/dist/global.cjs +1953 -0
- package/dist/global.cjs.map +1 -0
- package/dist/global.d.ts +90 -0
- package/dist/global.js +1950 -0
- package/dist/global.js.map +1 -0
- package/dist/group.d.ts +47 -0
- package/dist/index.cjs +2529 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.ts +25 -0
- package/dist/index.js +2495 -0
- package/dist/index.js.map +1 -0
- package/dist/internal/closest.d.ts +9 -0
- package/dist/internal/duration.d.ts +15 -0
- package/dist/internal/equal.d.ts +34 -0
- package/dist/internal/glob.d.ts +30 -0
- package/dist/internal/order.d.ts +24 -0
- package/dist/internal/path.d.ts +109 -0
- package/dist/internal/record.d.ts +18 -0
- package/dist/internal/values.d.ts +41 -0
- package/dist/limits.d.ts +64 -0
- package/dist/mongo.cjs +357 -0
- package/dist/mongo.cjs.map +1 -0
- package/dist/mongo.js +354 -0
- package/dist/mongo.js.map +1 -0
- package/dist/operators.d.ts +81 -0
- package/dist/plan.d.ts +79 -0
- package/dist/search.d.ts +42 -0
- package/dist/targets/mongo.d.ts +55 -0
- package/dist/text/index.d.ts +12 -0
- package/dist/text/parse.d.ts +91 -0
- package/dist/text/suggest.d.ts +16 -0
- package/dist/text/write.d.ts +34 -0
- package/dist/text.cjs +674 -0
- package/dist/text.cjs.map +1 -0
- package/dist/text.js +667 -0
- package/dist/text.js.map +1 -0
- package/dist/types.d.ts +236 -0
- package/dist/vocabulary.d.ts +106 -0
- package/docs/course/01-first-query.md +217 -0
- package/docs/course/02-operators.md +285 -0
- package/docs/course/03-arrays-and-paths.md +239 -0
- package/docs/course/04-combining.md +221 -0
- package/docs/course/05-dates.md +214 -0
- package/docs/course/06-typed-queries.md +240 -0
- package/docs/course/07-requests.md +261 -0
- package/docs/course/08-grouping.md +210 -0
- package/docs/course/09-explaining.md +171 -0
- package/docs/course/10-vocabulary.md +276 -0
- package/docs/course/11-the-search-box.md +349 -0
- package/docs/course/12-untrusted.md +257 -0
- package/docs/course/13-saved-filters.md +199 -0
- package/docs/course/14-streams-and-cli.md +276 -0
- package/docs/course/15-pushdown.md +240 -0
- package/docs/course/16-extending.md +199 -0
- package/docs/course/index.md +185 -0
- package/docs/design/decisions.md +198 -0
- package/docs/design/performance.md +102 -0
- package/docs/guides/adopting.md +81 -0
- package/docs/guides/pushdown.md +147 -0
- package/docs/guides/typescript.md +115 -0
- package/docs/guides/untrusted-input.md +86 -0
- package/docs/index.md +102 -0
- package/docs/reference/api.md +266 -0
- package/docs/reference/cli.md +103 -0
- package/docs/reference/index.md +12 -0
- package/docs/reference/specification.md +549 -0
- package/docs/reference/text-syntax.md +152 -0
- package/docs/start/quick-start.md +84 -0
- package/package.json +136 -0
|
@@ -0,0 +1,240 @@
|
|
|
1
|
+
# Lesson 15 — Pushing a query into a store
|
|
2
|
+
|
|
3
|
+
**Goal:** let a database answer the part of a query it can, and answer the rest here — without
|
|
4
|
+
either half quietly changing the question.
|
|
5
|
+
|
|
6
|
+
← [Course](index.md) · Previous: [Logs and streams](14-streams-and-cli.md) · Next: [Extending it, and proving it](16-extending.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Half a query is still useful
|
|
11
|
+
|
|
12
|
+
Every query so far has run over an array in memory. Harbour's orders really live in a
|
|
13
|
+
database, and that database can answer *some* of what a query asks — the indexed fields, the
|
|
14
|
+
operators it happens to have — and none of the rest.
|
|
15
|
+
|
|
16
|
+
`plan(query, capabilities)` splits a query in two:
|
|
17
|
+
|
|
18
|
+
```
|
|
19
|
+
pushed ∧ remaining ≡ query
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
The store filters with `pushed`; this engine filters what comes back with `remaining`. That
|
|
23
|
+
identity is the contract, and it has a direction: **neither half is ever wider than the
|
|
24
|
+
query**, so a caller that runs only `pushed` gets too many rows rather than too few.
|
|
25
|
+
|
|
26
|
+
It splits only on **conjunctions**, because those are the only parts that can go to either
|
|
27
|
+
side independently. A branch of an `$or` cannot be split without changing what is asked, so a
|
|
28
|
+
conjunct goes whole or not at all.
|
|
29
|
+
|
|
30
|
+
## Describing a store
|
|
31
|
+
|
|
32
|
+
Save this as `harbour/store.mjs`. It is what a modest store can do: a few indexed columns,
|
|
33
|
+
equality and ranges, nothing clever.
|
|
34
|
+
|
|
35
|
+
```js
|
|
36
|
+
/** What Harbour's database can answer for itself. */
|
|
37
|
+
export const INDEXED = {
|
|
38
|
+
fields: ["status", "customer.country", "placed", "total"],
|
|
39
|
+
operators: ["$eq", "$in", "$gt", "$gte", "$lt", "$lte", "$and"],
|
|
40
|
+
or: false,
|
|
41
|
+
not: false,
|
|
42
|
+
};
|
|
43
|
+
|
|
44
|
+
/** Print a plan the way a person reads one. */
|
|
45
|
+
export function show(result) {
|
|
46
|
+
console.log(" pushed ", JSON.stringify(result.pushed));
|
|
47
|
+
console.log(" remaining", JSON.stringify(result.remaining));
|
|
48
|
+
console.log(" complete ", result.complete);
|
|
49
|
+
for (const kept of result.kept) console.log(` kept ${kept.at}: ${kept.why}`);
|
|
50
|
+
}
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
| | |
|
|
54
|
+
| --- | --- |
|
|
55
|
+
| `fields` | the paths it can filter on, as **it** names them, or `"all"` |
|
|
56
|
+
| `operators` | the operators it can answer, in JQL's names, or `"all"` |
|
|
57
|
+
| `or` | whether it can answer an `$or`. Default `false` — plenty of key-value stores cannot |
|
|
58
|
+
| `not` | whether it can answer a negation. Default `false` |
|
|
59
|
+
| `accepts` | the last word on one field's condition, for a store whose limits depend on the *values* |
|
|
60
|
+
|
|
61
|
+
## Splitting a query
|
|
62
|
+
|
|
63
|
+
```js
|
|
64
|
+
import { plan } from "@osqd/jql";
|
|
65
|
+
import { INDEXED, show } from "./store.mjs";
|
|
66
|
+
|
|
67
|
+
const query = {
|
|
68
|
+
status: "paid",
|
|
69
|
+
"customer.country": "GB",
|
|
70
|
+
note: { $contains: "neighbour" },
|
|
71
|
+
"lines.quantity": { $gte: 3 },
|
|
72
|
+
};
|
|
73
|
+
|
|
74
|
+
show(plan(query, INDEXED));
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
pushed {"$and":[{"status":"paid"},{"customer.country":"GB"}]}
|
|
79
|
+
remaining {"$and":[{"note":{"$contains":"neighbour"}},{"lines.quantity":{"$gte":3}}]}
|
|
80
|
+
complete false
|
|
81
|
+
kept note: the store cannot filter on "note"
|
|
82
|
+
kept lines.quantity: the store cannot filter on "lines.quantity"
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Two conjuncts went to the store, two stayed here, and `kept` says why each one stayed.
|
|
86
|
+
|
|
87
|
+
That sentence — `the store cannot filter on "note"` — is something you can act on. Either
|
|
88
|
+
index `note` and add it to `fields`, or decide you do not care. A pushdown layer that silently
|
|
89
|
+
held things back would leave you guessing why a query is slow.
|
|
90
|
+
|
|
91
|
+
## The halves add up
|
|
92
|
+
|
|
93
|
+
```js
|
|
94
|
+
import { filter, plan } from "@osqd/jql";
|
|
95
|
+
import { orders } from "./orders.mjs";
|
|
96
|
+
import { INDEXED } from "./store.mjs";
|
|
97
|
+
|
|
98
|
+
const query = {
|
|
99
|
+
status: "paid",
|
|
100
|
+
"customer.country": "GB",
|
|
101
|
+
note: { $contains: "neighbour" },
|
|
102
|
+
"lines.quantity": { $gte: 3 },
|
|
103
|
+
};
|
|
104
|
+
|
|
105
|
+
const { pushed, remaining } = plan(query, INDEXED);
|
|
106
|
+
|
|
107
|
+
// `orders` stands in for the database here; in real life this is a round trip.
|
|
108
|
+
const fromStore = filter(orders, pushed ?? {});
|
|
109
|
+
const answer = filter(fromStore, remaining ?? {});
|
|
110
|
+
|
|
111
|
+
console.log("the store returned:", fromStore.map((o) => o.id));
|
|
112
|
+
console.log("the engine kept :", answer.map((o) => o.id));
|
|
113
|
+
console.log("whole query, here :", filter(orders, query).map((o) => o.id));
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
```
|
|
117
|
+
the store returned: [ 'o-1001', 'o-1003' ]
|
|
118
|
+
the engine kept : [ 'o-1001' ]
|
|
119
|
+
whole query, here : [ 'o-1001' ]
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
The store narrowed eight orders to two, using only indexed columns; the engine took one of
|
|
123
|
+
those away. The answer is the one the whole query gives.
|
|
124
|
+
|
|
125
|
+
## What cannot be split at all
|
|
126
|
+
|
|
127
|
+
```js
|
|
128
|
+
import { plan } from "@osqd/jql";
|
|
129
|
+
import { INDEXED, show } from "./store.mjs";
|
|
130
|
+
|
|
131
|
+
show(plan({ $or: [{ status: "open" }, { status: "paid" }] }, INDEXED));
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
```
|
|
135
|
+
pushed undefined
|
|
136
|
+
remaining {"$or":[{"status":"open"},{"status":"paid"}]}
|
|
137
|
+
complete false
|
|
138
|
+
kept $or: the store cannot answer an $or
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
`pushed` is `undefined` — there is nothing to send. Both branches are about an indexed field,
|
|
142
|
+
but this store cannot answer an `$or` at all, and half an `$or` is a different question. So
|
|
143
|
+
the whole thing stays here, and `kept` says so in one line.
|
|
144
|
+
|
|
145
|
+
## A store that can answer more
|
|
146
|
+
|
|
147
|
+
A store target is a capabilities object plus a translator. One ships with the library, for
|
|
148
|
+
document databases that speak the conventional query-document dialect:
|
|
149
|
+
|
|
150
|
+
```js
|
|
151
|
+
import { plan } from "@osqd/jql";
|
|
152
|
+
import { MONGO_CAPABILITIES, toMongoFilter } from "@osqd/jql/mongo";
|
|
153
|
+
|
|
154
|
+
const query = {
|
|
155
|
+
status: "paid",
|
|
156
|
+
"customer.country": "GB",
|
|
157
|
+
note: { $contains: "neighbour" },
|
|
158
|
+
"lines.quantity": { $gte: 3 },
|
|
159
|
+
};
|
|
160
|
+
|
|
161
|
+
const planned = plan(query, MONGO_CAPABILITIES);
|
|
162
|
+
|
|
163
|
+
console.log("complete:", planned.complete);
|
|
164
|
+
// A driver is handed real RegExp and Date objects, which JSON.stringify cannot show.
|
|
165
|
+
const readable = (value) => JSON.stringify(value, (_, held) => (held instanceof RegExp ? `/${held.source}/${held.flags}` : held));
|
|
166
|
+
console.log("filter :", readable(toMongoFilter(planned.pushed)));
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
```
|
|
170
|
+
complete: true
|
|
171
|
+
filter : {"$and":[{"status":"paid"},{"customer.country":"GB"},{"note":"/neighbour/"},{"lines.quantity":{"$gte":3}}]}
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
All four conjuncts went, so `complete` is `true` and there is no second pass to run:
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
const { pushed, remaining, complete } = plan(query, MONGO_CAPABILITIES);
|
|
178
|
+
const rows = await collection.find(toMongoFilter(pushed)).toArray();
|
|
179
|
+
const answer = complete ? rows : filter(rows, remaining);
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Look at what happened to `$contains`. The store has no such operator, so the target wrote an
|
|
183
|
+
**escaped pattern** that the store runs exactly as JQL would.
|
|
184
|
+
|
|
185
|
+
What a target will not do is guess. Any operator the store cannot answer *exactly* is left out
|
|
186
|
+
of its capabilities, so `plan` keeps those clauses here and the translator never sees them. A
|
|
187
|
+
translation that was *nearly* right would be the worst of both worlds: fewer rows than the
|
|
188
|
+
query asked for, from a store that looked as though it had answered.
|
|
189
|
+
|
|
190
|
+
## The limit of `complete`
|
|
191
|
+
|
|
192
|
+
Capabilities describe what a store can be *asked*. They cannot describe how it resolves a path
|
|
193
|
+
through your data, and those can differ.
|
|
194
|
+
|
|
195
|
+
The case to know about is an array held directly inside another array. JQL sees an array
|
|
196
|
+
through at every level (lesson 3); a store may apply a path segment to an array's elements but
|
|
197
|
+
not to the elements of *those* arrays. For such a document the two give different answers, and
|
|
198
|
+
`plan` cannot know, because nothing in the query says what shape the rows are.
|
|
199
|
+
|
|
200
|
+
If your collection stores arrays of arrays of documents, do not read `complete` as permission
|
|
201
|
+
to skip the second pass over those fields.
|
|
202
|
+
|
|
203
|
+
## Exercise
|
|
204
|
+
|
|
205
|
+
Harbour's store gains an index on `tags`. Add it to `INDEXED` and see what changes for
|
|
206
|
+
`{ status: "paid", tags: "repeat", note: { $contains: "same" } }`.
|
|
207
|
+
|
|
208
|
+
<details>
|
|
209
|
+
<summary>Answer</summary>
|
|
210
|
+
|
|
211
|
+
```js
|
|
212
|
+
import { plan } from "@osqd/jql";
|
|
213
|
+
import { INDEXED, show } from "./store.mjs";
|
|
214
|
+
|
|
215
|
+
const WITH_TAGS = { ...INDEXED, fields: [...INDEXED.fields, "tags"] };
|
|
216
|
+
|
|
217
|
+
show(plan({ status: "paid", tags: "repeat", note: { $contains: "same" } }, WITH_TAGS));
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
```
|
|
221
|
+
pushed {"$and":[{"status":"paid"},{"tags":"repeat"}]}
|
|
222
|
+
remaining {"note":{"$contains":"same"}}
|
|
223
|
+
complete false
|
|
224
|
+
kept note: the store cannot filter on "note"
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
`tags: "repeat"` pushed as an equality even though `tags` is a list, because *a value matches
|
|
228
|
+
any element* is a rule both sides share — and shared rules are exactly what makes a clause
|
|
229
|
+
safe to push.
|
|
230
|
+
|
|
231
|
+
`note` stayed, and the reason is worth reading closely: `the store cannot filter on "note"`.
|
|
232
|
+
Not "cannot answer `$contains`" — `note` is not in `fields` at all, so the field is refused
|
|
233
|
+
before the operator is ever considered. Two different fixes hide behind those two sentences
|
|
234
|
+
(index the column, or teach the store an operator), which is why `kept` says which one it is.
|
|
235
|
+
</details>
|
|
236
|
+
|
|
237
|
+
## Related
|
|
238
|
+
|
|
239
|
+
- [Pushing a query into a store](../guides/pushdown.md) — the guide, including writing your own target
|
|
240
|
+
- [Specification §14](../reference/specification.md#14-splitting-a-query) — the split, as a rule other implementations can follow
|
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# Lesson 16 — Extending it, and proving it
|
|
2
|
+
|
|
3
|
+
**Goal:** add an operator this project needs, check a filter before it meets anybody, and
|
|
4
|
+
finish Harbour.
|
|
5
|
+
|
|
6
|
+
← [Course](index.md) · Previous: [Pushing into a store](15-pushdown.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## Operators of your own
|
|
11
|
+
|
|
12
|
+
Some questions are not JSON. *"Is this address inside that network"*, *"does this score beat
|
|
13
|
+
the model's threshold"* — no amount of `$gt` expresses them, and the honest answer is to let a
|
|
14
|
+
project add an operator.
|
|
15
|
+
|
|
16
|
+
The cost of doing that is portability, so the language makes the cost **visible**: an added
|
|
17
|
+
operator's name must begin with `$x` and a capital letter. Somebody reading a stored filter
|
|
18
|
+
can see at a glance that it needs more than a standard engine, and a future version of JQL
|
|
19
|
+
cannot collide with a name you chose.
|
|
20
|
+
|
|
21
|
+
Save this as `harbour/within-days.mjs`:
|
|
22
|
+
|
|
23
|
+
```js
|
|
24
|
+
import { JqlError } from "@osqd/jql";
|
|
25
|
+
|
|
26
|
+
/** How recently an order was placed — the sort of question a project adds for itself. */
|
|
27
|
+
export const withinDays = {
|
|
28
|
+
name: "$xWithinDays",
|
|
29
|
+
cost: 2,
|
|
30
|
+
compile: (operand, at) => {
|
|
31
|
+
if (typeof operand !== "number" || !Number.isFinite(operand)) {
|
|
32
|
+
throw new JqlError(`takes a number of days, not ${JSON.stringify(operand)}`, at);
|
|
33
|
+
}
|
|
34
|
+
const since = Date.parse("2026-09-25T12:00:00Z") - operand * 86_400_000;
|
|
35
|
+
return (value) => typeof value === "string" && Date.parse(value) >= since;
|
|
36
|
+
},
|
|
37
|
+
};
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Using it
|
|
41
|
+
|
|
42
|
+
```js
|
|
43
|
+
import { filter } from "@osqd/jql";
|
|
44
|
+
import { orders } from "./orders.mjs";
|
|
45
|
+
import { withinDays } from "./within-days.mjs";
|
|
46
|
+
|
|
47
|
+
const operators = [withinDays];
|
|
48
|
+
const ids = (query) => filter(orders, query, { operators }).map((o) => o.id);
|
|
49
|
+
|
|
50
|
+
console.log("within 7 days :", ids({ placed: { $xWithinDays: 7 } }));
|
|
51
|
+
console.log("within 20 days:", ids({ placed: { $xWithinDays: 20 } }));
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
within 7 days : [ 'o-1006', 'o-1007', 'o-1008' ]
|
|
56
|
+
within 20 days: [ 'o-1003', 'o-1004', 'o-1005', 'o-1006', 'o-1007', 'o-1008' ]
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
From the query's side it is an ordinary operator: it composes with everything else, it can sit
|
|
60
|
+
inside `$or`, and `explain` describes it like any other clause.
|
|
61
|
+
|
|
62
|
+
## Three refusals that make it safe to have
|
|
63
|
+
|
|
64
|
+
```js
|
|
65
|
+
import { filter, validate } from "@osqd/jql";
|
|
66
|
+
import { orders } from "./orders.mjs";
|
|
67
|
+
import { withinDays } from "./within-days.mjs";
|
|
68
|
+
|
|
69
|
+
const say = (verdict) => console.log(" ", verdict.valid ? "accepted" : verdict.error.message);
|
|
70
|
+
|
|
71
|
+
console.log("an operand that is not a number:");
|
|
72
|
+
say(validate({ placed: { $xWithinDays: "seven" } }, { operators: [withinDays] }));
|
|
73
|
+
|
|
74
|
+
console.log("an engine that was not given it:");
|
|
75
|
+
say(validate({ placed: { $xWithinDays: 7 } }));
|
|
76
|
+
|
|
77
|
+
console.log("a name without the $x:");
|
|
78
|
+
try {
|
|
79
|
+
filter(orders, { placed: { $withinDays: 7 } }, { operators: [{ ...withinDays, name: "$withinDays" }] });
|
|
80
|
+
console.log(" accepted");
|
|
81
|
+
} catch (error) {
|
|
82
|
+
console.log(" ", error.message);
|
|
83
|
+
}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
an operand that is not a number:
|
|
88
|
+
at placed.$xWithinDays: takes a number of days, not "seven"
|
|
89
|
+
an engine that was not given it:
|
|
90
|
+
at placed.$xWithinDays: "$xWithinDays" is an added operator, and this query was compiled without it; pass it in `operators`
|
|
91
|
+
a name without the $x:
|
|
92
|
+
at operators: "$withinDays" cannot name an added operator: the name is $x and a capital, such as "$xCidr", so a query that needs more than standard JQL says so
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Each of those is a property worth having:
|
|
96
|
+
|
|
97
|
+
- **`compile` runs once**, when the query is compiled, and returns the test. Validate the
|
|
98
|
+
operand there and throw `JqlError` — the sentence reaches whoever wrote the query, with the
|
|
99
|
+
place in it.
|
|
100
|
+
- **A query that needs an added operator says so.** The second refusal is an engine *without*
|
|
101
|
+
the operator, naming it and saying what to do, rather than treating it as a typo and
|
|
102
|
+
matching nothing.
|
|
103
|
+
- **The naming rule is enforced**, so the visibility it buys is real.
|
|
104
|
+
|
|
105
|
+
There is a fourth property with no error message: **`cost`** is a hint for ordering an `$and`,
|
|
106
|
+
so a cheap equality runs before your expensive test. It changes the speed and never the
|
|
107
|
+
answer.
|
|
108
|
+
|
|
109
|
+
Before reaching for any of this, look at the field operators once more. `$word`, `$glob`,
|
|
110
|
+
`$length` and `{ "$field": … }` exist precisely because they are the four things people most
|
|
111
|
+
often added an escape hatch to do — and a query using them is still portable JQL.
|
|
112
|
+
|
|
113
|
+
## Proving a filter before it meets anybody
|
|
114
|
+
|
|
115
|
+
The language ships its own test suite as **data**: `conformance/cases.json`, 156 matching
|
|
116
|
+
cases and 27 request cases. Each is a document set, a query, and the answer any implementation
|
|
117
|
+
must give.
|
|
118
|
+
|
|
119
|
+
```json
|
|
120
|
+
{"group": "equality", "name": "an empty query matches everything", "documents": "people", "query": {}, "matches": [0, 1, 2, 3]}
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
You need it mostly if you implement JQL in another language. But it is also the clearest
|
|
124
|
+
statement of what an operator means, in twelve groups — equality, arrays and paths, existence
|
|
125
|
+
and type, ordering, strings, logic, text, references, length, glob, relative dates, refusals.
|
|
126
|
+
When you are unsure what something does, the case is faster to read than the prose.
|
|
127
|
+
|
|
128
|
+
For your *own* filters, the tools are ones you already have:
|
|
129
|
+
|
|
130
|
+
| | |
|
|
131
|
+
| --- | --- |
|
|
132
|
+
| `validate` | a query from outside, refused with a sentence (lesson 12) |
|
|
133
|
+
| `explain` | why one row did or did not match (lesson 9) |
|
|
134
|
+
| `jql --explain` | the same from a shell, over the first line of a real file (lesson 14) |
|
|
135
|
+
| `fingerprint` | whether the filter in the test is the filter in production (lesson 13) |
|
|
136
|
+
| `toText(…).complete` | whether the box is showing the whole filter (lesson 11) |
|
|
137
|
+
|
|
138
|
+
## Harbour, finished
|
|
139
|
+
|
|
140
|
+
Everything the course built, in one place:
|
|
141
|
+
|
|
142
|
+
```ts
|
|
143
|
+
// vocabulary.mjs — the names, once, for both front ends (lesson 10)
|
|
144
|
+
export const ORDERS = defineVocabulary()({ fields: { … }, text: [ … ] });
|
|
145
|
+
|
|
146
|
+
// the console's search box (lesson 11)
|
|
147
|
+
const query = parseText(typed, { vocabulary: ORDERS });
|
|
148
|
+
|
|
149
|
+
// the table (lessons 7, 15)
|
|
150
|
+
const { pushed, remaining, complete } = plan(query, STORE_CAPABILITIES);
|
|
151
|
+
const rows = await collection.find(toStoreFilter(pushed)).toArray();
|
|
152
|
+
const page = search(complete ? rows : filter(rows, remaining), {
|
|
153
|
+
sort: { placed: -1 }, skip, limit, omit: ["customer.email"],
|
|
154
|
+
}, { vocabulary: ORDERS });
|
|
155
|
+
|
|
156
|
+
// the number above it (lesson 8)
|
|
157
|
+
const byStatus = group(rows, "status", { vocabulary: ORDERS, where: query });
|
|
158
|
+
|
|
159
|
+
// "why isn't this here?" (lesson 9)
|
|
160
|
+
const why = explain(query, row, { vocabulary: ORDERS });
|
|
161
|
+
|
|
162
|
+
// saving it (lesson 13)
|
|
163
|
+
await filters.insert({ key: fingerprint(query), name, query: canonical(query) });
|
|
164
|
+
|
|
165
|
+
// and the public API, which trusts none of the above (lesson 12)
|
|
166
|
+
const checked = validate(JSON.parse(body), { vocabulary: PUBLIC, limits });
|
|
167
|
+
if (!checked.valid) return respond(400, { error: checked.error.message });
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
One query document runs in the console, in the API, in the store and in the nightly export,
|
|
171
|
+
and means the same thing in all four. That is the whole argument for a query being data.
|
|
172
|
+
|
|
173
|
+
## Where to go next
|
|
174
|
+
|
|
175
|
+
| | |
|
|
176
|
+
| --- | --- |
|
|
177
|
+
| [The specification](../reference/specification.md) | what every operator means, precisely — the thing to read when you disagree with an answer |
|
|
178
|
+
| [Adopting JQL in a project](../guides/adopting.md) | moving an existing project's filtering onto it |
|
|
179
|
+
| [Pushing a query into a store](../guides/pushdown.md) | writing a target for a store of your own |
|
|
180
|
+
| [Design decisions](../design/decisions.md) | what this library refuses to do, and what that costs |
|
|
181
|
+
| [Performance](../design/performance.md) | how fast it is, how that is measured, and what makes it so |
|
|
182
|
+
|
|
183
|
+
## A last exercise
|
|
184
|
+
|
|
185
|
+
Take the filter your own project uses most — the one behind a dashboard or an endpoint — and
|
|
186
|
+
write it as a JQL document. Two things usually come out of that:
|
|
187
|
+
|
|
188
|
+
1. a clause you cannot express, which is worth knowing about; and
|
|
189
|
+
2. a clause you *can* express that you had been doing in application code, which is worth
|
|
190
|
+
moving.
|
|
191
|
+
|
|
192
|
+
If it is the first, the [specification](../reference/specification.md) will tell you whether
|
|
193
|
+
it is a gap in the language or a gap in this course. If it is the second, you have just made a
|
|
194
|
+
filter storable, sendable and explainable — which is what the whole thing is for.
|
|
195
|
+
|
|
196
|
+
## Related
|
|
197
|
+
|
|
198
|
+
- [Course index](index.md) — all sixteen lessons
|
|
199
|
+
- [Library API](../reference/api.md) — every export, grouped by what it is for
|
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
# The JQL course
|
|
2
|
+
|
|
3
|
+
Sixteen lessons that build one real integration, from a first query to a filter you can put
|
|
4
|
+
behind a public endpoint.
|
|
5
|
+
|
|
6
|
+
← [Documentation](../index.md)
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## What this is
|
|
11
|
+
|
|
12
|
+
The [reference documentation](../index.md) answers *"how does X work?"*. This answers *"what
|
|
13
|
+
do I do, and in what order?"* — every capability of the library, taught in the order that
|
|
14
|
+
makes each one make sense, with something to run at every step.
|
|
15
|
+
|
|
16
|
+
It is written to be worked through rather than read. Every lesson is a handful of short,
|
|
17
|
+
self-contained programs, each followed by what it prints — and **every one of those outputs
|
|
18
|
+
was produced by running the code**, not by imagining what it would print. A script in this
|
|
19
|
+
repository runs all of them on the packed package before a release, so a page that drifts
|
|
20
|
+
from the library fails the build.
|
|
21
|
+
|
|
22
|
+
**Time:** about three hours to do properly. Lessons 1–5 are the language itself and are worth
|
|
23
|
+
slowing down for; everything after them assumes you have those.
|
|
24
|
+
|
|
25
|
+
## Who it is for
|
|
26
|
+
|
|
27
|
+
A TypeScript or JavaScript developer with a collection of things and a filter to write — rows
|
|
28
|
+
in a dashboard, records behind an API, lines in a log. You need no background in query
|
|
29
|
+
languages: the operator names are the conventional ones, so if you have written a filter in
|
|
30
|
+
almost any document store you will recognise most of them, and
|
|
31
|
+
[the specification](../reference/specification.md) says precisely what each one means here.
|
|
32
|
+
|
|
33
|
+
## The running example
|
|
34
|
+
|
|
35
|
+
You are building **Harbour**, the order desk of a small shop. It has an order book worth
|
|
36
|
+
searching, a support console worth giving a search box, and an API worth not letting strangers
|
|
37
|
+
run arbitrary queries against. Every lesson adds one thing to Harbour, and by lesson 16 you
|
|
38
|
+
have a complete, tested, production-shaped setup.
|
|
39
|
+
|
|
40
|
+
## Set up once
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
mkdir harbour && cd harbour
|
|
44
|
+
npm init -y && npm pkg set type=module
|
|
45
|
+
npm install @osqd/jql
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Then save the order book as `harbour/orders.mjs`. Every lesson imports it, and it is small
|
|
49
|
+
enough that you can check any answer by hand:
|
|
50
|
+
|
|
51
|
+
```js
|
|
52
|
+
/** The Harbour order book: eight orders, small enough to check an answer by hand. */
|
|
53
|
+
export const orders = [
|
|
54
|
+
{
|
|
55
|
+
id: "o-1001", placed: "2026-09-01T09:15:00Z", status: "paid", channel: "web",
|
|
56
|
+
customer: { name: "Ada Lovelace", country: "GB", email: "ada@example.com" },
|
|
57
|
+
lines: [{ sku: "pen-fine", title: "Fine liner", quantity: 3, price: 4.5 }, { sku: "ink-blue", title: "Blue ink", quantity: 1, price: 9 }],
|
|
58
|
+
total: 22.5, paid: 22.5, tags: ["gift"], note: "leave with the neighbour",
|
|
59
|
+
},
|
|
60
|
+
{
|
|
61
|
+
id: "o-1002", placed: "2026-09-03T14:02:00Z", status: "open", channel: "web",
|
|
62
|
+
customer: { name: "Grace Hopper", country: "US", email: "grace@example.com" },
|
|
63
|
+
lines: [{ sku: "pad-a5", title: "A5 pad", quantity: 10, price: 3 }],
|
|
64
|
+
total: 30, paid: 0, tags: [],
|
|
65
|
+
},
|
|
66
|
+
{
|
|
67
|
+
id: "o-1003", placed: "2026-09-07T08:40:00Z", status: "paid", channel: "phone",
|
|
68
|
+
customer: { name: "Alan Turing", country: "GB", email: "alan@example.com" },
|
|
69
|
+
lines: [{ sku: "pen-fine", title: "Fine liner", quantity: 1, price: 4.5 }],
|
|
70
|
+
total: 4.5, paid: 4.5, tags: ["repeat"], note: "same as last time",
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
id: "o-1004", placed: "2026-09-11T19:30:00Z", status: "refunded", channel: "web",
|
|
74
|
+
customer: { name: "Katherine Johnson", country: "US", email: "kj@example.com" },
|
|
75
|
+
lines: [{ sku: "ink-blue", title: "Blue ink", quantity: 2, price: 9 }, { sku: "ink-red", title: "Red ink", quantity: 2, price: 9 }],
|
|
76
|
+
total: 36, paid: 0, tags: ["damaged", "repeat"], note: "arrived leaking",
|
|
77
|
+
},
|
|
78
|
+
{
|
|
79
|
+
id: "o-1005", placed: "2026-09-14T11:05:00Z", status: "open", channel: "app",
|
|
80
|
+
customer: { name: "Tim Berners-Lee", country: "GB", email: "tim@example.com" },
|
|
81
|
+
lines: [{ sku: "desk-lamp", title: "Desk lamp", quantity: 1, price: 140 }, { sku: "pad-a5", title: "A5 pad", quantity: 10, price: 3 }],
|
|
82
|
+
total: 170, paid: 0, tags: ["fragile"],
|
|
83
|
+
},
|
|
84
|
+
{
|
|
85
|
+
id: "o-1006", placed: "2026-09-18T16:45:00Z", status: "paid", channel: "app",
|
|
86
|
+
customer: { name: "Barbara Liskov", country: "US", email: "barbara@example.com" },
|
|
87
|
+
lines: [{ sku: "pad-a5", title: "A5 pad", quantity: 2, price: 3 }, { sku: "pen-fine", title: "Fine liner", quantity: 6, price: 4.5 }],
|
|
88
|
+
total: 33, paid: 33, tags: [], note: "gift wrap, no receipt",
|
|
89
|
+
},
|
|
90
|
+
{
|
|
91
|
+
id: "o-1007", placed: "2026-09-21T07:20:00Z", status: "cancelled", channel: "web",
|
|
92
|
+
customer: { name: "Margaret Hamilton", country: "DE", email: "margaret@example.com" },
|
|
93
|
+
lines: [{ sku: "desk-lamp", title: "Desk lamp", quantity: 2, price: 140 }],
|
|
94
|
+
total: 280, paid: 0, tags: ["fragile"], note: null,
|
|
95
|
+
},
|
|
96
|
+
{
|
|
97
|
+
id: "o-1008", placed: "2026-09-24T13:10:00Z", status: "paid", channel: "phone",
|
|
98
|
+
customer: { name: "Ada Byron", country: "FR", email: "ada.b@example.com" },
|
|
99
|
+
lines: [{ sku: "ink-red", title: "Red ink", quantity: 1, price: 9 }, { sku: "pad-a5", title: "A5 pad", quantity: 1, price: 3 }],
|
|
100
|
+
total: 12, paid: 12, tags: ["repeat"],
|
|
101
|
+
},
|
|
102
|
+
];
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Look at it for a moment before you start. Three orders have no `note` at all, one has a `note`
|
|
106
|
+
of `null`, two have no tags, and one has two lines that are nothing like each other. Every one
|
|
107
|
+
of those is there to catch a lesson out.
|
|
108
|
+
|
|
109
|
+
Each example is a whole program: put it in a file next to `orders.mjs` and run it with
|
|
110
|
+
`node`. A few lessons introduce a second module — a vocabulary, a store description — and say
|
|
111
|
+
so where they do. Lesson 6 is the one that wants TypeScript; nothing here needs a server or a
|
|
112
|
+
database at all.
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## Part 1 — The language
|
|
117
|
+
|
|
118
|
+
| | | |
|
|
119
|
+
|-|-|-|
|
|
120
|
+
| 1 | [Your first query](01-first-query.md) | A query is data. Four helpers, and a refusal. |
|
|
121
|
+
| 2 | [Asking precisely](02-operators.md) | Comparison, membership, existence, strings, patterns. |
|
|
122
|
+
| 3 | [Arrays and paths](03-arrays-and-paths.md) | How a path meets an array. **The most important lesson here.** |
|
|
123
|
+
| 4 | [Combining and negating](04-combining.md) | `$or`, `$nor`, `$not` — and what "not" means over a list. |
|
|
124
|
+
| 5 | [Dates and windows](05-dates.md) | An instant in JSON, and a window that stays true tomorrow. |
|
|
125
|
+
|
|
126
|
+
## Part 2 — Letting the compiler help
|
|
127
|
+
|
|
128
|
+
| | | |
|
|
129
|
+
|-|-|-|
|
|
130
|
+
| 6 | [Typed queries](06-typed-queries.md) | A typo in a field name as a compile error rather than an empty result. |
|
|
131
|
+
|
|
132
|
+
## Part 3 — Whole questions about a collection
|
|
133
|
+
|
|
134
|
+
| | | |
|
|
135
|
+
|-|-|-|
|
|
136
|
+
| 7 | [Requests](07-requests.md) | Order, pages, and which fields you hand back. |
|
|
137
|
+
| 8 | [Counting by a field](08-grouping.md) | The number above the table. |
|
|
138
|
+
| 9 | [Why did this not match?](09-explaining.md) | The clause that decided it, with the values it read. |
|
|
139
|
+
|
|
140
|
+
## Part 4 — People typing
|
|
141
|
+
|
|
142
|
+
| | | |
|
|
143
|
+
|-|-|-|
|
|
144
|
+
| 10 | [A vocabulary](10-vocabulary.md) | The names your data has, in one place, for both front ends. |
|
|
145
|
+
| 11 | [The search box](11-the-search-box.md) | Text into JQL, JQL back into text, and completions. |
|
|
146
|
+
|
|
147
|
+
## Part 5 — Production
|
|
148
|
+
|
|
149
|
+
| | | |
|
|
150
|
+
|-|-|-|
|
|
151
|
+
| 12 | [Queries from outside](12-untrusted.md) | Accepting one from a URL without accepting everything. |
|
|
152
|
+
| 13 | [Saved filters](13-saved-filters.md) | One shape and one short name per question. |
|
|
153
|
+
| 14 | [Logs and streams](14-streams-and-cli.md) | The async helpers, and the `jql` command. |
|
|
154
|
+
| 15 | [Pushing into a store](15-pushdown.md) | Letting a backend answer the part it can. |
|
|
155
|
+
| 16 | [Extending it, and proving it](16-extending.md) | Operators of your own, the conformance suite, and a finished Harbour. |
|
|
156
|
+
|
|
157
|
+
---
|
|
158
|
+
|
|
159
|
+
## How to get the most out of it
|
|
160
|
+
|
|
161
|
+
**Type the code, do not paste it.** The examples are short on purpose.
|
|
162
|
+
|
|
163
|
+
**Predict the answer before you run it.** Eight orders is few enough to work out in your head,
|
|
164
|
+
and the times you are wrong are the times you learn where this language differs from the one
|
|
165
|
+
in your head.
|
|
166
|
+
|
|
167
|
+
**When something surprises you, chase it.** Every lesson ends with links into the reference
|
|
168
|
+
documentation and the specification for the thing you just used.
|
|
169
|
+
|
|
170
|
+
## One sentence to keep
|
|
171
|
+
|
|
172
|
+
> **A query that is not understood is refused, never run as something that looks like it
|
|
173
|
+
> worked.**
|
|
174
|
+
|
|
175
|
+
You will meet it in lesson 1, thirty seconds in, when a misspelt operator throws instead of
|
|
176
|
+
quietly matching nothing. Every other refusal in this course is that same rule: the failure
|
|
177
|
+
this language exists to prevent is a filter that looks applied and is not.
|
|
178
|
+
|
|
179
|
+
Start with [lesson 1](01-first-query.md).
|
|
180
|
+
|
|
181
|
+
## Related
|
|
182
|
+
|
|
183
|
+
- [Quick start](../start/quick-start.md) — if you want five minutes rather than three hours
|
|
184
|
+
- [The specification](../reference/specification.md) — what every operator means, precisely
|
|
185
|
+
- [Library API](../reference/api.md) — every export, grouped by what it is for
|