@jarenjs/json 0.43.1 → 0.46.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ARCHITECTURE.md +1 -1
- package/README.md +43 -4
- package/dist/types/query/operators.d.ts +8 -0
- package/docs/QUERY-FORMAT.md +97 -0
- package/package.json +2 -2
- package/schemas/jaren-jslt.draft-07.schema.json +7 -1
- package/schemas/jaren-jslt.schema.json +7 -1
- package/schemas/jaren-query.draft-07.schema.json +7 -1
- package/schemas/jaren-query.schema.json +4 -2
- package/src/query/normalize.js +16 -0
- package/src/query/operators.js +69 -0
package/ARCHITECTURE.md
CHANGED
|
@@ -163,7 +163,7 @@ Grouping, `$distinct` and the `$orderby` machinery need a total, deterministic e
|
|
|
163
163
|
|
|
164
164
|
### The operator registry
|
|
165
165
|
|
|
166
|
-
All
|
|
166
|
+
All 99 §8 operators live in one table in `operators.js` — the query-language analogue of `path.js`'s `FUNCTIONS` table. A test derives the count from the registry and asserts it against QUERY-FORMAT §8 *and* against every committed document that states a number — this file, the README twice, `site.md` and `@jarenjs/linq`'s architecture — so an operator cannot be added without moving all of them in the same change:
|
|
167
167
|
|
|
168
168
|
```javascript
|
|
169
169
|
'$substring': {
|
package/README.md
CHANGED
|
@@ -386,9 +386,9 @@ queryJson({
|
|
|
386
386
|
}, [1, 2, 3, 4, 5]); // [2, 3, 4] — a 3-point moving average
|
|
387
387
|
```
|
|
388
388
|
|
|
389
|
-
The operator library (
|
|
389
|
+
The operator library (99 operators: comparisons, IEEE-double arithmetic, logic, strings with I-Regexp `$match`/`$search`/`$replace`, aggregates, sequence tools like `$distinct`/`$subsequence`/`$range`, type predicates and casts, `$coalesce`, the RFC 3339 date family, the spatial family and `$similarity`) is cataloged in [QUERY-FORMAT.md §8](./docs/QUERY-FORMAT.md#8-operators).
|
|
390
390
|
|
|
391
|
-
**Extending the vocabulary (host opt-in).** The
|
|
391
|
+
**Extending the vocabulary (host opt-in).** The 99 are closed, but a host can add more the way `@jarenjs/validate` gains formats from `@jarenjs/formats`: `createJsltRegistry().use(mathPack).use(financePack)` composes packs of pure `@jarenjs/core` functions into a compiler, so `{ "$sqrt": "$.variance" }` and `{ "$npv": ["$.rate", "$.cashflows[*]"] }` work in a stylesheet, a bare query, and `@jarenjs/linq` — while a document compiled *without* the registry still rejects them. Aggregators fold a `seq` operand to an array before the pure call. See [JSLT-FORMAT.md §13](./docs/JSLT-FORMAT.md#13-registered-operators-host-opt-in-non-normative).
|
|
392
392
|
|
|
393
393
|
**Dates are RFC 3339 strings** ([§8.13](./docs/QUERY-FORMAT.md#813-dates-and-times)): `$is-date`/`$is-time`/`$is-datetime`/`$is-duration` test the lexical forms, `$year`…`$seconds` and `$offset` read components *lexically, in the value's own offset* (so "group by month" means what you expect), `$week`/`$week-year`/`$quarter`/`$weekday` add the derived calendar fields, and `$epoch`/`$datetime` convert to and from epoch milliseconds — the one place a value is shifted to UTC, and therefore the way to compare instants across offsets. There is deliberately no `current-dateTime`: a compiled query is cached by document identity and saved as a rule, so it must answer the same for the same input forever.
|
|
394
394
|
|
|
@@ -406,7 +406,7 @@ queryJson({
|
|
|
406
406
|
|
|
407
407
|
Bucketing by week is the shape components alone cannot express, because a week boundary is arithmetic rather than a field. Patterns stay locale-independent — `MMMM` and `EEEE` are rejected rather than silently rendered in English — because localized text belongs to the presentation layer, not to a query.
|
|
408
408
|
|
|
409
|
-
**Geography is GeoJSON** ([§8.14](./docs/QUERY-FORMAT.md#814-spatial)), for the same reason dates are RFC 3339 strings: it is what the document already holds. Operands are a bare `[longitude, latitude]` position, a geometry, a `Feature` or a `FeatureCollection`, and wrappers unwrap for you. `$distance`, `$area` and `$length` answer in metres on the WGS 84 sphere — never planar, because a Euclidean answer over raw degrees is wrong by two thirds over a kilometre at Dutch latitudes. `$within` tests containment, `$bbox`/`$centroid` measure, and `$bbox-intersects` is named for exactly what it tests, since an `$intersects` that compared only boxes would be a lie the first time two L-shapes shared one.
|
|
409
|
+
**Geography is GeoJSON** ([§8.14](./docs/QUERY-FORMAT.md#814-spatial)), for the same reason dates are RFC 3339 strings: it is what the document already holds. Thirteen of the 98 operators are spatial — eight measurements and predicates, five conversions — and JSLT and JTLT inherit every one. Operands are a bare `[longitude, latitude]` position, a geometry, a `Feature` or a `FeatureCollection`, and wrappers unwrap for you. `$distance`, `$area` and `$length` answer in metres on the WGS 84 sphere — never planar, because a Euclidean answer over raw degrees is wrong by two thirds over a kilometre at Dutch latitudes. `$within` tests containment, `$bbox`/`$centroid` measure, and `$bbox-intersects` is named for exactly what it tests, since an `$intersects` that compared only boxes would be a lie the first time two L-shapes shared one.
|
|
410
410
|
|
|
411
411
|
```js
|
|
412
412
|
queryJson({
|
|
@@ -442,6 +442,45 @@ The one trap: `coordinates` and `features` take the **array constructor** (the b
|
|
|
442
442
|
|
|
443
443
|
A geohash is a string, so **bucketing and tiling** need no operator: `$groupby` over `$substring` groups by cell and an index over the hash is a spatial index. **Proximity is different**, and a prefix test is not it — two points ten metres apart can differ in the *first* character of their cell, so a single prefix misses a neighbour at every cell boundary. `$geohash-neighbours` gives the nine-cell probe a correct proximity query needs; narrow it with `$distance` when an exact radius matters. The whole recipe, end to end, is in [HOWTO](../../docs/HOWTO.md#getting-geographic-data-in-and-out).
|
|
444
444
|
|
|
445
|
+
### Similarity, and why there is no `$knn`
|
|
446
|
+
|
|
447
|
+
An embedding is an array of numbers, which a JSON document can already hold, so
|
|
448
|
+
the language needs exactly one operator to compare two of them
|
|
449
|
+
([§8.15](./docs/QUERY-FORMAT.md#815-vectors)):
|
|
450
|
+
|
|
451
|
+
```javascript
|
|
452
|
+
queryJson({ $similarity: [[3, 4], [4, 3]] }, {}); // 0.96
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
`$similarity` is cosine, in [-1, 1], higher-is-better. One metric, because a
|
|
456
|
+
stored vector is normally normalized and cosine, the dot product and Euclidean
|
|
457
|
+
distance then rank the same candidates in the same order — a second metric
|
|
458
|
+
would buy a different number for the same answer. An operand that is not an
|
|
459
|
+
array of numbers is `JQ2001`; two vectors that cannot be compared — different
|
|
460
|
+
widths, empty, or carrying a component a computation made non-finite — answer
|
|
461
|
+
**empty**, so a document never carries a score nobody computed. Nothing is
|
|
462
|
+
padded or truncated to make a comparison possible.
|
|
463
|
+
|
|
464
|
+
"The k most similar" needs no keyword either: it is an ordering and a window,
|
|
465
|
+
and both already exist.
|
|
466
|
+
|
|
467
|
+
```javascript
|
|
468
|
+
queryJson({ $subsequence: [{
|
|
469
|
+
$for: { m: '$.memories[*]' },
|
|
470
|
+
$orderby: [
|
|
471
|
+
{ $key: { $similarity: ['$m.embedding', '$query'] }, $dir: 'desc', $empty: 'least' },
|
|
472
|
+
'$m.id',
|
|
473
|
+
],
|
|
474
|
+
$return: '$m.text',
|
|
475
|
+
}, 0, 10] }, data, { query: vector });
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
`$empty: 'least'` under a descending sort puts the rows with no vector **last**
|
|
479
|
+
— present in the input, never in the top k, never scored — and the second key
|
|
480
|
+
breaks ties by identity, so the same document answers with the same rows every
|
|
481
|
+
time it runs. `@jarenjs/linq` spells the whole chain fluently as
|
|
482
|
+
`.orderByDescending(m => m.embedding.similarity(q), { empty: 'least' }).take(10)`.
|
|
483
|
+
|
|
445
484
|
### External parameters
|
|
446
485
|
|
|
447
486
|
A variable no phrase binds is an **external**: use is the declaration. The compiled query exposes the collected names and takes bindings at call time:
|
|
@@ -524,7 +563,7 @@ The validate step is not redundant: provider structured-output implementations s
|
|
|
524
563
|
|
|
525
564
|
For those strict subsets each grammar also ships an **LLM profile** — [`jaren-query.llm-profile.schema.json`](./schemas/jaren-query.llm-profile.schema.json) and [`jaren-jslt.llm-profile.schema.json`](./schemas/jaren-jslt.llm-profile.schema.json) — a mechanically derived, pure *relaxation* of the canonical artifact: `patternProperties`, `propertyNames` and asserted `format`s are removed (each restated in the nearest `description`, which the model still reads) and every `oneOf` becomes `anyOf`. That last swap costs no precision: the canonical grammar discriminates its branches by member name — a map constructor has no `$`-prefixed key, every phrase pins its own (`$const`, `$map`, `$call`, `$return`, …) or draws its single key from a disjoint operator enum — so at most one branch can ever match a document, and exactly-one and at-least-one accept the same language. It is also *required* once the name constraints are gone: `oneOf` would then reject `{"$eq": [1, 2]}` for matching two now-overlapping branches. Every canonical-valid document validates under the profile; the reverse is deliberately not guaranteed, which is exactly why the local validate-then-compile step above stays mandatory. Hand the profile to the provider's constrained decoder, validate locally against the canonical schema. (`@jarenjs/ai`'s `createStructuredOutput` packages this handshake.)
|
|
526
565
|
|
|
527
|
-
A relaxation, though, is not a *shrink*: restating every removed constraint in a `description` means the JSLT profile is 19,158 characters against the canonical 18,
|
|
566
|
+
A relaxation, though, is not a *shrink*: restating every removed constraint in a `description` means the JSLT profile is 19,158 characters against the canonical 18,817, and on a small model an oversized `response_format` is the difference between a document and an empty reply. So the JSLT grammar also ships an **authoring profile** — [`jaren-jslt.authoring.schema.json`](./schemas/jaren-jslt.authoring.schema.json), **3,491 characters** — the same mechanical derivation with one more step: the `queryDocument` `$ref` that pulls the entire expression language into the document grammar is left *open*. What survives is the document shape (envelope, rules, match specs, modes); what leaves is the ~16 kB of phrase shapes. It is a pure widening of the canonical language, so nothing the engine can run becomes inexpressible, and it is deliberately weaker — a body of pure nonsense decodes under it. That is the trade, and it is only safe because the two authorities after the decoder are unchanged: validate against the canonical schema, then compile. The vocabulary the cut removes belongs in the prompt instead, where `@jarenjs/ai`'s `operatorCrib` puts it (operator names grouped by arity, ~1.1 kB, read off the query artifact). `@jarenjs/ai`'s [`createStylesheetAuthor`](../ai/README.md#when-the-grammar-is-too-big-to-decode-measured) packages the whole handshake.
|
|
528
567
|
|
|
529
568
|
Query documents are plain JSON, so they travel through the rest of an LLM toolchain as-is: function-call arguments, retrieval filters, audit logs, replay.
|
|
530
569
|
|
|
@@ -573,6 +573,14 @@ export declare const OPERATORS: Readonly<{
|
|
|
573
573
|
result: typeof resultEmptyPropagates;
|
|
574
574
|
compile: (gets: any, args: any) => (f: any) => any;
|
|
575
575
|
};
|
|
576
|
+
$similarity: {
|
|
577
|
+
params: Readonly<{
|
|
578
|
+
kinds: readonly string[];
|
|
579
|
+
min: 2;
|
|
580
|
+
}>;
|
|
581
|
+
result: typeof RESULT_OPT;
|
|
582
|
+
compile: (gets: any, args: any) => (f: any) => number | typeof EMPTY;
|
|
583
|
+
};
|
|
576
584
|
'$date-add': {
|
|
577
585
|
params: Readonly<{
|
|
578
586
|
kinds: readonly string[];
|
package/docs/QUERY-FORMAT.md
CHANGED
|
@@ -1385,6 +1385,13 @@ value. This matters wherever such an expression sits in a position that expects
|
|
|
1385
1385
|
one item: `[{"$bbox": expr}]` builds an empty array rather than a one-item one,
|
|
1386
1386
|
and `{"m": {"$bbox": expr}}` omits the member entirely.
|
|
1387
1387
|
|
|
1388
|
+
**Traversal is not validation.** `$within`, `$area` and `$length` walk the
|
|
1389
|
+
rings they are given and treat a ring as closed whether or not its last
|
|
1390
|
+
position repeats its first — the reading every implementation makes — so a
|
|
1391
|
+
ring `isValidGeoJson` (or the meta-schema) refuses still measures. A document
|
|
1392
|
+
that needs the judgment composes it: `$valid` against the GeoJSON meta-schema,
|
|
1393
|
+
or the `geojson` format, before the measurement, not inside it.
|
|
1394
|
+
|
|
1388
1395
|
**Boxes do not cross the antimeridian.** RFC 7946 §3.1.9 tells producers to cut
|
|
1389
1396
|
geometries at ±180° rather than let them span it, and this format follows that
|
|
1390
1397
|
rather than re-joining what a producer split: `$bbox` of an uncut geometry whose
|
|
@@ -1428,6 +1435,96 @@ the answer.
|
|
|
1428
1435
|
selects the cities inside a region and orders them by how far they are from a
|
|
1429
1436
|
point — a spatial filter and a spatial sort, in the language's own clauses.
|
|
1430
1437
|
|
|
1438
|
+
### 8.15 Vectors
|
|
1439
|
+
|
|
1440
|
+
A vector is an **array of numbers** — what a JSON document already holds when
|
|
1441
|
+
something has been embedded. There is no vector type to construct, no width to
|
|
1442
|
+
declare, and nothing to register: `[0.1, 0.2, 0.3]` is one.
|
|
1443
|
+
|
|
1444
|
+
| Operator | Definition |
|
|
1445
|
+
|---|---|
|
|
1446
|
+
| `$similarity` | `[a, b]` → the cosine similarity of two vectors, a number in [-1, 1]; two vectors that cannot be compared → empty |
|
|
1447
|
+
|
|
1448
|
+
Cosine is the only metric, and that is a decision rather than a first
|
|
1449
|
+
instalment. For vectors of unit length — the form an embedding is normally
|
|
1450
|
+
stored in — cosine, the dot product and Euclidean distance all rank the same
|
|
1451
|
+
candidates in the same order, so a second metric would buy a different *number*
|
|
1452
|
+
for the same *answer* while doubling every recipe below. **Higher is better**:
|
|
1453
|
+
`1` is the same direction, `0` unrelated, `-1` opposite, which is why a
|
|
1454
|
+
similarity sort is always descending.
|
|
1455
|
+
|
|
1456
|
+
**The operand split is §8.14's.** A value that is not an array of numbers has
|
|
1457
|
+
no similarity to anything — a string, an object, an array with a non-number in
|
|
1458
|
+
it — and is `JQ2001`, the same type-level refusal a non-geographic operand
|
|
1459
|
+
gets. Two operands that *are* arrays of numbers but cannot be compared —
|
|
1460
|
+
different widths, an empty array, or a component a computation made non-finite
|
|
1461
|
+
— answer the **empty sequence**, the same data-level answer a measurement with
|
|
1462
|
+
no bounded position gives. Nothing is padded, truncated or zero-filled to make
|
|
1463
|
+
a comparison possible: two vectors of different widths are not a near miss,
|
|
1464
|
+
they are unrelated.
|
|
1465
|
+
|
|
1466
|
+
One degenerate pair is a **score, not a refusal**: a vector of all zeros
|
|
1467
|
+
points nowhere, and it answers `0` against everything, including itself. That
|
|
1468
|
+
is the one number here not derived from a direction, and it is the truthful
|
|
1469
|
+
answer to "how aligned are these" when one of them has no alignment — an
|
|
1470
|
+
all-zero embedding sorts below every real one and above the rows that have no
|
|
1471
|
+
vector at all, which is exactly where it belongs.
|
|
1472
|
+
|
|
1473
|
+
Because of that rule `$similarity` is **optional-valued**: it can answer empty
|
|
1474
|
+
for two operands that are present, so `[{"$similarity": [a, b]}]` builds an
|
|
1475
|
+
empty array rather than a one-item one and `{"score": {"$similarity": [a, b]}}`
|
|
1476
|
+
omits the member entirely. That is the answer to want — a document that carries
|
|
1477
|
+
no score is honest, and one carrying `0` for a comparison that never happened
|
|
1478
|
+
is not, because `0` is a real similarity.
|
|
1479
|
+
|
|
1480
|
+
**Nearest-neighbours is a composition, not an operator.** "The k most similar"
|
|
1481
|
+
is an ordering and a window, and this format already spells both: `$orderby` on
|
|
1482
|
+
a `$similarity` key, descending, then `$subsequence`.
|
|
1483
|
+
|
|
1484
|
+
```json
|
|
1485
|
+
{ "$for": { "m": "$.memories[*]" },
|
|
1486
|
+
"$orderby": [ { "$key": { "$similarity": ["$m.embedding", "$query"] },
|
|
1487
|
+
"$dir": "desc", "$empty": "least" } ],
|
|
1488
|
+
"$return": "$m.text" }
|
|
1489
|
+
```
|
|
1490
|
+
|
|
1491
|
+
```json
|
|
1492
|
+
{ "$subsequence": [ { "$for": { "m": "$.memories[*]" },
|
|
1493
|
+
"$orderby": [ { "$key": { "$similarity": ["$m.embedding", "$query"] },
|
|
1494
|
+
"$dir": "desc", "$empty": "least" },
|
|
1495
|
+
"$m.id" ],
|
|
1496
|
+
"$return": "$m" },
|
|
1497
|
+
0, 10 ] }
|
|
1498
|
+
```
|
|
1499
|
+
|
|
1500
|
+
The first orders every memory by how close it is to an external `$query`
|
|
1501
|
+
vector; the second wraps that ordering in a window and takes the first ten of
|
|
1502
|
+
it, which is the whole of "k nearest". The window goes **outside** the phrase,
|
|
1503
|
+
because `$subsequence` inside `$return` would trim each tuple's own result
|
|
1504
|
+
rather than the stream of tuples.
|
|
1505
|
+
|
|
1506
|
+
Two details in the second document are load-bearing. `"$empty": "least"` places
|
|
1507
|
+
an empty key **last** under `"desc"` (§6.6), which is where a row with no
|
|
1508
|
+
vector, or with one of the wrong width, belongs: present in the input, never in
|
|
1509
|
+
the top k, and never scored. And the second key, `"$m.id"`, breaks ties by row
|
|
1510
|
+
identity — the sort is stable, but stability is only about the *input* order,
|
|
1511
|
+
and two candidates at an identical similarity are exactly what a corpus of
|
|
1512
|
+
near-duplicates produces. An explicit tie-break is what makes the same document
|
|
1513
|
+
answer with the same rows every time it runs.
|
|
1514
|
+
|
|
1515
|
+
A filter is the same key in a `$where`, and it composes with the ordering:
|
|
1516
|
+
|
|
1517
|
+
```json
|
|
1518
|
+
{ "$for": { "m": "$.memories[*]" },
|
|
1519
|
+
"$where": { "$gt": [{ "$similarity": ["$m.embedding", "$query"] }, 0.8] },
|
|
1520
|
+
"$return": "$m.text" }
|
|
1521
|
+
```
|
|
1522
|
+
|
|
1523
|
+
— everything closer than a threshold, in the language's own clauses. Note that
|
|
1524
|
+
`$gt` over an empty key answers `false`, so an unvectored row drops out of a
|
|
1525
|
+
threshold filter for free.
|
|
1526
|
+
|
|
1527
|
+
|
|
1431
1528
|
## 9. Variables, scoping, and external parameters
|
|
1432
1529
|
|
|
1433
1530
|
1. Variables are introduced by `$for`, `$let`, `$at`, `$count`, `$groupby`
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@jarenjs/json",
|
|
3
3
|
"private": false,
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.46.4",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./src/index.js",
|
|
7
7
|
"types": "./dist/types/index.d.ts",
|
|
@@ -90,6 +90,6 @@
|
|
|
90
90
|
"prepack": "npm run build:types"
|
|
91
91
|
},
|
|
92
92
|
"dependencies": {
|
|
93
|
-
"@jarenjs/core": "^0.
|
|
93
|
+
"@jarenjs/core": "^0.46.4"
|
|
94
94
|
}
|
|
95
95
|
}
|
|
@@ -621,7 +621,11 @@
|
|
|
621
621
|
"$bbox",
|
|
622
622
|
"$area",
|
|
623
623
|
"$length",
|
|
624
|
-
"$centroid"
|
|
624
|
+
"$centroid",
|
|
625
|
+
"$geo-parse",
|
|
626
|
+
"$geo-text",
|
|
627
|
+
"$geohash-bounds",
|
|
628
|
+
"$geohash-neighbours"
|
|
625
629
|
]
|
|
626
630
|
},
|
|
627
631
|
"additionalProperties": {
|
|
@@ -661,6 +665,8 @@
|
|
|
661
665
|
"$distance",
|
|
662
666
|
"$within",
|
|
663
667
|
"$bbox-intersects",
|
|
668
|
+
"$geo-simplify",
|
|
669
|
+
"$similarity",
|
|
664
670
|
"$default"
|
|
665
671
|
]
|
|
666
672
|
},
|
|
@@ -621,7 +621,11 @@
|
|
|
621
621
|
"$bbox",
|
|
622
622
|
"$area",
|
|
623
623
|
"$length",
|
|
624
|
-
"$centroid"
|
|
624
|
+
"$centroid",
|
|
625
|
+
"$geo-parse",
|
|
626
|
+
"$geo-text",
|
|
627
|
+
"$geohash-bounds",
|
|
628
|
+
"$geohash-neighbours"
|
|
625
629
|
]
|
|
626
630
|
},
|
|
627
631
|
"additionalProperties": {
|
|
@@ -661,6 +665,8 @@
|
|
|
661
665
|
"$distance",
|
|
662
666
|
"$within",
|
|
663
667
|
"$bbox-intersects",
|
|
668
|
+
"$geo-simplify",
|
|
669
|
+
"$similarity",
|
|
664
670
|
"$default"
|
|
665
671
|
]
|
|
666
672
|
},
|
|
@@ -530,7 +530,11 @@
|
|
|
530
530
|
"$bbox",
|
|
531
531
|
"$area",
|
|
532
532
|
"$length",
|
|
533
|
-
"$centroid"
|
|
533
|
+
"$centroid",
|
|
534
|
+
"$geo-parse",
|
|
535
|
+
"$geo-text",
|
|
536
|
+
"$geohash-bounds",
|
|
537
|
+
"$geohash-neighbours"
|
|
534
538
|
]
|
|
535
539
|
},
|
|
536
540
|
"additionalProperties": {
|
|
@@ -570,6 +574,8 @@
|
|
|
570
574
|
"$distance",
|
|
571
575
|
"$within",
|
|
572
576
|
"$bbox-intersects",
|
|
577
|
+
"$geo-simplify",
|
|
578
|
+
"$similarity",
|
|
573
579
|
"$default"
|
|
574
580
|
]
|
|
575
581
|
},
|
|
@@ -289,7 +289,8 @@
|
|
|
289
289
|
"$year", "$month", "$day", "$hours", "$minutes", "$seconds",
|
|
290
290
|
"$offset", "$epoch", "$datetime",
|
|
291
291
|
"$week", "$week-year", "$quarter", "$weekday",
|
|
292
|
-
"$bbox", "$area", "$length", "$centroid"
|
|
292
|
+
"$bbox", "$area", "$length", "$centroid",
|
|
293
|
+
"$geo-parse", "$geo-text", "$geohash-bounds", "$geohash-neighbours"
|
|
293
294
|
]
|
|
294
295
|
},
|
|
295
296
|
"additionalProperties": { "$ref": "#/$defs/expression" }
|
|
@@ -306,7 +307,8 @@
|
|
|
306
307
|
"$contains", "$starts-with", "$ends-with", "$match", "$search",
|
|
307
308
|
"$index-of", "$range", "$get",
|
|
308
309
|
"$start-of", "$end-of", "$date-format",
|
|
309
|
-
"$distance", "$within", "$bbox-intersects", "$
|
|
310
|
+
"$distance", "$within", "$bbox-intersects", "$geo-simplify",
|
|
311
|
+
"$similarity", "$default"
|
|
310
312
|
]
|
|
311
313
|
},
|
|
312
314
|
"additionalProperties": {
|
package/src/query/normalize.js
CHANGED
|
@@ -417,6 +417,22 @@ const OPERATOR_ALIASES = {
|
|
|
417
417
|
$project: "the renderer — the language cannot make a projected coordinate at all, so a measurement can never land on one; measurement here is geodesic",
|
|
418
418
|
"$geo-project": "the renderer, as for '$project'",
|
|
419
419
|
$srid: null, $transform: null,
|
|
420
|
+
// vectors: the spellings a writer reaches for from a ranking library.
|
|
421
|
+
// The metric names all point at the one operator because the packed
|
|
422
|
+
// form is normalized and they then rank identically; `$knn` and its
|
|
423
|
+
// synonyms get the composition, because ordering and windowing are
|
|
424
|
+
// clauses this language already has and a second spelling of them
|
|
425
|
+
// would be the drift, not the convenience
|
|
426
|
+
$cosine: "$similarity", "$cosine-similarity": "$similarity",
|
|
427
|
+
"$cos-sim": "$similarity", "$dot-product": "$similarity",
|
|
428
|
+
$dot: "$similarity", "$inner-product": "$similarity",
|
|
429
|
+
"$vector-distance": "$similarity (higher is closer; there is no distance metric)",
|
|
430
|
+
"$l2-distance": "$similarity (higher is closer; there is no distance metric)",
|
|
431
|
+
$knn: "$orderby on a $similarity key with $dir 'desc', then $subsequence for the k",
|
|
432
|
+
$nearest: "$orderby on a $similarity key with $dir 'desc', then $subsequence for the k",
|
|
433
|
+
"$nearest-neighbours": "$orderby on a $similarity key, then $subsequence",
|
|
434
|
+
"$nearest-neighbors": "$orderby on a $similarity key, then $subsequence",
|
|
435
|
+
"$top-k": "$orderby then $subsequence", $embed: null, $normalize: null,
|
|
420
436
|
};
|
|
421
437
|
|
|
422
438
|
// JQ0002 for an unknown $-key, with a "did you mean" suggestion. A
|
package/src/query/operators.js
CHANGED
|
@@ -56,6 +56,7 @@ import {
|
|
|
56
56
|
fixedUnitMs,
|
|
57
57
|
compileDateFormat,
|
|
58
58
|
} from '@jarenjs/core/dates';
|
|
59
|
+
import { isVector, cosineSimilarity } from '@jarenjs/core/vector';
|
|
59
60
|
import {
|
|
60
61
|
isPosition,
|
|
61
62
|
bboxOf,
|
|
@@ -840,6 +841,41 @@ function geoUnaryEntry(measure, check = geoArg) {
|
|
|
840
841
|
|
|
841
842
|
//#endregion
|
|
842
843
|
|
|
844
|
+
//#region vector operators (section 8.15)
|
|
845
|
+
// One operator, one metric. A vector here is what a JSON document can
|
|
846
|
+
// hold — an array of numbers — and the arithmetic lives once, in
|
|
847
|
+
// `@jarenjs/core/vector`, where the ledger and the store read it too.
|
|
848
|
+
//
|
|
849
|
+
// The kernel and the language answer a malformed comparison
|
|
850
|
+
// differently, on purpose. A kernel scores 0 so that one bad vector
|
|
851
|
+
// among ten thousand loses a ranking sweep instead of killing it; the
|
|
852
|
+
// language answers the empty sequence, the same thing §8.14 answers for
|
|
853
|
+
// a measurement it cannot make, so a document never carries a score
|
|
854
|
+
// that was never computed. The gate between the two is `isVector`, the
|
|
855
|
+
// one shape guard the suite shares.
|
|
856
|
+
|
|
857
|
+
// A vector operand, split the way §8.14 splits geography: a value that
|
|
858
|
+
// is not an array of numbers is the wrong TYPE and raises, while an
|
|
859
|
+
// array of numbers that cannot be compared — empty, or carrying a
|
|
860
|
+
// component a computation made non-finite — is a DATA-level answer of
|
|
861
|
+
// empty, decided by the caller against the other operand's width.
|
|
862
|
+
function vectorArg(v, docPath) {
|
|
863
|
+
if (!Array.isArray(v)) {
|
|
864
|
+
throw runtimeError('JQ2001',
|
|
865
|
+
`expected a vector (an array of numbers), got ${describeItem(v)}`, docPath);
|
|
866
|
+
}
|
|
867
|
+
for (let i = 0; i < v.length; i++) {
|
|
868
|
+
if (typeof v[i] !== 'number') {
|
|
869
|
+
throw runtimeError('JQ2001',
|
|
870
|
+
`expected a vector (an array of numbers), got ${describeItem(v[i])} at index ${i}`,
|
|
871
|
+
docPath);
|
|
872
|
+
}
|
|
873
|
+
}
|
|
874
|
+
return v;
|
|
875
|
+
}
|
|
876
|
+
|
|
877
|
+
//#endregion
|
|
878
|
+
|
|
843
879
|
// the RFC 3339 type tests, shaped like the section 8.10 $is-* family:
|
|
844
880
|
// one item of the right lexical form, never an error
|
|
845
881
|
function dateTestEntry(test) {
|
|
@@ -1748,6 +1784,39 @@ export const OPERATORS = Object.freeze({
|
|
|
1748
1784
|
|
|
1749
1785
|
//#endregion
|
|
1750
1786
|
|
|
1787
|
+
//#region section 8.15 - vectors
|
|
1788
|
+
|
|
1789
|
+
'$similarity': { // cosine similarity of two vectors, higher-is-better
|
|
1790
|
+
params: ARGS_2,
|
|
1791
|
+
// OPTIONAL: two present operands can still have no comparison — a
|
|
1792
|
+
// width mismatch, an empty vector, a component a computation made
|
|
1793
|
+
// non-finite. Declaring exactly-one would let the internal empty
|
|
1794
|
+
// marker escape into an array or object constructor.
|
|
1795
|
+
result: RESULT_OPT,
|
|
1796
|
+
compile: (gets, args) => {
|
|
1797
|
+
const aGet = gets[0];
|
|
1798
|
+
const aPath = args[0].docPath;
|
|
1799
|
+
const bGet = gets[1];
|
|
1800
|
+
const bPath = args[1].docPath;
|
|
1801
|
+
return (f) => {
|
|
1802
|
+
const a = aGet(f);
|
|
1803
|
+
const b = bGet(f);
|
|
1804
|
+
if (a === EMPTY || b === EMPTY)
|
|
1805
|
+
return EMPTY;
|
|
1806
|
+
const va = vectorArg(a, aPath);
|
|
1807
|
+
const vb = vectorArg(b, bPath);
|
|
1808
|
+
// the width is the second operand's obligation, so the same
|
|
1809
|
+
// guard answers both "is this comparable at all" and "is it
|
|
1810
|
+
// comparable to THAT" — nothing here pads or truncates
|
|
1811
|
+
if (!isVector(va) || !isVector(vb, va.length))
|
|
1812
|
+
return EMPTY;
|
|
1813
|
+
return cosineSimilarity(va, vb);
|
|
1814
|
+
};
|
|
1815
|
+
},
|
|
1816
|
+
},
|
|
1817
|
+
|
|
1818
|
+
//#endregion
|
|
1819
|
+
|
|
1751
1820
|
//#region section 8.13 - dates and times, continued
|
|
1752
1821
|
|
|
1753
1822
|
'$date-add': dateShiftEntry(1),
|