@xcitedbs/client 0.2.14 → 0.2.15

package/unquery-ai-guide.md

@@ -0,0 +1,1139 @@
+# Unquery — AI guide for writing queries
+
+> **Read this entire document before generating any Unquery query.** It is the single, self-contained reference an AI model needs (alongside the user's data schema) to write correct, idiomatic Unquery. For parser-level syntax detail and static-check rules, the companion document is [`docs/unquery-grammar.md`](./unquery-grammar.md) (also exposed as MCP resource `xcitedb://unquery-grammar`).
+
+Unquery is a declarative language for querying and transforming structured documents (JSON and XML). It is the query language of XCiteDB and the `unq` command-line tool. Every query is itself a JSON document. The result is also JSON.
+
+---
+
+## 0. AI preamble — must-read before writing a query
+
+Internalize these eight points first. They are the single largest source of mistakes in machine-generated Unquery.
+
+1. **A query *is* JSON.** Wrap your output in the same JSON shape you want back. Strings inside the JSON are Unquery **expressions**, not literal text.
+2. **Result mirrors template.** Objects produce objects, arrays produce arrays, strings produce single values. If the user wants a *list*, you need a *JSON array* in the query. If they want an *object*, use a JSON object. To repeat per-document, wrap in `[...]`.
+3. **String constants need single quotes inside the expression.** `"FirstName"` reads the field `FirstName`. To return the literal text `FirstName`, write `"'FirstName'"`. Double quotes are JSON syntax, single quotes are Unquery string delimiters.
+4. **Condition OR is single `|`. Context-OR is `||`.** Inside a `?`, `#if`, or constraint, write `a=1 | b=2`. Inside a key (between context modifiers), write `:.||Family[]`. Mixing them is a parse error.
+5. **`<-` does not exist as an operator.** It is lexed as `<` then unary `-`. To compare to a negative literal, use `< -5` or parentheses.
+6. **`..` is the parent step token; you almost always want `../`.** `../Field` means "go up, then read Field". `..Field` is a parse error.
+7. **Directive keys (`#if`, `#var`, `#assign`, `#exists`, `#notexists`, `#return`, `#returnif`, `#func`) get parsed differently from regular keys.** `#if` value is a *condition* (no `?`, no `@sort`); the others are full values. Constraints attach to the value of directives, but to the *object* of regular keys.
+8. **Aggregates (`$count`, `$sum`, `$avg`, `$min`, `$max`, `$prev`) can only be compared to literals**, not to other expressions. `"$avg(Salary)>100000"` is OK. `"$avg(Salary)>OtherField"` is a parse error.
+
+### Decision shortcuts
+
+| Want… | Use |
+|---|---|
+| Read field `X` | `"X"` |
+| Constant string `'X'` in result | `"'X'"` |
+| One value per document | bare object/string at root |
+| One result accumulating across documents | bare object at root |
+| Array of one value per document | `["expr"]` |
+| Array of objects per document | `[{...}]` |
+| Iterate inside a single document | context modifier `:Path[]` or `:{}` |
+| Filter rows | predicate `?cond` or constraint `Field op val` |
+| Filter the whole record | `#if` directive |
+| Group by | dynamic key `$(GroupExpr)` + array value |
+| Drop the wrapper object | `#return` |
+| Pick one of several shapes | `#returnif` chain |
+| Recursive transform of any JSON | `#func` returning `#returnif` per type |
+
+---
+
+## 1. Mental model
+
+A query is a **template**. Evaluation walks the template top-down, possibly many times (once per document, plus extra passes when iterating with context modifiers, dynamic keys, or `||`).
+
+- **Object in template → object in result.** Each key-value pair is evaluated in order. A key may produce zero, one, or many output keys.
+- **Array in template → array in result.** Each evaluation pass *appends* a new element. Arrays are how you "broadcast" across documents or iterations.
+- **String in template → single value.** A string is parsed as one Unquery expression and returns one JSON value (which can itself be a number, string, bool, object, or array).
+- **Numbers / booleans / null in template → constants.** Literal JSON values are passed through unchanged.
+
+### The "evaluate once vs. per pass" rule
+
+Outside an array, a value is set once and **does not update** on later passes. So `{"name":"FirstName"}` over many docs returns just the first `FirstName`. Wrap in `[]` to get one per pass:
+
+```json
+[{"name":"FirstName"}]
+```
+
+Aggregates are the exception: they update on every visit even outside arrays. `{"total":"$sum(Salary)"}` does compute the sum.
+
+### When does iteration happen?
+
+- One pass per document in the input set (always).
+- Extra passes from `:[]` (array traversal), `:{}` (object-key traversal), `**` (recursive descent), `||` (context-or), `->$children`/`->$descendants`/`->$ancestors` (XML), and from dynamic keys producing multiple values.
+
+### What changes the *current path*?
+
+- Path expressions in expressions (`Field.Sub`, `[0]`, `../`, `/`, `<<`).
+- Context modifiers in keys (`:Path`, `:[]`, `:{}`, `:**`, `->...`).
+
+---
+
+## 2. Path expressions
+
+A path selects a value from the current context.
+
+| Form | Meaning | Example |
+|---|---|---|
+| `.` | Current value (initially the document root) | `["."]` returns each document |
+| `Field` | Read named field | `"FirstName"` |
+| `Field.Sub` | Nested field | `"Address.City"` |
+| `` `field with spaces` `` | Backtick-quote a non-identifier name | `` "`First Name`" `` |
+| `[n]` | Array index (0-based) | `"Dependants[0].FirstName"` |
+| `[expr]` | Computed index | `"Field1[$index+1]"` |
+| `[]` | Whole array (projection over all elements) | `"Array1[].Field1"` |
+| `/Field` | Absolute from document root | `"/employees.$(.)"` |
+| `../Field` | Up one path level (skips array indices) | `"../DBInstanceIdentifier"` |
+| `<<Field` | Read same field in *previous* document context | `"<<Field1"` after a `->$file(...)` |
+| `$(expr)` | Use the value of `expr` as the *next* path segment / dynamic key | `"$var(states).$(Address.State)"` |
+
+Notes:
+
+- `Field.$x` is **not** a continued path. After `.`, if the next token starts with `$`, the path stops and you get the field `Field` followed by an expression `$x`. To use a computed subfield, write `.$(x)` explicitly.
+- Subfield works on any expression, not just paths: `"$var(x).Subfield1"`.
+- `[]` on a value that is *not* an array silently treats it as a one-element array. This is convenient and almost never wrong.
+- The `$()` evaluate function is *redundant* before a function token: write `$index`, not `$($index)`.
+
+---
+
+## 3. Expressions and operators
+
+### 3.1 Arithmetic and string
+
+| Op | Meaning | Notes |
+|---|---|---|
+| `+` | Add (numbers) or concatenate (strings) | Mixed types: behavior is engine-defined; cast first if unsure |
+| `-` | Subtract | Unary minus only on numeric literal: `-5` |
+| `*` | Multiply | |
+| `/` | Divide | Integer/integer = integer (truncation). Force float: `14/$float(10)` → `1.4` |
+| `mod` | Modulo | Word, not `%` |
+
+Precedence: `*`, `/`, `mod` > `+`, `-`. Use parentheses freely: `(Field1+2)*5`.
+
+### 3.2 String literals
+
+Single quotes inside the expression: `'hello'`. Or escape double quotes: `"\"hello\""`. Backticks (`` `...` ``) are for **field names**, not string constants — never use them for literals.
+
+### 3.3 Type-cast functions
+
+`$string(x)`, `$number(x)`, `$int(x)`, `$float(x)`, `$bool(x)`. Common uses:
+
+```json
+"$int('5') + 1"            // 6
+"14/$float(10)"            // 1.4   (force float division)
+"$sum($number(value))"     // sum of strings that look like numbers
+"$number(.)"               // smart-cast to int/float depending on content
+```
+
+### 3.4 Identifiers and variables
+
+- `$var(name)` reads a variable declared via `#var` or `#assign`.
+- `%name` is shorthand for `$var(name)` and parses identically.
+
+---
+
+## 4. Conditions
+
+Conditions appear in four places. Same grammar everywhere except `#if` cannot have `?`/`@sort`.
+
+| Place | Syntax | Effect |
+|---|---|---|
+| **Predicate** on a value | `"expr ? cond"` | Skip emitting the value when false. Aggregates skip the update. |
+| **Constraint** (object field only) | `"expr op rhs"` (no `?`) | If false, the *whole object* is skipped. |
+| **`#if`** directive | `{"#if":"cond", ...}` | If false, the whole object is skipped. Same effect as constraint, but explicit. |
+| **`$if(cond, then, else)`** | function call | Inline ternary. |
+| **Predicate after a context modifier** | `"key:Path[]?cond"` | Filters which traversal items run the sub-template. |
+
+### 4.1 Comparators
+
+`=`, `!=`, `<`, `>`, `<=`, `>=`. (Single `=`, not `==`.)
+
+### 4.2 Boolean glue
+
+- AND: `&` (higher precedence)
+- OR: `|` (lower precedence) — **single pipe**, never `||` inside a condition
+- NOT: `!cond`
+- Parentheses: `(...)`
+
+```text
+x=5 | (y>7 & x=z)
+Field1 contains 'Developer' & Field2 matches 'A.*b'
+```
+
+### 4.3 Existence
+
+Postfix `!` after a path: `"Field1.Field2!"` is true iff `Field2` exists in `Field1`. Works as a constraint too:
+
+```json
+{ "key1":"Field1!", "key2":"value_expr" }
+```
+
+### 4.4 Type tests (no RHS)
+
+`is_array`, `is_object`, `is_literal`, `is_string`, `is_number`, `is_int`, `is_float`, `is_bool`. Form: `"path is_int"`. Example — keep only ints from a mixed array:
+
+```json
+{ "numbers:[]": [". is_int"] }
+```
+
+### 4.5 String tests
+
+`contains`, `starts_with`, `ends_with`, `matches` (regex). Example:
+
+```text
+Title contains 'Developer'
+folder starts_with 'User/Admin'
+Name matches 'A.*b'
+```
+
+### 4.6 Set tests
+
+`in` and `not_in`. RHS must be array-shaped:
+
+```text
+Color in ['red','green','blue']
+Status not_in BadStatuses[]
+```
+
+### 4.7 Predicate vs. constraint — subtle but important
+
+Inside an **object field**, these differ:
+
+```json
+{ "key1": "value1?x!=1", "key2": "value2" }   // predicate: skips key1 only
+{ "key1": "value1 != 1", "key2": "value2" }   // constraint: skips the WHOLE object
+```
+
+The constraint form is equivalent to writing `{"#if":"value1!=1", ...}` with the same fields.
+
+### 4.8 Aggregate-in-condition rule
+
+You may compare an aggregate to a **literal**:
+
+```text
+$count > 1
+$avg(Age) > 40
+```
+
+You may **not** compare two non-literal sides where one is an aggregate. The parser rejects `"$avg(Age) > Threshold"`.
+
+---
+
+## 5. Object keys
+
+Keys are not just constant strings. The first character (or first token) decides what kind of key it is.
+
+### 5.1 Constant keys
+
+Alphanumeric/underscore identifiers, or backtick-quoted strings. The output key has the same name.
+
+```json
+{ "FirstName": "FirstName" }
+```
+
+### 5.2 Dynamic keys — `$(expr)`
+
+Evaluate `expr` and use the result as the key name. Each evaluation can add a new key.
+
+```json
+{ "$(LastName)": "." }                 // dictionary by last name
+{ "$(LastName+' '+FirstName)": "." }   // composite key
+```
+
+`$(...)` is **redundant** when the expression is itself a `$function` token. Use `"$index"` not `"$($index)"`. Use `"$key"` not `"$($key)"`.
+
+### 5.3 Group-by pattern
+
+A dynamic key with an **array** value collects all matching values per group:
+
+```json
+{ "$(bin)": ["value"] }
+```
+
+Equivalent to SQL `GROUP BY bin`. Combine with aggregate values for grouped stats:
+
+```json
+{ "$(Title)": "$avg(Salary)" }
+```
+
+### 5.4 Copy-all keys — `{}` and `{'regex'}`
+
+`{}` evaluates to *every* key in the queried object. `{'regex'}` evaluates to keys matching the regex.
+
+```json
+{ "{}": "value" }                  // every key gets the constant 'value' (rarely useful)
+{ "{}:": "." }                     // copy every key → its value (very useful)
+{ "{'A.*B'}:": "." }               // copy only keys matching the regex
+{ "{}:": ".?$key!='ID'" }          // copy all except ID (predicate filter)
+```
+
+The trailing `:` after `{}` is a context modifier ("for each key, switch context to that key"). Without `:`, `{}` is just the empty-key spec.
+
+### 5.5 Key utility functions
+
+- `$key` — name of the current key in the queried object (during traversal).
+- `$reskey` — name of the *result* key (useful in `#if` / dynamic situations).
+- `$path` — the full current path as a string.
+- `$index` — the last numeric index in the path.
+
+---
+
+## 6. Context modifiers
+
+Context modifiers appear **in the key**, after the key name, and scope the value (object/array/string under that key). Two families:
+
+- **Path modifiers** start with `:` — they change the path within the current document.
+- **Identifier/document modifiers** start with `->` (or `:->`) — they switch document, XML node, branch, or date.
+
+A modifier may be followed by `?cond` (predicate). Modifiers chain freely.
+
+### 6.1 Path modifiers
+
+| Form | Meaning |
+|---|---|
+| `:Path` | Switch context to `Path` |
+| `:` (nothing after) | Repeat the key name as the path: `"Field1:"` ≡ `"Field1:Field1"` |
+| `:[]` | Iterate over each array element |
+| `:[n]` | Switch to array index `n` |
+| `:{}` | Iterate over each object field |
+| `:{'regex'}` | Iterate over fields matching regex |
+| `:**` | Recursive descent (every path under current) |
+| `:.` | Stay at current path (rarely needed) |
+
+```json
+{ "Childen:Dependants[]?Relation='Child'": ["FirstName+' '+LastName"] }
+{ "result:Customers[]?Balance>100000:Accounts[]": ["accountNumber"] }
+{ "result:{}": ["."] }
+{ "#return:**": ["$key@unique_ascending"] }
+```
+
+### 6.2 The `:Field` shorthand (very common)
+
+`"FirstName:"` is identical to `"FirstName:FirstName"`. Use it whenever the result key matches the source field:
+
+```json
+[{ "FirstName:":".", "LastName:":".", "NumOfChildren:Dependants[]":"$count?Relation='Child'" }]
+```
+
+### 6.3 Identifier / document modifiers — `->`
+
+Switch the current "document". After a `->`, the path resets to the new document's root.
+
+| Form | Meaning |
+|---|---|
+| `->$file('name.json')` | Read another JSON file as new context |
+| `->$var(name)` | Switch to JSON stored in variable `name` |
+| `->$date('YYYY-MM-DD')` | Switch temporal context (XCiteDB only) |
+| `->$branch('name')` | Switch branch / workspace (XCiteDB only) |
+| `->$self`, `->$parent`, `->$children`, `->$descendants`, `->$descendants_and_self`, `->$all` | **Structural shortcuts** — parser lowers each to an identifier expression AST (`TQIdExpr`) with the same runtime as the path form in the right column (see §6.3.3). |
+| Path equivalents | `->$self` → `->./`; `->$parent` → `->../`; `->$children` → `->./[]`; `->$descendants` → `->./[]/**` (descendants only); `->$descendants_and_self` → `->./**`; `->$all` → root children + recursive descent under `/` (every identifier under `/`). |
+| `->$ancestors`, `->$ancestors_and_self` | Named-only structural shortcuts (same lowering idea; no single-segment path spellings). |
+| `->"/abs/path"` | Switch to the named XML identifier (also an `identifierExpression` with one quoted segment) |
+| `->/abs/...`, `->./...`, `->../...`, `->[]/...`, `->**/...`, `->seg/sub/...`, `->$(expr)/...`, `->/path[]`, `->/path/**` | **Identifier expression** — evaluates to **zero or more** identifiers (see §6.3.1, §6.3.3). |
+| `->$each(expr)` | Evaluate `expr` as JSON; if string or array of strings, iterate each as an identifier (must not be followed by `/` in the same modifier) |
+| `->Field` *(deprecated)* | JSON-field lookup: string or array of identifier strings (runtime deprecation warning; migrate to `->$each(Field)` or an `identifierExpression`) |
+| `->$(expr)` *(deprecated)* | Same lookup using `expr`’s string value as the field name (deprecation warning) |
+
+**Decision:** iterate **direct children** of a path → `->/path/to/node[]` or `->$children` from that node (after switching context with `->/path/to/node` if needed).
+
+When chaining identifier switches, the colon between them is optional: `:->$parent->$parent`.
+
+#### 6.3.1 Identifier expressions
+
+Use these when the path after `->` should be composed as an XCiteDB identifier (always normalized to a leading `/` at evaluation when needed), instead of reading a JSON field name. Evaluation can produce **multiple** identifiers; the sub-template runs once per identifier (same as `->$children` / `->$each` iteration).
+
+```json
+{ "balance:->/tables/bank/accounts/$(accountID)": "amount" }
+{ "holder:->../accounts/$(accountID)": "name" }
+{ "msg:->../../audit/$(customerID)/$(date)": "text" }
+{ "title:->$each(relatedSectionIds)": "heading" }
+```
+
+- **`/`** — absolute from project root.
+- **`./`** — continue under the **current** identifier.
+- **`../`** — go to parent identifier (repeat in the anchor only; more `../` in one anchor climbs multiple levels).
+- **Mid-slash paths** like `->customers/$(id)` — relative to current identifier (first segment is under current id).
+- **`[]`** — after the path built so far, expand to every **direct child** identifier (LMDB / `id_hier` when available).
+- **`**`** — after the path built so far, expand to that identifier **plus all descendants** (prefix scan).
+- **`[N]`** and non-empty `[…]` brackets are not allowed inside these paths (parse error); use `$(expr)` for dynamic segments.
+
+#### 6.3.3 Set-valued identifier expressions
+
+- **`->/registry/foo[]`** — run the remainder of the query once per direct child of `/registry/foo`.
+- **`->/registry/foo/**`** — run for `/registry/foo` and every descendant identifier under it.
+- **Anchor-less** `->[]/…` and `->**/…` are sugar for `->./[]/…` and `->./**/…` (start at the current identifier).
+- **Ordering** of produced ids follows LMDB / index iteration (stable for a given DB snapshot); treat as implementation-defined like JSON-path `[]` iteration.
+- **Predicates:** `?cond` after the modifier applies **per produced identifier** (the modifier is wrapped in `TQValueWithCond` as usual).
+
+Dynamic `$(expr)` / `%var` segments in Phase 2 are still evaluated **once** for the whole step (not per frontier element); per-element evaluation is a possible future extension.
+
+#### 6.3.4 Grammar reference
+
+Full EBNF and static rules: **`docs/unquery-grammar.md` §5.1** (identifier expressions) and **§8.2** (arrow dispatch).
+
+#### 6.3.2 Migrating from deprecated arrow forms
+
+| Before | After (Phase 1) |
+|--------|------------------|
+| `->relatedIds` where `relatedIds` is `["/a/1","/a/2"]` | `->$each(relatedIds)` |
+| `->relatedIds` where `relatedIds` is a single string path | `->$each(relatedIds)` (works for one string too) |
+| You meant a **child** path under the current identifier | `->./childName/...` or an absolute `->/prefix/...` |
+| `->$(nameField)` with no `/` (double indirection via JSON) | Keep until Phase 2; prefer `->$each(...)` if you only need iteration semantics |
+
+### 6.4 Chaining
+
+Write modifiers back-to-back. Each is a predicate-eligible step.
+
+```json
+{ "result:[]:{}":["."] }                                     // array → each element → each field
+{ "result->$file('employees.json'):Employees[]":["FirstName"] }
+```
+
+### 6.5 Context-OR — `||`
+
+Try **multiple** context paths and union the results. Each branch starts from the same point.
+
+```json
+{ "names:.||Family[]": ["FirstName"] }
+{ "names:.||Dependants[]": ["FirstName+' '+LastName"] }
+```
+
+`||` is a **lexer-level merge** of two `|` characters. It is recognized **only** in context-modifier position. Do **not** use `||` in a condition — that's a parse error; use single `|` for boolean OR.
+
+### 6.6 Context predicates
+
+Add `?cond` after each modifier step (not just at the end):
+
+```json
+{ "result:Customers[]?Balance>100000:Accounts[]":["accountNumber"] }
+{ "Dependants:Dependants[]?$index<2": ["FirstName+' '+LastName"] }
+```
+
+### 6.7 The `<<` previous-document operator
+
+After switching to a different document with `->`, `<<Field` reads `Field` from the *original* document. Useful for joins:
+
+```json
+{
+  "result->$file('another.unq')": {
+    "key1":"Field1",
+    "key2":"<<Field1"
+  }
+}
+```
+
+---
+
+## 7. Sorting
+
+Append a sort specifier at the *end* of a value string (after the expression and any predicate, before nothing). Four specifiers:
+
+- `@ascending`
+- `@descending`
+- `@unique_ascending` (also dedupes)
+- `@unique_descending` (also dedupes)
+
+```json
+["FirstName@ascending"]
+["FirstName@unique_ascending"]
+```
+
+### 7.1 Multi-key sort
+
+For arrays of objects, every sortable field carries its own specifier and a priority `(n)` (lower number = higher priority):
+
+```json
+[{ "firstname":"FirstName@descending(2)", "lastname":"LastName@ascending(1)" }]
+```
+
+Sorts by `lastname` ascending first, then `firstname` descending.
+
+### 7.2 Pitfall — multiple items in one array
+
+`["FirstName@ascending","Lastname"]` has *undefined* sort order. Only one element should drive the sort, or every element must use the same specifier:
+
+```json
+["FirstName@ascending","Lastname@ascending"]
+```
+
+### 7.3 `@sort` placement
+
+`@sort` ends parsing of the value string. It cannot follow a constraint. Put predicates and constraints *before* the `@`:
+
+```text
+"Salary?Title='Dev'@descending"   // OK: expr ? cond @sort
+"Salary>0 @descending"            // OK: expr constraint @sort
+```
+
+---
+
+## 8. Directives
+
+Directives are object keys starting with `#`. They are evaluated in the order they appear inside the object.
+
+### 8.1 `#var` — declare a variable
+
+```json
+{ "#var x": "1000", "value": "$var(x)" }     // 1000
+{ "#var x": "'Some string'", "v": "%x" }      // %x ≡ $var(x)
+```
+
+A variable can hold any JSON value — including the result of running a sub-template:
+
+```json
+{
+  "#var dic:Employees[]": { "$(EmployeeId)": "." },
+  "Employee1": "$var(dic).1001"
+}
+```
+
+Variables are **scoped to the enclosing object**. Inner `#var x` shadows outer `#var x` for the duration of that object only.
+
+### 8.2 `#assign` — update an existing variable
+
+```json
+{
+  "#var x": "1",
+  "obj": { "#assign x": "2" },
+  "x_value": "$var(x)"      // 2
+}
+```
+
+If the variable was never declared, `#assign` behaves like `#var`.
+
+### 8.3 `#if` — skip the object when condition is false
+
+The value of `#if` is parsed as a **condition** — no `?`, no `@sort`, no leading expression-then-predicate.
+
+```json
+[{
+  "#if": "Title!=CEO",
+  "FirstName:": ".",
+  "LastName:": "."
+}]
+```
+
+`#if` accepts a context modifier:
+
+```json
+[{
+  "#if:Dependants[]?Relation='Child'": "$count>1",
+  "FirstName:": ".",
+  "LastName:": "."
+}]
+```
+
+### 8.4 `#exists` / `#notexists` — value-non-empty / value-empty test
+
+Used like `#if`, but the *value* is a normal Unquery template; the directive tests whether the template produced any non-empty result.
+
+```json
+[{
+  "#exists:Employees[]": ["Salary>100000"],
+  "company": "CompanyName"
+}]
+[{
+  "#notexists:Employees[]": ["Salary<30000"],
+  "company": "CompanyName"
+}]
+```
+
+### 8.5 `#return` — flatten the wrapper object
+
+A `#return` makes the *value* of the directive be the entire object's result, dropping any other fields and the wrapping object itself. This breaks Principle 2 deliberately, to escape the "you must wrap in an object to use a context modifier" limitation.
+
+```json
+{ "result:Employees[]": ["FirstName"] }    // {"result":[ ...names... ]}
+{ "#return:Employees[]": ["FirstName"] }   // [ ...names... ]
+```
+
+Rules:
+
+- The first `#return` in an object wins.
+- Other fields in the same object are dropped.
+
+### 8.6 `#returnif` — first non-empty wins
+
+Like `#return`, but only effective when the value is non-null/non-empty. Multiple `#returnif`s form an ordered fallback chain.
+
+```json
+{
+  "#returnif:Tags[]?Key='Name'": "Value",
+  "#return": "'N/A'"
+}
+```
+
+This pattern is the canonical "default value" idiom: try a tagged value; if missing, fall back to the literal string.
+
+`#returnif` is the engine of polymorphic / recursive transforms (see §10 cookbook).
+
+### 8.7 `#func` — declare a (possibly recursive) function
+
+```json
+{ "#func fullname": "FirstName+' '+LastName",
+  "names:Employees[]": ["$fullname"] }
+
+{ "#func fullname(x,y)": "$var(x)+' '+$var(y)",
+  "names:Employees[]": ["$fullname(FirstName, LastName)"] }
+```
+
+Rules:
+
+- Function name and arity are registered in the symbol table when the `#func` key is parsed. Every call `$name(...)` must match that arity exactly.
+- A function body is a full Unquery template subtree (string, object, or array).
+- Functions can be recursive — call themselves from inside their own body. This is how generic JSON transforms are written (§10).
+- `$call(name)` invokes a zero-arg function by string name (rare; usually just `$name`).
+
+---
+
+## 9. Built-in functions and aggregates
+
+### 9.1 Function reference table
+
+| Function | Arity | Returns |
+|---|---|---|
+| `$()` (evaluate) | 1 | Value of the inner expression treated as a path/dynamic key |
+| `$if(c,a,b)` | 3 | `a` if `c` else `b` |
+| `$call(name)` | 1 (ident) | Result of `#func name` (zero-arg) |
+| `$var(name)` / `%name` | 1 (ident) | Variable value |
+| `$file(expr)` | 1 | Parsed JSON from file at `expr` |
+| `$csv(file [,delim] [,hdr])` | 1–3 | Array of objects from a CSV file (delim default `,`, headers default `true`) |
+| `$env(expr)` | 1 | Environment variable named by `expr` |
+| `$now` | 0 | Current Unix epoch (resolved at parse time) |
+| `$path` | 0 | Current path as string |
+| `$index` | 0 | Last numeric index in the current path |
+| `$key` | 0 | Last key in the current path |
+| `$reskey` | 0 | Result key being emitted |
+| `$filename` | 0 | Current source filename |
+| `$identifier` | 0 | Full XML identifier (XCiteDB) |
+| `$identifier(n)` | 1 (int) | Identifier path truncated before the n-th `/` |
+| `$lower(expr)` / `$upper(expr)` | 1 | Case-changed string |
+| `$length(expr)` | 1 | String length |
+| `$size(expr)` | 1 | Array size |
+| `$substr(s, start [,len])` | 2–3 | Substring |
+| `$find(s, t)` | 2 | Array of indices where `t` occurs in `s` |
+| `$ifind(s, t)` | 2 | Like `$find`, case-insensitive |
+| `$replace(s, from, to [,all?])` | 3–4 | String with replacements |
+| `$split(s, delim)` | 2 | Array of substrings |
+| `$join(arr [,delim])` | 1–2 | Joined string (delim default empty) |
+| `$to_time(s, fmt)` | 2 | Unix epoch from `strptime`-formatted string |
+| `$time_to_str(t, fmt)` | 2 | `strftime`-formatted string |
+| `$D"..."` | literal | Date literal → epoch via `stringToTime` |
+| `$string` `$number` `$int` `$float` `$bool` | 1 | Type cast |
+| `$node_date` | 0 | XML node revision time (epoch) |
+| `$data_date [(expr)]` | 0–1 | Metadata field last-change time (epoch) |
+| `$xml` / `$xml_no_children` | 0 | Current XML node as string (with/without children) |
+| `$node` | 0 | Tag name of the root element of the current XML node |
+| `$attr("name")` | 1 (quoted) | XML attribute value |
+| `$child("name")` | 1 (quoted) | Direct XML child element as XML |
+| `$xpath("expr")` / `$lxpath("expr")` | 1 (quoted) | First XPath match (leaf-style for `lxpath`) |
+| `$text(expr)` | 1 | Concatenated text content of an XML node |
+| `$in_filter("name")` | 1 (quoted) | True if current XML element belongs to the named filter group |
+
+Notes on quoted-string args: `$attr`, `$child`, `$xpath`, `$lxpath`, `$in_filter` require a **string literal in single or double quotes**, not an expression.
+
+### 9.2 Aggregate functions
+
+Aggregates **update on every visit** to their position. They live "outside" the per-pass value rule.
+
+| Aggregate | Arity | Notes |
+|---|---|---|
+| `$count` | 0 | Number of visits (use `?cond` to count selectively) |
+| `$sum(expr)` | 1 | Sum |
+| `$avg(expr)` | 1 | Average |
+| `$min(expr)` | 1 | Minimum |
+| `$max(expr)` | 1 | Maximum |
+| `$prev(default)` | 1 | First visit returns `default`; subsequent visits return the value of the enclosing expression on the previous visit |
+
+`$prev` lets you build custom reductions:
+
+```text
+"$prev('') + Text"    // concatenate Text from every document
+"$prev(0) + 1"        // count via reduction
+```
+
+### 9.3 Aggregates with arrays / context modifiers
+
+- Aggregate at the *top level* gives a single overall value.
+- Aggregate *inside an array element with no context modifier* gives one update per pass — usually just an array of single values, not what you want.
+- Aggregate inside a context-traversal **does** what you expect — one accumulator per outer pass:
+
+```json
+[{
+  "name": "FullName",
+  "avgFamilyAge:Family[]": "$avg(Age)"
+}]
+```
+
+### 9.4 Aggregates with predicates
+
+A `?cond` after an aggregate **skips the update** when false:
+
+```text
+"$avg(Age) ? Age >= 18"
+"$count ? Salary > 200000"
+```
+
+### 9.5 Aggregates in conditions
+
+Allowed only against literals (see §0.8 and §4.8):
+
+```json
+[{
+  "name":"FullName",
+  "avgFamilyAge:Family[]": "$avg(Age) > 40"
+}]
+```
+
+---
+
+## 10. Recipe cookbook
+
+Each recipe shows a small input-shape sketch, the Unquery template, and (where useful) the equivalent jq one-liner for cross-reference.
+
+### 10.1 Pick one field per document → array
+
+```json
+["FirstName"]
+```
+
+Sorted, unique:
+
+```json
+["FirstName@unique_ascending"]
+```
+
+### 10.2 Multiple fields per document → array of objects
+
+```json
+[{
+  "fullname": "FirstName+' '+LastName",
+  "title": "Title",
+  "city": "Address.City"
+}]
+```
+
+### 10.3 Filter rows by predicate
+
+```json
+["LastName?Salary>200000"]
+```
+
+Filter by string contains (constraint form):
+
+```json
+[{
+  "FirstName": "FirstName",
+  "LastName": "LastName",
+  "Title": "Title contains 'Developer'"
+}]
+```
+
+Filter by `#if`:
+
+```json
+[{
+  "#if": "$size(Dependants)>=3",
+  "FirstName": "FirstName",
+  "LastName": "LastName"
+}]
+```
+
+### 10.4 Filter array of objects by inner field — return field
+
+`jq '.[] | select(.location=="Stockholm") | .name'`
+
+```json
+{ "#return:{}": ["name?location='Stockholm'"] }
+```
+
+Or using `[]` traversal directly (when input is an array of objects):
+
+```json
+{ "#return:[]": ["name?location='Stockholm'"] }
+```
+
+### 10.5 Recursive descent — collect every value under a key name
+
+`jq '..|.foo? // empty'`
+
+```json
+{ "#return:**?$key='foo'": "." }
+```
+
+### 10.6 Filter array, return only selected field
+
+`jq '. - map(select(.Names[] | contains("data"))) | .[].Id'`
+
+```json
+{
+  "#return:[]": [{
+    "#notexists:Names[]": ". contains 'data'",
+    "#return": "Id"
+  }]
+}
+```
+
+### 10.7 Multi-shape result with `#returnif` chain
+
+```json
+[{
+  "#returnif:StatusInfos[]?StatusType='read replication'": {
+    "RepStatus": "Status",
+    "RepStatusType": "StatusType",
+    "ReadReplicaSourceDBInstanceIdentifier": "../ReadReplicaSourceDBInstanceIdentifier",
+    "DBInstanceIdentifier": "../DBInstanceIdentifier"
+  },
+  "#return": "."
+}]
+```
+
+If any `StatusInfo` matches, return the projection; otherwise return the whole object.
+
+### 10.8 Default value via `#returnif` + `#return`
+
+```json
+{
+  "#returnif:Tags[]?Key='Name'": "Value",
+  "#return": "'N/A'"
+}
+```
+
+### 10.9 Drop a column from every record
+
+`jq '[.[] | del(.timestamp)]'`
+
+```json
+{
+  "#return:[]": [{
+    "{}:": ".?$key!='timestamp'"
+  }]
+}
+```
+
+### 10.10 Single-element array trick — keep only `[0]` from a sub-array
+
+```json
+{
+  "blah0:": ".",
+  "Array:Array[]?$index=0": ["."]
+}
+```
+
+### 10.11 Boolean-OR predicate inside array element
+
+`jq '.theList | map(select(.id==2 or .id==4))'`
+
+```json
+{ "#return:theList[]": [".?id=2 | id=4"] }
+```
+
+### 10.12 Build a lookup table
+
+`jq 'map({(.name):.id}) | add'`
+
+```json
+{ "#return:{}": { "$(name)": "id" } }
+```
+
+### 10.13 Group-by + sum across nested arrays
+
+`reduce .[] as $o (null; . + ($o | f))` style.
+
+```json
+{
+  "#return:[]:cost[]": {
+    "$(../account)": "$sum($float(totalcost))"
+  }
+}
+```
+
+Plain sum over nested apps:
+
+```json
+{ "#return:apps[]": "$sum(memory*instances)" }
+```
+
+Stitch fields into a key, sum a numeric one:
+
+```json
+{
+  "#return:results[]": {
+    "#var account:[]?field='AccountId'": "value",
+    "#var requests:[]?field='number_of_requests'": "$number(value)",
+    "$var(account)": "$sum($var(requests))"
+  }
+}
+```
+
+### 10.14 Bucketize with thresholds (`<<` joins back to original context)
+
+```json
+{
+  "#var dates": [
+    "'2018-08-22'",
+    "'2018-09-22'",
+    "'2018-10-22'",
+    "'2018-11-22'"
+  ],
+  "#return:data[]": {
+    "#var i->%dates:[]?$to_time(<<date)<$to_time(.)": "'up to '+.",
+    "%i": "$sum(value)"
+  }
+}
+```
+
+Notes:
+
+- `->%dates` switches context to the array stored in variable `dates`.
+- `<<date` reads `date` from the *outer* `data[]` element (before the switch).
+- `#var i` captures the bucket label so it can be reused as a *dynamic key* in `%i`.
+
+### 10.15 Generic recursive merge / transform
+
+```json
+{
+  "#func merge": {
+    "#returnif:{}": { "$key": "$merge" },
+    "#returnif:[]": ["$merge"],
+    "#return": "."
+  },
+  "#return": "$merge"
+}
+```
+
+This `merge` function recurses into every object key, every array element, and falls through for leaves. The `#returnif:{}` branch fires only if `.` is an object (yields a non-empty object result); same for `#returnif:[]`.
+
+### 10.16 Aggregate dictionary-style: per-key sum
+
+```json
+{
+  "#return:[]": {
+    "$(A)": {
+      "A": "A",
+      "{}:": "$sum($number(.))"
+    }
+  }
+}
+```
+
+### 10.17 Invert object: `{key: value}` → `{value: [key, ...]}`
+
+```json
+{ "#return:{}": { "$(.)": ["$key"] } }
+```
+
+### 10.18 Wildcard prefix filter
+
+```json
+{ "#return:[]?folder starts_with 'User/Admin'": ["."] }
+```
+
+### 10.19 Path-aware key construction
+
+Strip a known prefix from `$path`, then group:
+
+```json
+{
+  "#return:**": {
+    "$substr($path, $length('content.'))": {
+      "$(.)": "$count"
+    }
+  }
+}
+```
+
+### 10.20 Multi-field projection with dynamic key
+
+```json
+{
+  "msg": "message",
+  "status": "status",
+  "details:details[]": {
+    "$(test)": {
+      "amount:": ".",
+      "pre:": "."
+    }
+  }
+}
+```
+
+### 10.21 Add filename to each record
+
+```json
+[{
+  "Filename": "$filename",
+  "{}:": "."
+}]
+```
+
+### 10.22 Translate codes via variable-as-dictionary
+
+```json
+{
+  "#var states": {
+    "CA": "'California'",
+    "NJ": "'New Jersey'",
+    "NY": "'New York'",
+    "TX": "'Texas'"
+  },
+  "employees": [{
+    "FirstName:": ".",
+    "LastName:": ".",
+    "State": "$var(states).$(Address.State)"
+  }]
+}
+```
+
+### 10.23 Group-by + count at multiple levels
+
+Top-level group-by:
+
+```json
+{ "$(Title)": "$avg(Salary)" }
+```
+
+Nested per-employee count of children:
+
+```json
+[{
+  "FirstName:": ".",
+  "LastName:": ".",
+  "NumOfChildren:Dependants[]": "$count?Relation='Child'"
+}]
+```
+
+### 10.24 Filter by aggregated count (uses `#if` + context-modifier with `$count`)
+
+```json
+[{
+  "#if:Dependants[]?Relation='Child'": "$count>1",
+  "FirstName:": ".",
+  "LastName:": "."
+}]
+```
+
+### 10.25 Convert array-of-objects ↔ dictionary
+
+Array → dictionary (keyed by `ID`):
+
+```json
+{ "$(ID)": "." }
+```
+
+Dictionary → array of values (drops keys):
+
+```json
+{ "employees:{}": ["."] }
+```
+
+Add the original key into each value (e.g. promote `$key` → `Filename` field):
+
+```json
+{
+  "$(ID):{}": {
+    "Filename": "$filename",
+    "$key": "."
+  }
+}
+```
+
+Drop a known key while reshaping:
+
+```json
+{
+  "$(ID):{}?$key!='ID'": {
+    "$key": "."
+  }
+}
+```
+
+### 10.26 Walk children with stored context
+
+Snapshot the parent name into a variable, then iterate dependants and inject:
+
+```json
+{
+  "#var employee": "FirstName+' '+LastName",
+  "all_dependants:Dependants[]": [{
+    "{}:": ".",
+    "Employee": "$var(employee)"
+  }]
+}
+```
+
+### 10.27 Sort keys / collect all keys in a corpus
+
+```json
+{ "#return:**": ["$key@unique_ascending"] }
+```
+
+### 10.28 Aggregate over the entire corpus
+
+```json
+"$avg(Salary)"       // overall average
+"$count"             // document count
+"$count?Salary>200000"
+```
+
+---
+
+## 11. Common idioms — when to reach for which feature
+
+| Symptom | Use |
+|---|---|
+| "Same key on output as input" | `"Field:": "."` |
+| "Need iterative array, not just first value" | wrap in `[...]` |
+| "Need a flat array, not `{key:[...]}`" | `#return:` instead of a wrapper key |
+| "Value comes from one of N possibilities" | `#returnif` chain ending with `#return` for default |
+| "Need group-by on field X" | `{ "$(X)": [...] }` or `{ "$(X)": "$agg(...)" }` |
+| "Need dictionary keyed by a field" | `{ "$(field)": "." }` (no array) |
+| "Need to iterate fields of an object" | `:{}` or `{}:` |
+| "Need to iterate array elements" | `:[]` or `[]` in path |
+| "Need to walk every node anywhere" | `:**` recursive descent |
+| "Need to filter object keys by regex" | `{'regex'}` or `:{}?$key matches 'regex'` |
+| "Need to remove an existing field" | `"{}:": ".?$key!='X'"` |
+| "Need to add a field while copying others" | `[{ "X": "...", "{}:": "." }]` |
+| "Need to compute new key from value" | `{ "$(expr)": ... }` |
+| "Need a default when something is missing" | `#returnif:cond` then `#return:'default'` |
+| "Need recursion over arbitrary JSON" | `#func` calling itself via three `#returnif` cases (object/array/leaf) |
+| "Need to remember a value while changing context" | `#var` before the context-changing key, then `%name` / `$var(name)` inside |
+| "Need a value from the previous (outer) document" | `<<Field` after `->...` switch |
+
+---
+
+## 12. Static-check checklist for AI output
+
+Before returning a query, scan it against this checklist. These rules come straight from the parser and are the most common reasons an Unquery query fails to parse or fails to do what you intended.
+
+1. **Whole string consumed.** Each value string must parse cleanly to EOF — no trailing characters. If you concatenated something, recheck.
+2. **Aggregate vs. literal.** Either side of an aggregate comparison must be a literal. Reject `"$sum(x) > field"`.
+3. **`#if` value is a *condition only*.** No `?`, no `@sort`, no leading expression-then-predicate.
+4. **`#func` arity matches.** A `#func name(a,b)` declaration registers arity 2; every `$name(...)` in this template must pass exactly 2 comma-separated argument expressions.
+5. **`<-` is not a token.** To compare `< -5`, write it with whitespace or parentheses: `Field < -5`, `Field<(-5)`. Never type `Field<-5` and expect `<-`.
+6. **`..` is the parent step; you usually want `../`.** `..Field` is invalid. Write `../Field`.
+7. **`@sort` ends value parsing.** Place it last in the value string, after any `?cond` or constraint.
+8. **Condition OR is `|`.** `||` in a condition is a parse error. `||` is reserved for context-OR in keys.
+9. **`%x` ≡ `$var(x)`.** Use whichever is clearer; behavior is identical.
+10. **`#func` definitions go *before* their first use** in the same object.
+11. **Quoted args of XML helpers (`$attr`, `$child`, `$xpath`, `$lxpath`, `$in_filter`)** must be quoted string literals, not expressions.
+12. **Don't wrap a function token in `$()`.** Write `$index`, not `$($index)`.
+13. **Backticks are for field names.** `` `name with spaces` `` is a path segment, not a string literal.
+14. **Don't confuse predicate vs constraint inside object fields.** Predicate (`?`) skips just that key. Constraint (no `?`) skips the whole object.
+15. **Sort across an array of multiple items is undefined unless every item shares the same specifier.** Prefer single-item arrays for sorting.
+
+---
+
+## 13. Cross-reference
+
+- [`docs/unquery-grammar.md`](./unquery-grammar.md) — Parser-faithful EBNF, AST nodes, lexer tokens, every static rule. Use this for syntax-checking, error attribution, or when extending Unquery tooling. MCP resource: `xcitedb://unquery-grammar`.
+- [`web/src/docs/content/unquery-language-reference.html`](../web/src/docs/content/unquery-language-reference.html) — Human-oriented reference manual.
+- [`web/src/docs/content/unquery-tutorial.html`](../web/src/docs/content/unquery-tutorial.html) — Hands-on tutorial with the employees dataset.
+- XCiteDB-specific bits (workspaces, branches, identifiers, `->$date`, `->$branch`, `->$all`, `$xml*`, `$attr`, `$xpath`, `$node_date`, `$data_date`) only apply when running against XCiteDB. With the `unq` CLI on plain JSON files, ignore them.
+
+---
+
+## Document history
+
+- Synthesized from `docs/unquery-grammar.md`, the language reference and tutorial under `web/src/docs/content/`, and the canonical stackoverflow recipes under `Xcential/Unquery/tests/stackoverflow/`. Keep in sync when grammar or recipes change.