deepclause-sdk 0.1.0 → 0.1.2

package/dist/system/assets/docs/DML_REFERENCE.md

@@ -11,6 +11,7 @@
   - [exec/2](#exec2)
   - [Tool Definitions](#tool-definitions)
   - [Tool Scoping](#tool-scoping)
+  - [Model Selection](#model-selection)
   - [system/1, user/1](#memory-predicates)
   - [output/1, yield/1, log/1](#output-predicates)
   - [answer/1](#answer1)
@@ -104,8 +105,28 @@ agent_main :-
 
 ### Task Predicates
 
+Both `task/N` and `prompt/N` run an LLM step and bind typed outputs. They differ in
+**what conversation the model sees**, which is the single most important thing to
+know about them.
+
+| | `task/N` | `prompt/N` |
+|---|---|---|
+| Memory | the accumulated conversation, **and it is written back** | **fresh** — empty; does not read or write it |
+| Tools | the harness's `tool/2` predicates | **none** (plain text completion) |
+| Use for | a step in a multi-step flow that should know what came before | classification, extraction, transformation, or a bounded question that must not be swayed by the conversation |
+
 #### `task(+Description)`
 
+Invoke an LLM to perform a task. The description is sent to the model.
+
+```prolog
+agent_main :-
+    system("You are a coding assistant."),
+    task("Write a function to calculate factorial in Python."),
+    answer("Code generated!").
+```
+
+
 Invoke an LLM to perform a task. The description is sent to the model.
 
 ```prolog
@@ -158,6 +179,132 @@ Bind up to three variables.
 > `listing` and `export` reverse this, so you always read/write `task/N` and
 > `prompt/N` — never author `task_named/3` yourself.
 
+### Memory: what the model sees, and how it grows
+
+A `task/N` call is **not** a fresh conversation. It is sent the memory the session
+has accumulated, and the agent's full resulting memory is written back — so each
+`task/N` sees every earlier one, plus its own prompt and reply:
+
+```prolog
+step(Text, Summary, Tags) :-
+    task("Summarise: {Text}", Summary),     % sees: nothing yet
+    task("Tag this summary: {Summary}", Tags).   % sees: the first task's prompt and reply
+```
+
+`system/1` and `user/1` add to that same memory, and **they are backtrackable** —
+a `system/1` inside a branch that is later undone stops contributing:
+
+```prolog
+brief(Text, Out) :-
+    system("Answer in one sentence."),
+    task("Summarise: {Text}", Out).
+```
+
+Two consequences worth planning for:
+
+- **Memory grows with every task.** A loop of `task/N` calls sends an ever longer
+  conversation, which costs tokens and eventually dilutes the instruction. Pass
+  values as explicit arguments rather than relying on the model to remember, and
+  start a fresh step when the history is no longer relevant.
+- **A `task/N` inside a `tool/2` starts with empty memory.** Tool bodies run in
+  their own engine, which is created with no memory of the caller, so a nested
+  agent sees only what the *tool arguments* carry. Put the context in the argument,
+  or add it inside the tool body with `system/1`:
+
+  ```prolog
+  summarise(Text, Summary) :-
+      task("Summarise this text in two sentences: {Text}", Summary).
+
+  tool(summarise(Text, Summary), "Summarise a passage.") :-
+      summarise(Text, Summary).
+  ```
+
+**A fresh context: `prompt/N`, or `with_fresh_memory/2`.** `prompt/N` takes the same
+arguments as `task/N` but sends **no memory and no tools**, and leaves the memory
+untouched. Use it when the step must answer from its own prompt alone — a
+classifier, an extractor, a translation — or when a long history has started to
+interfere:
+
+```prolog
+route(Message, Team) :-
+    prompt("Classify as billing, orders or account: {Message}", Team).
+
+% The conversation so far cannot leak into the classification, and the
+% classification cannot leak into it.
+```
+
+Because `prompt/N` has no tools, a step that must read a file or search an index
+has to be a `task/N`. Choose by what the step needs: the harness's capabilities
+(`task/N`) or an uncontaminated answer (`prompt/N`).
+
+`prompt/N` isolates **one call**. To isolate a **group** of steps — several
+`task/N` calls that share a fresh context with each other but nothing with the
+caller — wrap them in `with_fresh_memory/1`:
+
+```prolog
+describe(Paper, Report) :-
+    with_fresh_memory((
+        task("Summarise the method of: {Paper}", Method),
+        task("Summarise the results of: {Paper}", Results),
+        task("Write a two-sentence abstract from {Method} and {Results}", Report)
+    )).
+% The three steps see each other's turns, because they accumulate within the
+% scope. Nothing the caller accumulated reaches them, and nothing they
+% accumulate reaches the caller — the memory is restored on exit.
+```
+
+With `with_tools/2` / `without_tools/2` it composes into a scope that controls both
+the conversation and the capabilities:
+
+```prolog
+audit(App, Finding) :-
+    with_fresh_memory(without_tools([apply_fix],
+        task("Review {App} against the checklist and report one finding.", Finding))).
+```
+
+### Declared Output Shapes: `schema/2`
+
+A type says "a list of dicts"; a schema says what is *in* them. Declare the shape
+once, name it in the `task/N` output, and every step gets a value that matches it —
+defaults filled, and a missing required field or a wrong-typed field failing the
+step instead of quietly yielding an empty result:
+
+```prolog
+schema(author, [
+    field(name,        string, required),
+    field(affiliation, string, default("not stated")),
+    field(email,       string, default("")),
+    field(country,     string, default(""))
+]).
+
+extract_authors(Text, Authors) :-
+    task("From the front matter extract every author: {Text}",
+         list(schema(author, Authors))).
+% Authors = [author{name:"Ashish Vaswani", affiliation:"Google Brain", email:"", country:""} …]
+```
+
+- An absent field with a `default(Value)` is filled.
+- An absent `required` field **fails the step** with a message naming the field, so
+  the agent can correct it in the same turn.
+- A field of the wrong type fails the same way. Types are `string`, `integer`,
+  `number`, `boolean`, `object`, `array`.
+- An **undeclared** field is kept, not dropped: a model mistake stays visible
+- The field list is **described to the model** in the step's own instructions, and a
+  missing required or wrong-typed field is reported by name inside the same turn,
+  so the agent usually fixes it before the step ends
+  rather than disappearing.
+- `schema/2` is a fact, like `doc/2` — it appears in the listing and travels with
+  `export`.
+
+For a single object rather than a list, use `schema(author, Author)` (no `list/1`).
+You can also apply a schema by hand to any dict:
+
+```prolog
+schema_fill(author, Raw, Author),        % fills defaults, validates
+```
+
+`end_turn` reports a `task/N` that names a schema which was never declared.
+
 ### Type-Safe Output Variables
 
 You can wrap output variables with type specifiers to enforce strict validation on the LLM's output. The runtime will ensure the value matches the specified type.
@@ -216,11 +363,15 @@ Or positional if the tool supports it:
 exec(get_time, CurrentTime)
 ```
 
-The `bash` tool accepts an optional wall-clock `timeoutMs` and always returns the
-wall-clock duration as `timeMs`:
+The `bash` tool takes exactly two inputs — `command` (a shell string) or `argv` (a
+list, quoted by the runtime) — plus an optional wall-clock `timeoutMs`. There is
+**no `stdin`**: pass data as an `argv` element, or write it to a workspace file and
+name the file. stdin is `/dev/null`, so a command that reads it sees EOF rather
+than waiting. It always returns the wall-clock duration as `timeMs`:
 
 ```prolog
 exec(bash(command: "sleep 30", timeoutMs: 5000), R),
+exec(bash(argv: ["python3", Script, Query]), R),      % no quoting to write
 get_dict(timeMs, R, Took),
 (   get_dict(exitCode, R, 124)
 ->  format(string(Msg), "timed out after ~w ms", [Took])
@@ -253,6 +404,25 @@ tool(count_words(File, Count), "Count words in a workspace file.") :-
     count_words(File, Count).
 ```
 
+The head and its delegate must have the **same name and the same arity**. The
+tool body is what runs when the nested agent calls the tool, so a body that
+calls a different arity either calls nothing (`existence_error`) or calls the
+tool again and recurses forever:
+
+```prolog
+% WRONG: declares /2 but calls /3, so the tool fails when a nested agent uses it.
+tool(bibliography(Query, Max), "…") :-
+    bibliography(Query, Max, Entries).
+
+% WRONG: calls itself; a nested agent recurses until the budget is gone.
+tool(citations(Text, Citations), "…") :-
+    citations(Text, Citations).
+```
+
+Nothing is checked when the clause is loaded, so both mistakes look fine until a
+nested agent calls the tool. `end_turn` reports them as contract warnings; treat
+them as errors, not style notes.
+
 A predicate defined *only* inside a `tool/2` clause is invisible to
 `execute` / `/goal` — the loader asserts it as `tool(Head)`, never as `Head`. Split
 the two so one implementation is reachable from both surfaces:
@@ -262,6 +432,12 @@ the two so one implementation is reachable from both surfaces:
 task("Count the words in report.txt.", N).   % nested agent calls the tool
 ```
 
+The only tools a nested agent can call are the harness's `tool/2` predicates;
+registered host tools (for example `bash`) are reachable from DML code through
+`exec/2`, not as tools by name. So a task description that says "use the
+`fs_read` tool" works only if the module itself defines `fs_read/2` **and**
+annotates it.
+
 The tool head's **last argument is the output**; the description is shown to the
 nested agent. Keep the `tool/2` body a single delegation — put no logic in it.
 
@@ -284,26 +460,103 @@ Tools can use `task()` and `prompt()` internally to combine Prolog logic with LL
 
 ```prolog
 % A predicate that computes then explains.
+% The nested task is told to work from the given figures — it has no tools it
+% needs, so it cannot try to re-enter this predicate.
 explain_calculation(A, B, Explanation) :-
     Sum is A + B,  % Prolog computation
-    format(string(Desc), "Explain ~w + ~w = ~w to a child", [A, B, Sum]),
-    task(Desc, Explanation).  % LLM explanation
+    format(string(Desc),
+           "Explain ~w + ~w = ~w to a child, using only these figures.",
+           [A, B, Sum]),
+    task(Desc, Explanation).
 
 tool(explain_calculation(A, B, Explanation), "Calculate and explain the result.") :-
     explain_calculation(A, B, Explanation).
+```
 
-% A predicate that uses the LLM for formatting.
-format_data(Data, FormattedOutput) :-
-    format(string(Desc), "Format this data nicely: ~w", [Data]),
-    task(Desc, FormattedOutput).
+**Do not wrap an LLM step in a `tool/2` that advertises the same job.** The tool
+is declared to the agent running *inside* that same `task/N`, so the description
+invites the agent to do the very thing it was asked, by calling the tool:
 
-tool(format_data(Data, FormattedOutput), "Format data nicely using LLM.") :-
-    format_data(Data, FormattedOutput).
+```prolog
+% WARNING: this shape self-delegates.
+extract_authors(Text, Authors) :-
+    task("Extract every author and their affiliation(s) from: {Text}", list(object(Authors))).
+
+tool(extract_authors(Text, Authors), "Extract every author and their affiliation(s).") :-
+    extract_authors(Text, Authors).
 ```
 
+Called as a predicate (`/goal extract_authors(…, A)`, `execute`, or from another
+predicate), `extract_authors` runs its `task/N`, and that agent is offered
+`extract_authors` — a tool described as doing exactly its job. The runtime refuses
+the re-entrant call (a tool whose predicate is already running is withheld, and a
+call that slips through is refused with a reason), so it cannot recurse forever.
+But the agent still spends a turn on it, is told the tool failed, and then does
+the work by hand — a confusing transcript and a wasted call.
+
+Two ways to avoid it, in order of preference:
+
+**Keep the LLM step a skill, and give `tool/2` only to capabilities.** A skill is
+called from DML (`/goal`, `execute`, another predicate) and does not need to be a
+tool. Annotate the operations a nested agent should *use* — deterministic or
+side-effecting work such as reading a PDF, searching an index, or shelling out:
+
+```prolog
+% Skill: an LLM step. Call it from DML; no tool/2.
+extract_authors(Text, Authors) :-
+    task("Extract every author and their affiliation(s) from: {Text}", list(object(Authors))).
+
+% Tools: capabilities for nested agents. No task/N, so no self-reference.
+read_pdf(Path, Text) :-
+    script_file(pdf_to_text, Script),
+    format(atom(Cmd), "python3 '~w' '~w'", [Script, Path]),
+    exec(bash(command: Cmd), R), get_dict(stdout, R, Text).
+
+tool(read_pdf(Path, Text), "Convert a workspace PDF to text.") :- read_pdf(Path, Text).
+```
+
+**Or scope the nested task's tools so the tool cannot be reached.** If a `task/N`
+inside a predicate must not use a tool, say so with `without_tools/2` (or list
+exactly what it may use with `with_tools/2`):
+
+```prolog
+extract_authors(Text, Authors) :-
+    without_tools([extract_authors],
+        task("Extract every author and their affiliation(s) from: {Text}",
+             list(object(Authors)))).
+```
+
+Scoping is the narrower fix: it keeps the tool advertised to *other* agents (which
+is usually why it exists) while removing it from this one.
+
 **Memory Isolation:** When `task()` or `prompt()` runs inside a tool, the nested agent starts with **fresh memory**. It does NOT inherit the parent's accumulated memory from `system()`, `user()`, or previous `task()` calls. If you need context in a nested task, pass it explicitly as a tool argument or add it with `system()` inside the tool body.
 
-**Tool Access:** The nested agent has access to all DML tools **except** the tool that is currently executing (automatic call stack exclusion to prevent infinite recursion). You can further control tool access using `with_tools/2` and `without_tools/2`.
+**Tool Access:** A nested agent's only callable tools are the harness's `tool/2`
+predicates; host tools such as `bash` are reachable from DML code via `exec/2`,
+never by name from a task description. A tool that is **already executing** cannot
+be re-entered: a call to it from inside itself is refused with an explanation, so a
+tool cannot recurse into itself. The refusal is deliberate and visible — the tool is
+declared to the agent with the reason on its description, so the call comes back as
+"already running further up the call chain" rather than as a missing tool, which
+would invite a retry.
+
+> The withholding is keyed on *tool* execution, and only that is guaranteed. Whether a
+> running tool is also *removed* from the tool list presented to the agent inside it
+> is not something the current test suite covers; the refusal above is what stops the
+> recursion. Do not rely on the tool being absent from `task/N`'s tool list.
+
+That keying has a consequence worth knowing: a predicate that is also a tool does
+**not** count as "executing" when it is called as a predicate (`/goal`, `execute`,
+or from another predicate). So for
+
+```prolog
+extract_authors(Text, Authors) :- task("…", list(object(Authors))).
+tool(extract_authors(Text, Authors), "…") :- extract_authors(Text, Authors).
+```
+
+`/goal extract_authors(Text, A)` still offers `extract_authors` to the agent inside
+`task/N`. Use `with_tools/2` / `without_tools/2` to scope it, or keep the LLM step
+out of the tool set entirely (see **Using `task()` Inside Tools** above).
 
 ```prolog
 % Example: passing context explicitly.
@@ -326,15 +579,19 @@ You can control which tools are available to nested `task()` calls using `with_t
 
 #### `with_tools(+ToolList, +Goal)`
 
-Run Goal with only the specified tools available to nested tasks:
+Run Goal with only the specified tools available to nested tasks. `ToolList`
+holds tool *names* (atoms), not predicates:
 
 ```prolog
-% Only allow tool_a and tool_b in the nested task
-tool(restricted_task(Input, Output), "Runs with limited tools") :-
-    with_tools([tool_a, tool_b], (
-        format(string(Desc), "Process '~w' using available tools", [Input]),
+% The implementation defines what the tool does; the annotation exposes it.
+restricted_work(Input, Output) :-
+    with_tools([search_papers, read_paper], (
+        format(string(Desc), "Process '~w' using the available tools", [Input]),
         task(Desc, Output)
     )).
+
+tool(restricted_task(Input, Output), "Runs with limited tools") :-
+    restricted_work(Input, Output).
 ```
 
 #### `without_tools(+ToolList, +Goal)`
@@ -342,12 +599,14 @@ tool(restricted_task(Input, Output), "Runs with limited tools") :-
 Run Goal with the specified tools excluded from nested tasks:
 
 ```prolog
-% Exclude expensive_tool from the nested task
-tool(cheap_task(Input, Output), "Runs without expensive tools") :-
+cheap_work(Input, Output) :-
     without_tools([expensive_tool], (
         format(string(Desc), "Process '~w' cheaply", [Input]),
         task(Desc, Output)
     )).
+
+tool(cheap_task(Input, Output), "Runs without expensive tools") :-
+    cheap_work(Input, Output).
 ```
 
 #### Using Tool Scoping in `agent_main`
@@ -371,6 +630,37 @@ agent_main :-
 
 ---
 
+### Model Selection
+
+#### `with_model(+Spec, +Goal)`
+
+Run `Goal` with a different model — and optionally a reasoning level — for every
+`task/N`, `prompt/N` and `sample_token/N` call inside it. Judgments
+(`choose/rate/verify/probability`) are not affected.
+
+```prolog
+% a cheap model for classification
+with_model("openai:gpt-4o-mini",
+    prompt("Classify as billing/orders/account: {Message}", Team)).
+
+% a strong model with high reasoning, only for the hard step
+with_model("anthropic:claude-sonnet-4#high",
+    task("Review this diff and list defects:\n{Diff}", Notes)).
+
+% full control with a dict
+with_model(model{provider: "google", id: "gemini-2.5-pro",
+                 temperature: 0.2, maxTokens: 8192}, Goal).
+```
+
+`Spec` is a string — `"provider:model"`, `"provider/model"`, `"model"`, each with
+an optional `#reasoning` suffix (`off|minimal|low|medium|high|xhigh|max`) — or a
+`model{provider, id, temperature, maxTokens, reasoning}` dict. The innermost
+`with_model` wins, and a nested `with_reasoning/2` overrides the spec's reasoning.
+The default is the session's active model. An unknown or unsigned-in model fails
+with a clear error.
+
+---
+
 ### Memory Predicates
 
 Memory predicates build the conversation context sent to the LLM.
@@ -429,24 +719,77 @@ Memory stores `media{path, mime}` — never the image bytes. Judgments
 
 ---
 
+#### Running a command with data in it
+
+`argv` takes the command and its arguments as a **list**, and the runtime quotes
+each element. Pass values here and you never write a quote, so an argument
+containing a space, a quote, `$`, a backtick or `;` is inert:
+
+```prolog
+exec(bash(argv: ["python3", Script, UserText]), R), get_dict(stdout, R, Text).
+```
+
+It is the same shell underneath (and the same sandbox), so pipelines and redirection
+still work with `command`; `argv` is the way to pass *data*. Prefer it whenever an
+argument comes from a file, a model, or the user. There is no `stdin` argument.
+
+#### Do not parse in DML — shell out or ask the model
+
+String handling is the fiddliest part of Prolog, and hand-rolled parsing is the
+easiest way to write DML that looks fine and is quietly wrong. Do not build a
+parser out of `sub_string/5`, `split_string/4`, `atom_concat/3`, `format/2` and an
+accumulator. Two better answers, depending on what the work actually is:
+
+- **Mechanical** (pull a field, change a format, count, filter, join): write a
+  **helper script** and run it. A script is a file the harness links with
+  `script_file/2`, it is testable on its own, and Python/jq/sed are built for this.
+  Have it emit **JSON** and read the result with `json_get/3` / `json_gets/3` /
+  `json_text/2`, rather than splitting its stdout by hand.
+- **Meaning** (find the field named X, classify, summarise, decide): make it a
+  **model call** with a typed output — `task("…", list(string(Items)))`,
+  `schema/2`, or `choose`/`rate`/`verify` for a bounded judgment. The model is the
+  right tool for understanding text; Prolog is the right tool for *sequencing and
+  checking* it.
+
+So `exec(bash(command: "cat", stdin: "…"), R), get_dict(stdout, R, O)` has no
+place here on either count: there is no stdin, and the structured reading of the
+result is a job for a script or a model, not for the DML around the call.
+
 ### File I/O and paths — prefer the shell
 
-**Prefer `exec(bash(...))` for file operations.**
-Reading, writing, copying, or converting files is usually clearer and more
-robust as a shell command; reach for the DML predicates below only when the
-content must be bound and reasoned over inside Prolog. Pulling a file into DML
-just to hand it back to the shell is a smell (`DML_AUTHORING.md` §2).
+**`/workspace` is the project, and it is the working directory.** A DML path is
+either absolute under `/workspace` or relative to it, and the two forms name the
+same file:
+
+    out/report.txt          and  /workspace/out/report.txt
+
+Prefer the `/workspace/...` form: the same name works from DML, from
+`exec(bash(...))`, and from the shell tool, on every backend, so a path never has
+to be translated between them. There is **no paths file and no search-path
+registry** — nothing to declare, register, or discover. `script_file(Name, Path)`
+links a file that belongs to the harness (it is bundled on `export`); it does not
+resolve anything at run time.
+
+**Everything outside `/workspace` is the ephemeral VM filesystem.** `/etc`,
+`/tmp`, `/home` are not this machine. They are an in-memory filesystem that
+belongs to the session and is discarded when it ends, so they are fine for scratch
+and useless for results. **Only files under `/workspace` reach the host.** For
+scratch, use `$TMPDIR`, which is a per-session directory in every backend (a bare
+`/tmp` is the real one when nothing is sandboxed).
 
-If you do use them: every path in DML is **workspace-relative** and confined to
-the workspace. The
-engine mounts the workspace at `/workspace`, but you must not write
-`/workspace/...` in DML — passing an absolute path is rejected, as is `..`. File
-predicates all take a relative path:
+**Prefer `exec(bash(...))` for file operations.** Reading, writing, copying, or
+converting files is usually clearer and more robust as a shell command; reach for
+the DML predicates below only when the content must be bound and reasoned over
+inside Prolog. Pulling a file into DML just to hand it back to the shell is a
+smell (`DML_AUTHORING.md` §2).
+
+The file predicates take a path — absolute under `/workspace`, or relative:
 
 ```prolog
-read_file("images/a.png", Bytes),
-write_file("out/report.txt", Text),
-make_directory_path("out/nested"),          % create a directory (and parents)
+read_file("/workspace/images/a.png", Bytes),
+read_file("images/a.png", Bytes),            % same file
+write_file("/workspace/out/report.txt", Text),
+make_directory_path("/workspace/out/nested"),   % create a directory (and parents)
 append_file("out/log.txt", Line),
 delete_file("out/report.txt"),
 directory_files("images", Names),        % bare entry names: ["a.png", ...]
@@ -458,10 +801,24 @@ time_file("images/a.png", Modified).
 `write_file/2` does **not** create parent directories — call
 `make_directory_path/1` first. `directory_files/2` returns bare entry names (no
 directory), so build a path with `atom_concat('images/', Name, Path)` when you
-need one. `bash` runs with a wider view of the sandbox than DML has; convert
-anything from the shell to a workspace-relative path before handing it to DML.
+need one.
+
+These predicates are **real clauses in the session module**, not only
+interpreter handlers, so DML code, a nested `task/N`, and a direct host query
+(the SDK's `runDML`, or `deepclause repl`) all resolve the same definition and
+the same path.
+
+`open/3`, `close/1`, `read_string/3` and `write/2` are SWI builtins and cannot be
+shadowed, but they resolve paths the same way — the working directory is
+`/workspace`, so `open("f.txt", read, S)` reads the workspace file. They are not
+a special case with their own path rule.
+
+Paths are **isolated by the mount, not by string checks**: there is no rewriting
+layer, so a path is passed through to the filesystem as written. `..` does not
+climb out of the workspace, because above `/workspace` is the VM's own empty
+filesystem. The one thing a mount cannot police is a **symlink inside the
+workspace** pointing outside it; avoid committing those.
 
----
 
 ### Output Predicates
 
@@ -477,6 +834,25 @@ agent_main :-
     answer("Done").
 ```
 
+**Use it for debugging.** When a predicate does not do what you expect, add
+`output/1` calls between the steps: you see how far execution gets, what the
+intermediate values actually are, and which call returned nothing. It costs one
+line, it is invisible to the model (unlike a `task/N` description), and it is the
+fastest way to find where a chain goes wrong:
+
+```prolog
+summarise(Doc, Summary) :-
+    extract_sections(Doc, Sections),
+    format(string(S), "sections: ~w", [Sections]), output(S),
+    pick_main(Sections, Main),
+    format(string(M), "main: ~w", [Main]), output(M),
+    task("Summarise this in one sentence: ~w", [Main], Summary).
+```
+
+Remove them before the final version, or keep the ones that read as progress
+reports — a predicate that says what it is doing is easier to debug the next
+time too.
+
 #### `yield(+Text)`
 
 Alias for `output/1`.
@@ -564,6 +940,40 @@ agent_main(Topic) :-
 
 ---
 
+## Patterns: one template, build or parse
+
+`pattern/3` is `format/3` and its inverse. Literal text matches exactly and `$name`
+marks a slot, so the same template renders a string or parses it — the direction
+follows from what is bound:
+
+```prolog
+pattern("Name: $name, age: $age", Text, [name="Ada", age=36]).
+% Text = "Name: Ada, age: 36"
+
+pattern("Name: $name, age: $age", "Name: Ada, age: 36", Bindings).
+% Bindings = [name="Ada", age="36"]
+
+pattern("DOI: $doi", Text, [doi=D]).                % build a DOI line
+pattern("n=$n:int", "n=42", B).                     % B = [n=42]   (typed slot)
+```
+
+- Bindings are a list of `Name=Value`, so `member(Name=Value, Bs)` reads them.
+- A slot consumes **up to the next literal**, so the literals decide the split and
+  `"$left - $right"` against `"a - b - c"` gives `left="a"`, `right="b - c"`.
+- `$name:type` converts on parse and validates on build. Types: `string` (default),
+  `int`, `integer`, `number`. A parse that does not match throws, rather than
+  binding the text.
+- Slot names match case- and separator-insensitively, like `dict_field/3`.
+- `$` is the marker rather than `{…}`, so a template can carry JSON or any other
+  brace-bearing text verbatim, and it never collides with the `{Var}`
+  interpolation in `task/N` descriptions. `$$` is a literal dollar.
+- A template must be **anchored by literal text**: `"$a$b"` is rejected as
+  ambiguous rather than resolved arbitrarily.
+
+This is the deterministic alternative to asking the model to reformat text: parse
+the parts you can read exactly, and use a `task/N` only for what genuinely needs
+language understanding.
+
 ## Backtracking
 
 DML supports full Prolog backtracking, **including across LLM calls**. This enables powerful retry logic:
@@ -623,20 +1033,112 @@ all: arithmetic, counting, ordering, membership, `findall/3`, `aggregate_all/3`,
 list/set operations, and constraint or optimisation problems. Reach for `task/N`
 only when the step genuinely needs language understanding.
 
-### Lists
+### Lists and common operations
+
+A list is `[1,2,3]` or `[Head|Tail]`; the recursion below is the shape everything
+else is built on. Prefer the library predicates to hand-rolled recursion.
+
 ```prolog
 process_items([]).
 process_items([H|T]) :-
     format(string(Msg), "Processing: ~w", [H]),
     output(Msg),
     process_items(T).
+```
 
-agent_main :-
-    process_items([apple, banana, cherry]),
-    answer("All items processed").
+`maplist/3`, `include/3`, `exclude/3`, `foldl/4` and `predsort/3` take a **goal with
+extra arguments**, written `[X,Y]>>Goal` (a *yall* lambda). The list before `>>` is
+the extra arguments — so `[X,Y]>>(Y is X*X)` is a two-argument goal. The lambda's
+arity must match the predicate you are using.
+
+**Transform**
+
+| Want | Write |
+|---|---|
+| Apply to each | `maplist([X,Y]>>(Y is X*X), [1,2,3], L)` → `[1,4,9]` |
+| Combine two lists | `maplist([A,B,C]>>(C is A+B), [1,2], [10,20], L)` → `[11,22]` |
+| Apply, skipping failures | `convlist([X,Y]>>(integer(X), Y is X*2), [1,a,3], L)` → `[2,6]` |
+| Keep / drop matching | `include(P, L, Keep)` / `exclude(P, L, Drop)` |
+| Reduce to one value | `foldl([X,A0,A]>>(A is A0+X), [1,2,3], 0, S)` → `6` |
+| Reverse | `reverse([1,2,3], L)` |
+| Take / split | `append(Chunk, Rest, List)`, or `length(Chunk, 2)` to bound it |
+| A list of N blanks | `length(L, 3)` |
+
+**Select and aggregate**
+
+| Want | Write |
+|---|---|
+| Element at position | `nth0(1, [a,b,c], E)` (0-based) / `nth1(1, …, E)` (1-based) |
+| Remove one occurrence | `select(b, [a,b,c], L)` → `[a,c]` |
+| Check membership (once) | `memberchk(b, L)` — semidet, no choice point |
+| Last element | `last([a,b,c], E)` |
+| Sum | `sum_list([1,2,3], S)` → `6` |
+| Largest / smallest | `max_member(M, L)` / `min_member(M, L)` |
+| Any aggregate | `aggregate_all(max(N), member(N,[1,9,3]), M)` → `9`; also `min`, `sum`, `count`, `bag` |
+
+**Sort, group, dedupe**
+
+| Want | Write |
+|---|---|
+| Sort and dedupe | `sort([3,1,2,1], L)` → `[1,2,3]` |
+| Sort keeping duplicates | `msort([3,1,2,1], L)` → `[1,1,2,3]` (stable) |
+| Sort pairs by key | `keysort([b-2,a-1], L)` → `[a-1,b-2]` (stable) |
+| Sort compounds by an argument | `sort(2, @>=, [a-1,b-3,c-2], L)` — the `2` is the argument position |
+| Sort with your own comparison | `predsort([O,A,B]>>compare(O,A,B), L, Sorted)` |
+| Dedupe, keeping first order | `list_to_set([a,b,a,c], L)` → `[a,b,c]` |
+| Group by key | `group_pairs_by_key([a-1,a-2,b-3], G)` → `[a-[1,2], b-[3]]` |
+| Set difference / intersect / union | `subtract/3`, `intersection/3`, `union/3` |
+| Zip / unzip | `pairs_keys_values(Pairs, Keys, Values)`, `pairs_keys/2`, `pairs_values/2` |
+
+**Strings in a list.** `split_string/4` leaves empty strings behind, which is the
+usual reason a list "has blanks in it":
+
+```prolog
+split_string("a,,b", ",", "", Parts),      % Parts = ["a","","b"]
+exclude(==(""), Parts, Clean).              % Clean = ["a","b"]
+include(dif(""), Parts, Clean).             % the same, by unification
+exclude(=([]), Lists, NonEmpty).            % drop empty lists
+```
+
+**A field from a list of dicts** — the most common list task with model output:
+
+```prolog
+json_gets(Doc, "rows.*.name", Names),                       % one call, collects
+findall(N, (member(D, Dicts), dict_field(D, name, N)), Names).   % from a list
+```
+
+To sort a list of dicts, use `predsort/3`: `sort/4` addresses arguments of a
+compound and a dict has none.
+
+**Batch a list into chunks.** There is no builtin; this is the definition:
+
+```prolog
+chunks([], _, []).
+chunks(List, N, [Chunk|Rest]) :-
+    N > 0,
+    length(Chunk, N),
+    append(Chunk, Tail, List),
+    chunks(Tail, N, Rest).
+chunks(List, N, [List]) :-
+    N > 0,
+    length(List, Len),
+    Len < N.
+% chunks([1,2,3,4,5], 2, L) -> L = [[1,2],[3,4],[5]]
 ```
 
-Not all builtin SWI Prolog predicates are guaranteed to work at this stage!
+### Traps
+
+- **`setof/3` and `bagof/3` fail when there are no solutions**; `findall/3` succeeds
+  with `[]`. Use `findall/3` unless you want that failure.
+- **`sort/2` silently removes duplicates.** Use `msort/2` to keep them.
+- **`max_list([])` fails**, and `sum_list([a,b])` raises a type error — guard a
+  possibly-empty or possibly-non-numeric list before aggregating.
+- **A nondeterministic goal inside `maplist`, `include` or `foldl` multiplies
+  solutions**, and re-runs any model call in it. Keep the goal semidet
+  (`once/1`, `->`, a cut).
+- **Not available in this build:** `nth0/4` (replace an element), `numlist/4` (range
+  with a step), `chunk/3`, `rename_keys/3`. Replace an element with
+  `nth0(I, L, _, Tail), nth0(I, New, V, Tail)`, and use `chunks/3` above.
 
 ### Arithmetic
 ```prolog
@@ -806,6 +1308,71 @@ exact_max(X, Y) :-
 ```
 
 ### Dictionaries (SWI-Prolog)
+
+**Keys are atoms.** `get_dict/3` is strict about the key type and *throws* rather
+than failing: `get_dict("affiliation", D)` raises `type_error(dict-key,
+affiliation)`, so no guard sees it as a "no". A string is not even valid dict
+literal syntax (`x{"a":1}` is a syntax error).
+
+Model output makes this worse, because the field name varies — `Name`, `name`,
+`full_name`. Use `dict_field/3,4`, which matches an atom **or** a string key,
+ignoring case and `_`/`-`/space:
+
+```prolog
+dict_field(Dict, "affiliation", Affiliation),          % fails if absent
+dict_field(Dict, full_name, "unknown", Name),          % with a default
+```
+
+**JSON.** `atom_json_dict/3` works in both directions, which is the whole story —
+reading and writing:
+
+```prolog
+atom_json_dict(Text, Dict, []),          % JSON text -> dict  (Text bound)
+atom_json_dict(Text, Dict, []),          % dict -> JSON text  (Text unbound)
+```
+
+Reading a nested value is one call with a path rather than a chain of lookups:
+
+```prolog
+json_get(Doc, "results.0.authors.1.name", Name),        % dotted path
+json_get(Doc, [results, 0, authors, 1, name], Name),    % or a list
+json_get(Doc, "results.0.year", "unknown", Year),       % with a default
+json_gets(Doc, "results.*.title", Titles),              % project: collect `*`
+json_gets(Doc, "results.*.authors.*.name", Names),      % nested `*` flattens
+json_put(Doc, "meta.reviewed", yes, Updated),           % set a key path
+json_text(Doc, Text),                                   % compact JSON
+```
+
+A missing path **fails**; it does not throw and does not give `""`, so absence is
+visible. Key matching is the same tolerant match as `dict_field/3` (atom or string,
+any case, `_`/`-`/space ignored). `json_put/4` sets keys and creates intermediate
+dicts; it rejects an index path, so lists are read-only. For layout or streaming,
+`json_write_dict/3` from `library(http/json)` is available.
+
+**Parsing text with a grammar.** DCGs work — the loader runs `expand_term/2`, so
+`-->` compiles, and `library(dcg/basics)` imports:
+
+```prolog
+:- use_module(library(dcg/basics)).
+
+key_value(K, V) --> string_without(":", K), ":", string(V).
+
+parse_kv(Text, Pairs) :-
+    string_chars(Text, Chars),          % phrase/2 wants a LIST, not a string
+    phrase(lines(Pairs), Chars).
+
+lines([K-V|Rest]) --> key_value(K, V), "\n", !, lines(Rest).
+lines([K-V]) --> key_value(K, V).
+```
+
+`phrase/2,3` takes a **code or char list**, so `string_chars/2` (or
+`string_codes/2`) comes first — passing a string raises `type_error(list, …)`.
+Also useful here: `number_codes/2`, `read_term_from_atom/3`,
+`split_string/4`. DCGs give you a documented, testable parser instead of a chain of
+regex calls, and the same grammar can generate text as well as accept it.
+
+The rest of this section is plain SWI dict access, for keys you control:
+
 ```prolog
 agent_main :-
     D = point{x: 10, y: 20},
@@ -946,6 +1513,52 @@ backtracking reuses an answer instead of re-querying the model.
   always one of your option atoms.
 - Prefer one `judge/2` batch over several one-off calls when you need more than
   one or two judgments.
+### Load-time directives
+
+A `.dml` file may contain directives. These are handled:
+
+```prolog
+:- use_module(library(clpfd)).     % library import
+:- consult("other.dml").           % load another workspace file
+:- param(topic, "the subject").    % declared session parameter
+:- repl_delete(old_skill/2).       % remove a predicate (same as the bare fact form)
+:- require_sandbox.                % refuse to load unless the shell is sandboxed
+```
+
+`:- require_sandbox.` is a declaration that the harness must run with an
+isolated shell. It is checked **at load**, and the load **fails** when the
+effective backend is not a sandbox (`bwrap`, AgentVM, or `sandbox-exec`) — it
+does not silently run unsandboxed. "Effective" is deliberate: bubblewrap being
+installed while the session resolves to unsandboxed execution still fails the
+check. Only the **user** sets this (the `/set-require-sandbox` command); an agent
+cannot add or remove it.
+
+`:- repl_delete(Name/Arity).` abolishes a predicate, which `retract/1` cannot do:
+retracting the last clause leaves an empty predicate whose calls fail silently.
+The bare fact form `repl_delete(Name/Arity).` works the same way.
+
+These apply to a file **loaded** into a session (`deepclause repl -l file.dml`).
+They are not the way to edit a live harness. `consult/1`, a file-list directive and
+`use_module/1` on a workspace `.dml` assert straight into the module rather than
+through the REPL's edit tools, so the predicates they add are absent from the
+transaction's tracked set — the nondeterminism probe does not see them, and a
+directive that fails inside the file only warns. They do still appear in the next
+commit's diff, so the change is recorded without any account of where it came from —
+and outside a transaction there is no snapshot, so nothing validates the load at all.
+An agent changing a harness adds code with `assert`/`assert_many`/`define` inside a
+transaction instead.
+
+Any other directive does nothing, and the loader now says so — it reports
+`unrecognised directive ignored: …` after the file loads, so a directive that
+never took effect is not mistaken for one that did. `dynamic/1`,
+`discontiguous/1` and `multifile/1` are accepted and ignored silently, since
+session predicates are asserted and already behave that way.
+
+A **predicate with no clauses fails silently** when called — it does not error —
+so an emptied or never-written predicate looks like "no results" at the call site.
+`current_predicate(Name/Arity)` in a `/goal` answers whether it exists, and a
+`doc/2` for a predicate with no clauses is reported at `end_turn`.
+
 ## The DML REPL (`deepclause repl`)
 
 DML skills can also be grown interactively in a persistent **living module**.