deepclause-sdk 0.0.89 → 0.1.0

package/dist/system/assets/docs/DML_REFERENCE.md

@@ -14,7 +14,6 @@
   - [system/1, user/1](#memory-predicates)
   - [output/1, yield/1, log/1](#output-predicates)
   - [answer/1](#answer1)
-  - [param/3](#param3)
 - [Control Flow](#control-flow)
 - [String Interpolation](#string-interpolation)
 - [Backtracking](#backtracking)
@@ -58,7 +57,7 @@ agent_main :-
 ```
 
 ### agent_main/1
-Receives the first parameter:
+Receives the first argument:
 ```prolog
 agent_main(Name) :-
     format(string(Msg), "Hello, ~w!", [Name]),
@@ -66,7 +65,7 @@ agent_main(Name) :-
 ```
 
 ### agent_main/2
-Receives first two parameters:
+Receives the first two arguments:
 ```prolog
 agent_main(Name, Task) :-
     system("You are a helpful assistant."),
@@ -142,6 +141,23 @@ agent_main :-
 
 Bind up to three variables.
 
+> **Note — `task/N` is load-time sugar.** `task/1,2,3,4` and `prompt/2,3,4` are
+> rewritten by the DML loader into `task_named/3` / `prompt_named/3` with typed
+> output variables. The rewrite runs whenever code is loaded (`deepclause run`,
+> the REPL `-l` and commit paths, `consult`, and the REPL `assert`/`assert_many`
+> tools). The untransformed form has no runtime handler,
+> so code that somehow reaches the runtime without the rewrite raises
+> `domain_error(unrewritten_task_form, Goal)` instead of failing silently.
+> `task/1`/`prompt/1` are handled directly and are unaffected.
+
+> **Internal note.** At load time `task/N`/`prompt/N` are rewritten to
+> `task_named/3`/`prompt_named/3` (which carry the output-variable names) and
+> `{Var}` descriptions are hoisted into a `format/3`. That is why a description
+> like `task("Answer: {Question}", Answer)` sometimes *appears* as
+> `format(string(C), "Answer: ~w", [Question]), task_named(C, [Answer], ...)`.
+> `listing` and `export` reverse this, so you always read/write `task/N` and
+> `prompt/N` — never author `task_named/3` yourself.
+
 ### Type-Safe Output Variables
 
 You can wrap output variables with type specifiers to enforce strict validation on the LLM's output. The runtime will ensure the value matches the specified type.
@@ -200,38 +216,66 @@ Or positional if the tool supports it:
 exec(get_time, CurrentTime)
 ```
 
+The `bash` tool accepts an optional wall-clock `timeoutMs` and always returns the
+wall-clock duration as `timeMs`:
+
+```prolog
+exec(bash(command: "sleep 30", timeoutMs: 5000), R),
+get_dict(timeMs, R, Took),
+(   get_dict(exitCode, R, 124)
+->  format(string(Msg), "timed out after ~w ms", [Took])
+;   format(string(Msg), "finished in ~w ms", [Took])
+).
+```
+
+A command is also killed when it produces no output for `DC_HOST_SHELL_IDLE_TIMEOUT_MS`
+(default 180 000 ms); `DC_HOST_SHELL_TIMEOUT_MS` sets a default wall-clock cap
+(0 = disabled). Exit code `124` means a timeout.
+
 ---
 
 ### Tool Definitions
 
-Define tools in DML that the LLM can call during `task()`. Tools use the `tool/2` predicate with a description:
+`tool/2` is an **annotation**, not an implementation. It exposes an existing
+harness predicate to the nested agent that runs inside `task/N`. Define the
+predicate normally, then add a `tool/2` clause that delegates to it:
 
 ```prolog
-% Define a tool - makes it available to the LLM with description
-tool(calculate(Expression, Result), "Calculate a mathematical expression") :-
-    exec(calculator(expression: Expression), Result).
+% 1. The capability: a normal predicate, callable from `execute` and `/goal`.
+count_words(File, Count) :-
+    script_file(count_words, Script),
+    format(atom(Cmd), "python3 '~w' '~w'", [Script, File]),
+    exec(bash(command: Cmd), Result),
+    get_dict(stdout, Result, Count).
 
-agent_main :-
-    system("You have a calculate tool."),
-    task("What is the square root of 144?"),
-    answer("Done").
+% 2. The annotation: exposes it to the nested agent. Thin body only.
+tool(count_words(File, Count), "Count words in a workspace file.") :-
+    count_words(File, Count).
 ```
 
-Tool definitions have the form:
+A predicate defined *only* inside a `tool/2` clause is invisible to
+`execute` / `/goal` — the loader asserts it as `tool(Head)`, never as `Head`. Split
+the two so one implementation is reachable from both surfaces:
+
 ```prolog
-tool(name(Arg1, Arg2, ..., Result), "Description") :- Body.
+/goal count_words("report.txt", C).          % authoring agent
+task("Count the words in report.txt.", N).   % nested agent calls the tool
 ```
 
-The last argument is typically the output. The description is shown to the LLM.
-```
+The tool head's **last argument is the output**; the description is shown to the
+nested agent. Keep the `tool/2` body a single delegation — put no logic in it.
+
+Logic lives in the predicate, not the annotation:
 
-You can add Prolog logic in tool implementations:
 ```prolog
-tool(safe_divide(A, B, Result), "Safely divide two numbers") :-
+safe_divide(A, B, Result) :-
     (   B =:= 0
     ->  Result = error("Division by zero")
-    ;   exec(calculator(expression: A/B), Result)
+    ;   Result is A / B
     ).
+
+tool(safe_divide(A, B, Result), "Safely divide two numbers.") :-
+    safe_divide(A, B, Result).
 ```
 
 #### Using `task()` Inside Tools
@@ -239,20 +283,22 @@ tool(safe_divide(A, B, Result), "Safely divide two numbers") :-
 Tools can use `task()` and `prompt()` internally to combine Prolog logic with LLM reasoning:
 
 ```prolog
-% A tool that computes then explains
-:- tool(explain_calculation(A, B), "Calculate and explain the result").
-
-tool(explain_calculation(A, B, Explanation)) :-
+% A predicate that computes then explains.
+explain_calculation(A, B, Explanation) :-
     Sum is A + B,  % Prolog computation
     format(string(Desc), "Explain ~w + ~w = ~w to a child", [A, B, Sum]),
     task(Desc, Explanation).  % LLM explanation
 
-% A tool that uses LLM for formatting
-:- tool(format_data(Data), "Format data nicely using LLM").
+tool(explain_calculation(A, B, Explanation), "Calculate and explain the result.") :-
+    explain_calculation(A, B, Explanation).
 
-tool(format_data(Data, FormattedOutput)) :-
+% A predicate that uses the LLM for formatting.
+format_data(Data, FormattedOutput) :-
     format(string(Desc), "Format this data nicely: ~w", [Data]),
     task(Desc, FormattedOutput).
+
+tool(format_data(Data, FormattedOutput), "Format data nicely using LLM.") :-
+    format_data(Data, FormattedOutput).
 ```
 
 **Memory Isolation:** When `task()` or `prompt()` runs inside a tool, the nested agent starts with **fresh memory**. It does NOT inherit the parent's accumulated memory from `system()`, `user()`, or previous `task()` calls. If you need context in a nested task, pass it explicitly as a tool argument or add it with `system()` inside the tool body.
@@ -260,12 +306,16 @@ tool(format_data(Data, FormattedOutput)) :-
 **Tool Access:** The nested agent has access to all DML tools **except** the tool that is currently executing (automatic call stack exclusion to prevent infinite recursion). You can further control tool access using `with_tools/2` and `without_tools/2`.
 
 ```prolog
-% Example: Passing context explicitly
-tool(smart_search(Context, Query, Summary), "Search with context") :-
-    system(Context),  % Add context to nested task's memory
-    exec(web_search(query: Query), Results),
-    format(string(Desc), "Summarize: ~w", [Results]),
+% Example: passing context explicitly.
+smart_grep(BuildContext, Pattern, Summary) :-
+    system(BuildContext),            % add context to the nested task's memory
+    format(atom(Cmd), "grep -rn '~w' .", [Pattern]),
+    exec(bash(command: Cmd), Results),
+    format(string(Desc), "Summarize these matches: ~w", [Results]),
     task(Desc, Summary).
+
+tool(smart_grep(BuildContext, Pattern, Summary), "Grep with context.") :-
+    smart_grep(BuildContext, Pattern, Summary).
 ```
 
 ---
@@ -309,7 +359,7 @@ agent_main :-
     system("You are a safe assistant."),
     
     % Run task with only safe tools
-    with_tools([read_file, list_files], (
+    with_tools([search, fetch_page], (
         task("Analyze the files in the workspace.", Analysis)
     )),
     
@@ -351,6 +401,68 @@ agent_main :-
 
 ---
 
+### Media Predicates
+
+Images are **workspace files**; DML carries only a reference. See
+`docs/DML_MEDIA.md`.
+
+#### `read_media(+Path, -Media)`
+
+Bind `Media = media(Path, Mime)` for a workspace-relative image (PNG, JPEG, GIF,
+WebP). The bytes are not read into Prolog.
+
+#### `attach_media(+Media)`
+
+Append the image to conversation memory (reference only), so the next `task/N`
+call sends it to the model as an image block:
+
+```prolog
+agent_main :-
+    read_media("photos/fox.png", Img),
+    attach_media(Img),
+    task("Describe the image in one sentence.", Description),
+    answer(Description).
+```
+
+Memory stores `media{path, mime}` — never the image bytes. Judgments
+(`choose`/`rate`/`verify`/`probability`) stay text-only.
+
+---
+
+### File I/O and paths — prefer the shell
+
+**Prefer `exec(bash(...))` for file operations.**
+Reading, writing, copying, or converting files is usually clearer and more
+robust as a shell command; reach for the DML predicates below only when the
+content must be bound and reasoned over inside Prolog. Pulling a file into DML
+just to hand it back to the shell is a smell (`DML_AUTHORING.md` §2).
+
+If you do use them: every path in DML is **workspace-relative** and confined to
+the workspace. The
+engine mounts the workspace at `/workspace`, but you must not write
+`/workspace/...` in DML — passing an absolute path is rejected, as is `..`. File
+predicates all take a relative path:
+
+```prolog
+read_file("images/a.png", Bytes),
+write_file("out/report.txt", Text),
+make_directory_path("out/nested"),          % create a directory (and parents)
+append_file("out/log.txt", Line),
+delete_file("out/report.txt"),
+directory_files("images", Names),        % bare entry names: ["a.png", ...]
+exists_file("images/a.png"),
+size_file("images/a.png", Size),
+time_file("images/a.png", Modified).
+```
+
+`write_file/2` does **not** create parent directories — call
+`make_directory_path/1` first. `directory_files/2` returns bare entry names (no
+directory), so build a path with `atom_concat('images/', Name, Path)` when you
+need one. `bash` runs with a wider view of the sandbox than DML has; convert
+anything from the shell to a workspace-relative path before handing it to DML.
+
+---
+
 ### Output Predicates
 
 #### `output(+Text)`
@@ -397,25 +509,6 @@ agent_main :-
 
 ---
 
-### `param(+Key, +Description, -Value)`
-
-Access parameters passed from the caller:
-
-```prolog
-agent_main :-
-    param(name, "User's name", Name),
-    param(age, "User's age", Age),
-    format(string(Msg), "Hello ~w, you are ~w years old!", [Name, Age]),
-    answer(Msg).
-```
-
-Called from TypeScript:
-```typescript
-dc.runDML(code, { params: { name: 'Alice', age: 30 } });
-```
-
----
-
 ## Control Flow
 
 DML supports standard Prolog control flow:
@@ -460,14 +553,13 @@ throw(Error)
 
 ## String Interpolation
 
-Text arguments support `{variable}` interpolation with parameters:
+Text arguments support `{variable}` interpolation with bound variables:
 
 ```prolog
-agent_main :-
-    param(topic, "Research topic", Topic),
+agent_main(Topic) :-
     system("You are a research assistant."),
-    task("Research the topic: {topic}"),  % Topic is interpolated
-    answer("Research on {topic} complete!").
+    task("Research the topic: {Topic}"),  % Topic is interpolated
+    answer("Research on {Topic} complete!").
 ```
 
 ---
@@ -524,7 +616,12 @@ agent_main :-
 
 ## Full Prolog Support
 
-DML is built on SWI-Prolog, so you have access to the full Prolog language:
+DML is built on SWI-Prolog, so you have access to the full Prolog language — and
+to the constraint libraries (`clpfd`, `clpq`, `clpr`). Many questions are best
+answered by a plain Prolog or CLP goal run with `execute`, with no model call at
+all: arithmetic, counting, ordering, membership, `findall/3`, `aggregate_all/3`,
+list/set operations, and constraint or optimisation problems. Reach for `task/N`
+only when the step genuinely needs language understanding.
 
 ### Lists
 ```prolog
@@ -568,6 +665,44 @@ agent_main :-
     answer(Msg).
 ```
 
+### Bounding a goal (inference limit)
+
+The interpreter has **no wall-clock timeout for a single builtin call** — the
+session `gas` budget counts meta-interpreter steps, and a runaway `findall/3`,
+`aggregate_all/3`, or `label/1` runs inside one step. Wrap such a call in
+`call_with_inference_limit/3` (core SWI-Prolog; available in this runtime, which
+has no threads) to bound it by inference count:
+
+```prolog
+bounded_findall(Goal, Xs) :-
+    call_with_inference_limit(findall(X, call(Goal, X), Xs), 1000000, Result),
+    (   Result == inference_limit_exceeded
+    ->  throw(error(inference_limit_exceeded, context(bounded_findall/2, Goal)))
+    ;   true
+    ).
+```
+
+`Result` is `inference_limit_exceeded` when the limit is hit (otherwise the
+normal call result). Use it for `findall/3`, `aggregate_all/3`, and CLP
+`label/1` over a large or under-specified domain:
+
+```prolog
+sudoku_solve(Puzzle, Solution) :-
+    ...
+    call_with_inference_limit(label(Solution), 500000, Result),
+    (   Result == inference_limit_exceeded
+    ->  throw(error(inference_limit_exceeded, context(sudoku_solve/2, label)))
+    ;   true
+    ).
+```
+
+System libraries imported with `:- use_module(library(...)).` (e.g. `clpfd`) are
+also visible inside meta-calls, so `label/1`, `all_distinct/1`, and `ins/2` work
+unqualified through the wrapper. If a library predicate is somehow not found
+inside a wrapper (`existence_error`), qualify it: `clpfd:label(Xs)`.
+
+For shell commands, pass `exec(bash(command: Cmd, timeoutMs: N), R)` instead.
+
 ### Assert/Retract (within session)
 ```prolog
 agent_main :-
@@ -577,24 +712,97 @@ agent_main :-
     answer(Msg).
 ```
 
-### Constraint Logic Programming (CLP)
+### Constraint Logic Programming (CLP) — advanced, use sparingly
+
+Use CLP only for a **genuine constraint problem** (scheduling, assignment,
+puzzles, exact search). It is the wrong tool for ordinary arithmetic and
+transformations — prefer a `task/N`/`prompt/N` step when the numbers or the
+operation are open-ended (`DML_AUTHORING.md` §0). CLP also regresses easily: an
+under-specified model, or one that is simply **unsatisfiable**, can search
+essentially forever, so a buggy skill hangs the turn instead of failing fast.
+If you use it:
 
-DML supports Prolog's CLP libraries for solving optimization and logic problems without needing Python:
+- **Bound every domain first.** For one variable use `X in 0..100`; for a list use
+  `[X,Y,Z] ins 0..100`. (`X ins 0..100` on a bare variable is an
+  `instantiation_error` — use `in` for a single variable.)
+- `all_distinct/1` / `all_different/1` over the list constrains it pairwise.
+- Call `label/1` (or `labeling/2` with options such as `ff`) only when you need
+  concrete values; wrap it in `once/1` when a single witness is enough.
+- To bound a search that can run away, wrap it in `call_with_inference_limit/3`
+  (see *Bounding a goal* above):
+  `call_with_inference_limit(label(Xs), 500000, R)` — `R == inference_limit_exceeded`
+  when it gave up. This works for CLP(FD) predicates in this runtime.
+- Keep the model small; a big model is a sign the step belongs in a `task/N`.
 
-- **CLP(FD)**: Finite domains (integers). Use `:- use_module(library(clpfd)).`
-- **CLP(Q)**: Rational numbers. Use `:- use_module(library(clpq)).`
-- **CLP(R)**: Real numbers. Use `:- use_module(library(clpr)).`
+- **CLP(FD)**: finite domains (integers). `:- use_module(library(clpfd)).`
+- **CLP(Q)**: rationals. `:- use_module(library(clpq)).`
+- **CLP(R)**: reals. `:- use_module(library(clpr)).`
 
 ```prolog
 :- use_module(library(clpfd)).
 
-agent_main(Total) :-
-    [X,Y] ins 0..100,
+% Small, fully bounded search; one witness.
+solve_sum_product(Total, X, Y) :-
+    [X, Y] ins 0..100,      % bound the domains; never leave them open
     X + Y #= Total,
     X * Y #= 2400,
-    label([X,Y]),
-    format(string(Msg), "X=~w, Y=~w", [X, Y]),
-    answer(Msg).
+    once(label([X, Y])).
+
+% A search bounded by inference count instead of by the domain.
+bounded_label(Xs, Result) :-
+    call_with_inference_limit(label(Xs), 500000, Result).
+```
+
+#### Optimisation (LP / MIP)
+
+The CLP family here is **CLP(FD)** (integers), **CLP(Q)** (exact rationals) and
+**CLP(R)** (reals) — there is no `clpfq`. `library(simplex)` loads but returns
+unreliable answers in this runtime (its minimum/maximum come back wrong); do not
+build on it.
+
+**Continuous LP** — `clpr` (reals) / `clpq` (rationals), same interface. `{...}`
+posts the constraints; `maximize/1` / `minimize/1` find the optimum:
+
+```prolog
+:- use_module(library(clpr)).
+
+lp(X, Y) :-
+    {X >= 0, Y >= 0, X =< 10, Y =< 10, X + 2*Y =< 14},
+    maximize(X + Y).               % X = 10.0, Y = 2.0
+```
+
+> **Caveat:** `maximize/1` / `minimize/1` bind the variables only when the optimum
+> is a *unique vertex*. If the optimum is a face (a family of optima), they succeed
+> and bind nothing. Use `sup/2` / `inf/2` to get the optimum **value** reliably —
+> `{X >= 0, Y >= 0, X + Y =< 10, 2*X + Y =< 15}, sup(X + Y, S)` → `S = 10.0`.
+> (`inf/3` / `sup/3` do not exist in this build.)
+
+**Integer optimum** — negate the objective and use `bb_inf/5`; the 4th argument
+comes back as the integer vertex:
+
+```prolog
+:- use_module(library(clpr)).
+
+mip(Inf, Vertex) :-
+    {X >= 0, Y >= 0, X + Y =< 4, X + 3*Y =< 6},
+    bb_inf([X, Y], -(3*X + 2*Y), Inf, Vertex, 0.001).
+    % Inf = -12.0, Vertex = [4,0]     (negate the objective to maximise)
+```
+
+**Exact integer optimum** — CLP(FD) with a `max(Expr)` label option. Prefer this
+when the problem is genuinely integer: it is exact, bounded, and well-tested.
+Bound every domain first and keep `label`/`labeling` inside
+`call_with_inference_limit/3`:
+
+```prolog
+:- use_module(library(clpfd)).
+
+exact_max(X, Y) :-
+    [X, Y] ins 0..10,
+    X + Y #=< 4,
+    X + 3*Y #=< 6,
+    call_with_inference_limit(labeling([max(3*X + 2*Y), down], [X, Y]), 500000, _).
+    % X = 4, Y = 0
 ```
 
 ### Dictionaries (SWI-Prolog)
@@ -619,10 +827,8 @@ step(N, Desc) :-
     format(string(Msg), "Step ~w: ~w", [N, Desc]),
     output(Msg).
 
-% Main entry point
-agent_main :-
-    param(topic, "Research topic", Topic),
-    
+% Main entry point; the topic is an argument.
+agent_main(Topic) :-
     % Set up the assistant
     system("You are a thorough research assistant."),
     system("Provide detailed, accurate information."),
@@ -649,7 +855,7 @@ agent_main :-
 Run with:
 ```typescript
 for await (const event of dc.runDML(code, { 
-  params: { topic: 'quantum computing' } 
+  args: ['quantum computing'] 
 })) {
   console.log(event.type, event.content);
 }
@@ -718,6 +924,11 @@ with_judgment(jev, choose("state", "Which team?", [billing, orders], Team)).
 require_judgment(calibrated, verify("state", "Is this urgent?", Truth)).
 ```
 
+> **In the DML REPL**, an `llm` backend is always registered (it reuses the
+> active model), and `jev` is registered when `judgment.jev.enabled` is set in
+> `.deepclause/config.json` and its API key is available. `with_judgment(jev,
+> ...)` fails if `jev` is not registered.
+
 `require_judgment/2` fails the wrapped goal before it runs when the active
 backend does not provide the capability, so a skill that depends on calibrated
 probabilities cannot silently run on an uncalibrated backend. Available
@@ -734,4 +945,33 @@ backtracking reuses an answer instead of re-querying the model.
 - Values are constrained to the options/levels you supply. A `choose` answer is
   always one of your option atoms.
 - Prefer one `judge/2` batch over several one-off calls when you need more than
-  one or two judgments.
+  one or two judgments.
+## The DML REPL (`deepclause repl`)
+
+DML skills can also be grown interactively in a persistent **living module**.
+
+- `/goal <dml>` runs a goal and enumerates its solutions (the legacy dotted
+  `goal: <dml>.` form still works).
+- Plain English builds the module: the **authoring agent** inspects and edits it
+  through tools (`listing`, `begin_turn`, `define`/`assert`/`assert_many`,
+  `retract`/`delete`, `execute`, `end_turn`/`rollback`). Edits are transactions:
+  `begin_turn` is required before any edit, `end_turn` commits a version.
+  Nothing is auto-run; the agent uses `execute` to test during a turn.
+- There is **no implicit `agent_main` requirement** in the REPL. Run any
+  predicate with `/goal <name>(…)`. Define `agent_main` in the module when it
+  should also run with `deepclause run` / `/dc-run`.
+- `freeze.` makes the module read-only; `export <file>.` serializes it.
+  `export harness.zip` bundles the module with its linked script files;
+  `export harness.dml` inlines them into one portable file.
+- `input/2` / `exec(ask_user(...))` (module) ask the human. They never block in
+  non-interactive runs (`-g`, piped stdin, CI).
+
+**Self-containment.** The harness is the `.dml` **plus the files it links**.
+Materialize each program with a DML goal (`make_directory_path(...)`,
+`write_file(...)`), link it with `script_file(Name, Path)`, and run it with
+`exec(bash(...))`. The
+exporter bundles the linked files, so they travel with the harness. Pass
+untrusted data through files or stdin, never interpolated into the shell
+command.
+
+See [`docs/REPL.md`](../../../../docs/REPL.md) for the full guide.