deepclause-sdk 0.1.3 → 0.2.1

package/dist/system/assets/docs/AGENT_PROMPT.md

@@ -189,6 +189,54 @@ than performing it as an edit.
 Other tools: `reference(topic)` reads the DeepClause docs (`language`,
 `authoring`, `patterns`, `examples`, `snippets`, `tools`).
 
+# Concurrency and subagents
+
+A **subagent** is a session-scoped DML process that keeps running after this
+turn. Reach for one when work is long-running, must wait for messages, or can
+run in parallel — not when a single `task/N` or `execute` call does the job.
+
+| The work is… | Use |
+|---|---|
+| one model call, or a quick goal | `task/N` / `execute` |
+| the same goal over a list, in parallel | `swarm/3` / `pmap/3` |
+| long, and the user can wait | `spawn(Goal)` — report the id and carry on |
+| a long-lived message handler | `serve(Name, Loop)` + `service_call`/`service_cast` |
+| a decision from the user mid-run | `input/2` inside the subagent |
+| several specialised roles collaborating | `serve` each role; a coordinator wires them |
+
+A subagent is isolated and asynchronous:
+
+- **Private snapshot.** Its `assert`/`retract` do not touch the harness, and it
+  does not see commits made after it started.
+- **Value semantics.** `spawn` and `send` copy terms. A child cannot bind a
+  variable in the parent, so a result comes back by `send/2` or as the child's
+  `answer/1` — never by an output argument. Write `send(Parent, result(...))` in
+  the child and `receive` it in the parent (pass `self(Me)` into the child).
+- **It outlives the turn.** Do not busy-wait; report the id and use `ps` later.
+
+Patterns:
+
+- **Fan out over a list** with `swarm(Worker, Items, Results)` — ordered, one
+  process per item (at most 4 at once), `ok(Value)`/`failed` per item — or
+  `pmap/3` (bare values, fails on any failure). Bound the list: each item is a
+  process and usually a model call, and the session has a process limit.
+- **Background a long goal** with `spawn(Goal)`. If you need the result, have the
+  goal `send(Parent, result(...))` and `receive` it, or make it a service. A
+  one-shot subagent's terminal result can also be read with `wait(Pid, ok(Text))`.
+- **A service** is `serve(Name, Loop)` where `Loop` handles
+  `request(From, Ref, Request)` by `send(From, reply(Ref, Reply))`, then recurses.
+  Call it with `service_call(Name, Request, Reply)`.
+- **Recursive delegation** is a subagent that spawns subagents; bound the depth.
+
+Subagent tools: `spawn(goal, name?)`, `ps`, `send(to, message)`,
+`call(to, message)`, `stop(target)`, `swarm(worker, items)`, `wait(pid)`. Users
+also have `/spawn`, `/send`, `/call`, `/stop`, and `/ps --watch --logs` (a live
+follow of every subagent's streamed output, reasoning, and tool calls).
+
+Do not spawn a subagent for a single model call, and do not fan out an unbounded
+list. If the work needs the harness's shared state, or the user's answer now,
+keep it in the foreground.
+
 End your turn by replying with text and no tool call. To ask the human a
 question, write it in that final message.
 
@@ -754,6 +802,7 @@ More patterns: `reference("patterns")`; shapes: `reference("examples")`.
   - [system/1, user/1](#memory-predicates)
   - [output/1, yield/1, log/1](#output-predicates)
   - [answer/1](#answer1)
+- [Processes and concurrency](#processes-and-concurrency)
 - [Control Flow](#control-flow)
 - [String Interpolation](#string-interpolation)
 - [Backtracking](#backtracking)
@@ -1624,6 +1673,150 @@ agent_main :-
 
 ---
 
+## Processes and concurrency
+
+A **process** (a "subagent") is an isolated DML execution with its own private
+memory, a private snapshot of this module, and a mailbox. Processes outlive the
+turn that started them, so a harness can background long work, wait for
+messages, or run many things at once.
+
+Process primitives work anywhere a goal runs, including inside a `tool/2` body:
+a tool executes in its own engine, but `self/1` is the process the tool runs in,
+so a tool can route work to other processes.
+
+A process runs the harness **as of its spawn**: later edits to a predicate do not
+reach a process already running, while a process spawned afterwards sees them.
+Inspect the code a process runs with `ps(code: "<pid or name>")`, and the
+session's current harness version with `ps(code: "")`.
+
+### `spawn(+Goal, -Pid)` / `spawn_opt(+Goal, -Pid, +Options)`
+
+Start `Goal` in a new process and return its pid immediately. `Goal` is **copied**:
+variables in the child are fresh, so a child can never bind a variable in the
+parent. A result comes back by message (`send/2`) or as the child's `answer/1`.
+
+```prolog
+worker(Topic, Parent) :-
+    task("Summarise {Topic} in one sentence.", string(Summary)),
+    send(Parent, done(Topic, Summary)),
+    answer(Summary).
+
+fan_out(Topics, Results) :-
+    self(Me),
+    maplist(spawn_worker(Me), Topics, _Pids),
+    collect(Topics, Results).
+
+spawn_worker(Me, Topic, Pid) :- spawn(worker(Topic, Me), Pid).
+
+collect([], []).
+collect([Topic|Topics], [Topic-Summary|Rest]) :-
+    receive(done(Topic, Summary), 300000),
+    collect(Topics, Rest).
+```
+
+`spawn_opt/3` takes a dict of options: `name`, `priority`
+(`foreground`/`interactive`/`background`), `timeout` (ms), and `code`.
+
+The goal's shape decides the lifecycle:
+
+- **one-shot** — calls `output/1`/`answer/1` and returns.
+- **message-waiting** — loops on `receive/1,2`, handling terms from other
+  processes or from the user.
+- **interactive** — calls `input/2` and blocks until the user answers.
+
+### `send(+Target, +Message)` / `receive(?Message[, +Timeout])`
+
+`send/2` copies a term into another process's mailbox; `Target` is a pid, a
+registered name, or a process label. `receive/2` pops the oldest message and
+fails on timeout. `receive_match/3` pops the oldest message that unifies with a
+template, requeues the rest, and fails on timeout.
+
+```prolog
+receive(request(From, Ref, Query), 30000),
+handle(Query, Answer),
+send(From, reply(Ref, Answer)).
+```
+
+### `self/1`, `register/2`, `whereis/2`
+
+`self(Pid)` is this process's pid. `register(Name, Pid)` names a process in the
+session; `whereis(Name, Pid)` resolves it. A name is released when the process
+exits. A process's default label is its goal's functor, so `send(keeper, stop)`
+reaches an unregistered `keeper/…` process.
+
+### `serve(+Name, +Goal)` / `service_call(+Target, +Request, -Reply[, +Timeout])` / `service_cast(+Target, +Message)`
+
+`serve/2` is `spawn/2` + `register/2`. `service_call` sends
+`request(Self, Ref, Request)` and waits for `reply(Ref, Reply)`, ignoring other
+messages; `service_cast` sends without waiting. A service is an ordinary
+`receive` loop:
+
+```prolog
+counter_service(Count) :-
+    receive(Msg),
+    (   Msg = request(From, Ref, read)  -> send(From, reply(Ref, Count)), counter_service(Count)
+    ;   Msg = request(From, Ref, bump)  -> Next is Count + 1, send(From, reply(Ref, Next)), counter_service(Next)
+    ;   Msg = stop                       -> true
+    ;   counter_service(Count)
+    ).
+```
+
+### `swarm(+Worker, +Items, -Results[, +Concurrency])` / `pmap(+Worker, +Items, -Values)`
+
+Run `call(Worker, Item, Value)` once per item, each in its own process, and
+collect the results **in input order**. At most `Concurrency` items run at once
+(default 4), so a large list does not fire every model call simultaneously.
+`swarm` returns `ok(Value)`/`failed` per item; `pmap` returns bare values and
+fails if any item failed.
+
+```prolog
+swarm(summarise_file, Files, Results, 4),
+findall(S, member(ok(S), Results), Summaries).
+```
+
+### `wait(+Pid[, -Outcome])`
+
+Block until a process terminates, then report what happened. `Outcome` is
+`ok(Result)`, `failed(Error)`, or `killed(Reason)`; `Result` is the process's
+terminal text as a Prolog string. `wait/1` is semidet — it succeeds only on a
+successful finish.
+
+```prolog
+spawn(research("qec", Me), Pid),
+wait(Pid, ok(Text)),          % Text is the subagent's terminal text
+```
+
+For a structured result, prefer the message form: have the child
+`send(Parent, result(Term))` and `receive` it in the parent.
+
+### `exit/1,2` and stopping
+
+`exit(Reason)` terminates the current process. `exit(Pid, Reason)` terminates
+another (a registered name or a process label works too); there is no `stop/1`
+predicate, so `exit/2` is how DML cancels a peer. (The `/stop` command and the
+`stop` tool are the interactive equivalents.) `process_info(Pid, Info)` returns a
+dict (`pid`, `name`, `state`, `priority`, `mailbox`, `usage`, …). `sleep(Ms)`
+yields; `now_ms(T)` and `make_ref(Ref)` are available.
+
+### Rules
+
+- **Isolation.** A process runs against a private module snapshot: its
+  `assert`/`retract` do not affect the harness, and it does not see commits made
+  after it started.
+- **Value semantics.** `spawn` and `send` copy terms; never rely on a shared
+  variable with a child.
+- **Determinism.** A process that calls `task/N`, `exec/2`, `read_media/2`, or
+  writes a file should yield one solution (`once/1`, `->`, `!`).
+- **Bound the work.** Every `swarm` item is a process and usually a model call;
+  the session has a **live-process limit** (default 16 concurrent — finished
+  processes do not count, so a long session can fan out again) and the provider
+  has rate limits. A `swarm` item whose process cannot be spawned is reported as
+  `failed(...)`; `pmap` then fails.
+- **CPU-bound Prolog** blocks the in-process host (L1). Long compute belongs in
+  the REPL's `--execution worker` (L2), not a `swarm` on the default host.
+
+---
+
 ## Control Flow
 
 DML supports standard Prolog control flow:

package/dist/system/assets/docs/DML_AUTHORING.md

@@ -272,3 +272,24 @@ text-only — never attach media to them.
   `/dc-run` skill.
 - Media: reference images with `media(Path, Mime)` (`read_media/2` +
   `attach_media/1`); never inline bytes.
+
+## 7. Concurrency and subagents
+
+Most skills are a short sequence of `task/N` calls in the foreground. Reach for a
+**subagent** only when the work must outlive the turn, wait for a message, or run
+in parallel.
+
+- **Fan out over a list** (`swarm/3`, `pmap/3`): the same worker over many items,
+  each in its own process, results in input order. Bound the list.
+- **Background a long goal** (`spawn/2`): report the id and carry on; the user
+  watches it with `/ps --watch --logs`.
+- **A service** (`serve/2` + `service_call/3`): a long-lived `receive` loop that
+  answers requests; also good for a single-writer or rate-limited step.
+
+A subagent is isolated (private module snapshot) and asynchronous (value
+semantics): its `assert`/`retract` do not touch the harness, and a result comes
+back by `send/2` or the child's `answer/1`, never by an output argument. Prefer
+`swarm` over hand-rolled `spawn`+`receive` unless you need a custom protocol.
+
+Do not fan out an unbounded list (each item is a process and usually a model
+call), and do not spawn for a single model call.

package/dist/system/assets/docs/DML_REFERENCE.md

@@ -15,6 +15,7 @@
   - [system/1, user/1](#memory-predicates)
   - [output/1, yield/1, log/1](#output-predicates)
   - [answer/1](#answer1)
+- [Processes and concurrency](#processes-and-concurrency)
 - [Control Flow](#control-flow)
 - [String Interpolation](#string-interpolation)
 - [Backtracking](#backtracking)
@@ -885,6 +886,150 @@ agent_main :-
 
 ---
 
+## Processes and concurrency
+
+A **process** (a "subagent") is an isolated DML execution with its own private
+memory, a private snapshot of this module, and a mailbox. Processes outlive the
+turn that started them, so a harness can background long work, wait for
+messages, or run many things at once.
+
+Process primitives work anywhere a goal runs, including inside a `tool/2` body:
+a tool executes in its own engine, but `self/1` is the process the tool runs in,
+so a tool can route work to other processes.
+
+A process runs the harness **as of its spawn**: later edits to a predicate do not
+reach a process already running, while a process spawned afterwards sees them.
+Inspect the code a process runs with `ps(code: "<pid or name>")`, and the
+session's current harness version with `ps(code: "")`.
+
+### `spawn(+Goal, -Pid)` / `spawn_opt(+Goal, -Pid, +Options)`
+
+Start `Goal` in a new process and return its pid immediately. `Goal` is **copied**:
+variables in the child are fresh, so a child can never bind a variable in the
+parent. A result comes back by message (`send/2`) or as the child's `answer/1`.
+
+```prolog
+worker(Topic, Parent) :-
+    task("Summarise {Topic} in one sentence.", string(Summary)),
+    send(Parent, done(Topic, Summary)),
+    answer(Summary).
+
+fan_out(Topics, Results) :-
+    self(Me),
+    maplist(spawn_worker(Me), Topics, _Pids),
+    collect(Topics, Results).
+
+spawn_worker(Me, Topic, Pid) :- spawn(worker(Topic, Me), Pid).
+
+collect([], []).
+collect([Topic|Topics], [Topic-Summary|Rest]) :-
+    receive(done(Topic, Summary), 300000),
+    collect(Topics, Rest).
+```
+
+`spawn_opt/3` takes a dict of options: `name`, `priority`
+(`foreground`/`interactive`/`background`), `timeout` (ms), and `code`.
+
+The goal's shape decides the lifecycle:
+
+- **one-shot** — calls `output/1`/`answer/1` and returns.
+- **message-waiting** — loops on `receive/1,2`, handling terms from other
+  processes or from the user.
+- **interactive** — calls `input/2` and blocks until the user answers.
+
+### `send(+Target, +Message)` / `receive(?Message[, +Timeout])`
+
+`send/2` copies a term into another process's mailbox; `Target` is a pid, a
+registered name, or a process label. `receive/2` pops the oldest message and
+fails on timeout. `receive_match/3` pops the oldest message that unifies with a
+template, requeues the rest, and fails on timeout.
+
+```prolog
+receive(request(From, Ref, Query), 30000),
+handle(Query, Answer),
+send(From, reply(Ref, Answer)).
+```
+
+### `self/1`, `register/2`, `whereis/2`
+
+`self(Pid)` is this process's pid. `register(Name, Pid)` names a process in the
+session; `whereis(Name, Pid)` resolves it. A name is released when the process
+exits. A process's default label is its goal's functor, so `send(keeper, stop)`
+reaches an unregistered `keeper/…` process.
+
+### `serve(+Name, +Goal)` / `service_call(+Target, +Request, -Reply[, +Timeout])` / `service_cast(+Target, +Message)`
+
+`serve/2` is `spawn/2` + `register/2`. `service_call` sends
+`request(Self, Ref, Request)` and waits for `reply(Ref, Reply)`, ignoring other
+messages; `service_cast` sends without waiting. A service is an ordinary
+`receive` loop:
+
+```prolog
+counter_service(Count) :-
+    receive(Msg),
+    (   Msg = request(From, Ref, read)  -> send(From, reply(Ref, Count)), counter_service(Count)
+    ;   Msg = request(From, Ref, bump)  -> Next is Count + 1, send(From, reply(Ref, Next)), counter_service(Next)
+    ;   Msg = stop                       -> true
+    ;   counter_service(Count)
+    ).
+```
+
+### `swarm(+Worker, +Items, -Results[, +Concurrency])` / `pmap(+Worker, +Items, -Values)`
+
+Run `call(Worker, Item, Value)` once per item, each in its own process, and
+collect the results **in input order**. At most `Concurrency` items run at once
+(default 4), so a large list does not fire every model call simultaneously.
+`swarm` returns `ok(Value)`/`failed` per item; `pmap` returns bare values and
+fails if any item failed.
+
+```prolog
+swarm(summarise_file, Files, Results, 4),
+findall(S, member(ok(S), Results), Summaries).
+```
+
+### `wait(+Pid[, -Outcome])`
+
+Block until a process terminates, then report what happened. `Outcome` is
+`ok(Result)`, `failed(Error)`, or `killed(Reason)`; `Result` is the process's
+terminal text as a Prolog string. `wait/1` is semidet — it succeeds only on a
+successful finish.
+
+```prolog
+spawn(research("qec", Me), Pid),
+wait(Pid, ok(Text)),          % Text is the subagent's terminal text
+```
+
+For a structured result, prefer the message form: have the child
+`send(Parent, result(Term))` and `receive` it in the parent.
+
+### `exit/1,2` and stopping
+
+`exit(Reason)` terminates the current process. `exit(Pid, Reason)` terminates
+another (a registered name or a process label works too); there is no `stop/1`
+predicate, so `exit/2` is how DML cancels a peer. (The `/stop` command and the
+`stop` tool are the interactive equivalents.) `process_info(Pid, Info)` returns a
+dict (`pid`, `name`, `state`, `priority`, `mailbox`, `usage`, …). `sleep(Ms)`
+yields; `now_ms(T)` and `make_ref(Ref)` are available.
+
+### Rules
+
+- **Isolation.** A process runs against a private module snapshot: its
+  `assert`/`retract` do not affect the harness, and it does not see commits made
+  after it started.
+- **Value semantics.** `spawn` and `send` copy terms; never rely on a shared
+  variable with a child.
+- **Determinism.** A process that calls `task/N`, `exec/2`, `read_media/2`, or
+  writes a file should yield one solution (`once/1`, `->`, `!`).
+- **Bound the work.** Every `swarm` item is a process and usually a model call;
+  the session has a **live-process limit** (default 16 concurrent — finished
+  processes do not count, so a long session can fan out again) and the provider
+  has rate limits. A `swarm` item whose process cannot be spawned is reported as
+  `failed(...)`; `pmap` then fails.
+- **CPU-bound Prolog** blocks the in-process host (L1). Long compute belongs in
+  the REPL's `--execution worker` (L2), not a `swarm` on the default host.
+
+---
+
 ## Control Flow
 
 DML supports standard Prolog control flow:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "deepclause-sdk",
-  "version": "0.1.3",
+  "version": "0.2.1",
   "description": "Neuro-symbolic runtime: a self-modifying DML/Prolog REPL, one-shot runner, and TypeScript SDK",
   "type": "module",
   "main": "dist/index.js",
@@ -38,6 +38,7 @@
     "examples": "tsx sdk-examples/basic-usage.ts",
     "examples:smoke": "tsx sdk-examples/no-api-key.ts && npm run example && npm run examples && tsx sdk-examples/neurosymbolic.ts 1 && tsx sdk-examples/deep-research.ts \"test topic\"",
     "cli": "tsx src/cli/index.ts",
+    "repl": "tsx src/cli/index.ts repl",
     "sync:agent-prompt": "node scripts/sync-agent-prompt.mjs",
     "check:agent-prompt": "node scripts/sync-agent-prompt.mjs --check"
   },