deepclause-pi 0.1.4 → 0.1.5

package/docs/BLOG_POST_PI_EXTENSION.md

@@ -0,0 +1,253 @@
+# DeepClause meets pi: executable agent plans inside your coding session
+
+### DML orchestration with pi’s models, tools and context
+
+`tldr;` I built a [pi](https://github.com/badlogic/pi-mono) extension for [DeepClause](https://github.com/deepclause/deepclause-sdk). It runs DML programs with pi’s active model, credentials and session context. It can also turn a normal planning request into an executable DML plan. The plan can delegate individual steps back to pi, with a fixed set of tools for each step.
+
+The extension is available here: [https://github.com/deepclause/deepclause-pi](https://github.com/deepclause/deepclause-pi)
+
+[The following is mostly written by a real person ;-]
+
+I have spent quite a bit of time using DeepClause as a standalone agent runtime. That works well for benchmarks and self-contained workflows, but it leaves out something useful: the environment of a coding agent that is already running.
+
+A coding agent such as pi already has:
+
+- a selected model and working authentication
+- the current conversation and compacted session history
+- repository instructions and loaded skills
+- file, shell and extension tools
+- a terminal UI, cancellation and usage accounting
+
+Reimplementing all of that in DeepClause would make little sense. The more useful option is to let pi remain the host and use DeepClause for the part it is good at: explicit orchestration.
+
+This is what the new extension does.
+
+[Image: `/dc` status and command overview]
+
+### Running DML inside pi
+
+The basic case is simple. DML programs live in `.pi/deepclause/skills/` and can be executed with a slash command:
+
+```text
+/dc-run example --debug
+```
+
+The extension uses the model currently selected in pi. It does not ask for another API key or try to guess the provider. Model calls, cancellation, input prompts and usage stay connected to the current pi session.
+
+A minimal skill still looks like ordinary DML:
+
+```prolog
+agent_main(Topic) :-
+    system("You are a concise technical analyst."),
+    format(string(Request),
+           "Explain ~w. Store the final explanation in Summary.",
+           [Topic]),
+    task(Request, string(Summary)),
+    answer(Summary).
+```
+
+Run it as follows:
+
+```text
+/dc-run skills/explain.dml "constraint logic programming"
+```
+
+There are three context modes:
+
+- `turn` imports the current request and its immediate context
+- `branch` imports a bounded part of the active session branch
+- `isolated` starts without pi conversation history
+
+For example:
+
+```text
+/dc-run skills/explain.dml "constraint logic programming" --context=isolated
+```
+
+Pi remains the session owner. DeepClause does not create another chat history next to it.
+
+### Tools are deliberately boring
+
+Ordinary DML programs do not receive pi’s complete tool registry. They start with two narrow host operations:
+
+- `pi_workspace_list` lists one directory level inside the workspace
+- `pi_bash` runs an approved command inside the workspace
+
+Every `pi_bash` call requires user confirmation. Paths are checked against the active workspace, including resolved symlinks.
+
+A DML program can wrap these operations in a more useful tool predicate:
+
+```prolog
+tool(bing_search(Query, Results),
+     "Search Bing RSS and return the response body") :-
+    format(string(QueryArg), "q=~w", [Query]),
+    exec(pi_bash("curl", [
+        "--fail", "--silent", "--show-error", "--location", "--get",
+        "--data-urlencode", QueryArg,
+        "https://www.bing.com/search?format=rss&count=8"
+    ]), Result),
+    get_dict(stdout, Result, Results).
+```
+
+The repository includes a small deep-research example built on this pattern. It asks the user to review a search plan through pi’s input UI, runs approved `curl` requests and produces a cited report.
+
+I intentionally did not expose all coding tools directly through `exec/2`. Pi extensions can describe their tools, but there is no public generic API for another extension to invoke any tool by name. More importantly, copying tool execution into DeepClause would bypass behavior owned by pi or another extension, such as approval dialogs and policy checks.
+
+This becomes relevant for planning.
+
+### DML is the plan
+
+The extension adds this command:
+
+```text
+/dc-plan <request> [--name=slug]
+```
+
+For example:
+
+```text
+/dc-plan build a small Three.js Space Invaders game and verify it --name=threejs-space-invaders
+```
+
+This starts a normal pi turn. The planner can inspect the repository, read its instructions, see loaded skills and consider the currently active tools. It does not directly return a Markdown checklist and it does not generate arbitrary DML source in one shot.
+
+Instead, the planning turn finishes by calling a temporary tool named `dc_plan_commit`. The tool accepts a typed plan specification with:
+
+- a title and objective
+- ordered steps
+- an executor for each step
+- exact required tool names
+- relevant pi skills
+- an expected result for every step
+- a fallback message
+
+The extension validates this object and shows a preview. After user confirmation, it deterministically assembles the DML file, parses the generated program and writes it under `.pi/deepclause/plans/` without overwriting an existing file.
+
+This follows the same general idea as the planner used in my DeepPlanning experiments: ask the model for a constrained intermediate representation and let normal code produce the executable DML. This is simpler and more reliable than asking the model to get every comma, variable and fallback clause right.
+
+A generated plan looks like this:
+
+```prolog
+agent_main :-
+    output("Step 1/3: Inspect the workspace"),
+    exec(pi_agent_step(
+        instruction: "Inspect repository instructions and identify the target app.",
+        tools: ["bash", "read"],
+        expected: "A target directory and concrete implementation baseline.",
+        skills: []
+    ), Step1Summary),
+    Step1Summary \= "",
+
+    output("Step 2/3: Implement the application"),
+    exec(pi_agent_step(
+        instruction: "Implement the agreed application and keep changes scoped.",
+        tools: ["read", "write", "edit"],
+        expected: "A runnable implementation with a concise change summary.",
+        skills: []
+    ), Step2Summary),
+    Step2Summary \= "",
+
+    output("Step 3/3: Validate the result"),
+    exec(pi_agent_step(
+        instruction: "Run the relevant checks and fix implementation failures.",
+        tools: ["bash", "read", "write", "edit"],
+        expected: "Passing checks or a precise account of remaining failures.",
+        skills: []
+    ), Step3Summary),
+    Step3Summary \= "",
+
+    answer([Step1Summary, Step2Summary, Step3Summary]).
+```
+
+The full file is executable. There is no Markdown-to-DML compilation step in the pi integration.
+
+[Image: generated DML plan in the editor]
+
+### What `pi_agent_step` does
+
+`pi_agent_step` is the bridge between DML orchestration and a normal pi coding turn.
+
+When the DML runtime reaches one of these calls, the extension:
+
+1. checks that every named tool is installed and currently active
+2. rejects DeepClause control tools to prevent recursive planning or execution
+3. saves pi’s current active-tool set
+4. temporarily activates only the tools named by the step
+5. sends the bounded instruction through a normal pi turn
+6. captures the final textual summary and tool failures
+7. restores the previous active-tool set on success, failure or cancellation
+
+The important point is that pi still executes the turn. A third-party extension tool therefore keeps its own UI, approvals and policy. DeepClause only controls which tools are available to that particular step and what should happen next.
+
+The user must start such a plan explicitly:
+
+```text
+/dc-run plans/threejs_space_invaders.dml
+```
+
+Before execution, the extension shows the required tools and asks for confirmation.
+
+The optional model-callable `dc_run` tool cannot execute contextual plans. Allowing an agent turn to start a plan that starts more agent turns would make recursion and session ownership needlessly difficult. For this first version, the boundary is simple: reusable self-contained skills may be model-called after explicit enablement; contextual plans are user-called.
+
+### A concrete example
+
+I used `/dc-plan` to create a five-step plan for a small Three.js Space Invaders game. The generated DML fixes the execution order:
+
+1. inspect the workspace and choose the integration boundary
+2. create the browser application foundation
+3. implement deterministic game state and collision logic
+4. connect the state to rendering, controls and UI
+5. run tests and build the production bundle
+
+Each step has a different tool set. Inspection gets `bash` and `read`. Pure implementation steps get `read`, `write` and `edit`. Validation gets all four. The plan records these requirements before any work starts.
+
+This does not guarantee a correct game. The model can still write bad code, misunderstand an API or produce an incomplete summary. What it does guarantee is a more explicit execution structure. The agent cannot quietly skip from initial inspection to a confident final answer without the intervening DML goals succeeding.
+
+This is the same reason I find DML useful with smaller models. Long context alone does not make long-running execution reliable. An explicit program can carry the sequence, checks, retries and fallback while the model deals with the parts that actually require judgment.
+
+### Installation
+
+The current release is `0.1.2` and depends on `deepclause-sdk` `0.0.87`.
+
+Install directly from GitHub:
+
+```sh
+pi install git:github.com/deepclause/deepclause-pi
+```
+
+Or install it for one project:
+
+```sh
+pi install git:github.com/deepclause/deepclause-pi -l
+```
+
+For a project-local installation, start pi in the directory containing `.pi/settings.json`. The startup screen should list the DeepClause extension. If it does not, `/dc-run` will be treated as an ordinary message and sent to the model.
+
+Useful commands:
+
+```text
+/dc
+/dc-list
+/dc-plan inspect this repository and propose a safe migration --name=migration
+/dc-run plans/migration.dml
+/dc-tool enable
+/dc-cancel
+```
+
+`dc_run` is disabled by default. `/dc-tool enable` makes it available to the model for that workspace. This does not grant ordinary DML programs access to all pi tools.
+
+### Some more thoughts and notes
+
+1. This is an early integration. The main execution path works and has tests for plan creation, tool preflight, contextual delegation, cancellation and tool restoration. There are still plenty of rough edges to find in real repositories.
+
+2. `task/N` supports stream events, but the current pi model adapter receives a completed model response and forwards it as one chunk. Tool activity, DML progress, input requests and usage are already live.
+
+3. Generated plans currently record tool names, but not stable hashes of their schemas. A tool can change between planning and execution. The runtime catches missing or inactive tools, while deeper schema-drift checks can be added later.
+
+4. DML backtracking does not undo external effects. If a plan writes a file and later fails, Prolog can try another clause, but the file is still there. Generated coding plans therefore use ordered steps and explicit summaries rather than pretending that side effects are transactional.
+
+5. Does this produce better coding results than an unconstrained pi turn? I have some good examples, but no benchmark yet. The useful claim for now is narrower: it gives us an executable, inspectable plan with a clear boundary around context and tools. Please try it and report what breaks.
+
+Repository: [https://github.com/deepclause/deepclause-pi](https://github.com/deepclause/deepclause-pi)
+
+DeepClause SDK: [https://github.com/deepclause/deepclause-sdk](https://github.com/deepclause/deepclause-sdk)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "deepclause-pi",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "description": "Pi-hosted runtime for DeepClause DML programs",
   "type": "module",
   "license": "MIT",
@@ -25,11 +25,15 @@
   "files": [
     "dist",
     "docs",
-    "src"
+    "src",
+    "skills"
   ],
   "pi": {
     "extensions": [
       "./src/index.ts"
+    ],
+    "skills": [
+      "./skills"
     ]
   },
   "scripts": {

package/skills/handbook-dml/SKILL.md

@@ -0,0 +1,265 @@
+---
+name: handbook-dml
+description: Convert a long handbook/SOP into one DeepClause DML skill per workflow — an LLM-first subagent (agentic task/N leaves, narrow tool/2 capabilities) that takes a generic natural-language request as input and uses deterministic Prolog only for mechanical checks, with LLM fallback. Also teaches a tool audit, user confirmation, and how to write the repo-root AGENTS.md policy-routing table so pi calls the skills automatically via dc_run. Use when asked to turn a handbook or procedures manual into executable DML, update a handbook-derived skill, or wire the policy router.
+---
+
+# Handbook → DML (procedures)
+
+Turn a long handbook into **one DML skill per workflow/procedure**. Each skill is
+an **LLM-first subagent**: agentic `task/N` leaves do the reading, reasoning,
+and acting, while Prolog handles only what is genuinely mechanical (arithmetic,
+counting, exact equality) or a hard safety invariant.
+
+Before writing DML, read `.pi/deepclause/AGENTS.md` and
+`.pi/deepclause/DML_REFERENCE.md`. They are authoritative for syntax.
+
+## Mental model
+
+- **Pi authors; DML is the runtime artifact.** Decomposition and authoring happen
+  in a normal pi turn. There is no Markdown→DML compiler.
+- **LLM-first.** Rules and flow are `task/N` / `prompt/N` by default. Use Prolog
+  only where a rule is mechanical or must never be wrong.
+- **Input is a generic request.** `agent_main(Request)` takes free text; the
+  first step is an LLM `task/N` that parses it into a typed `object/1` case (or
+  the request is used directly for simple skills).
+- **Forbidden actions become tool scoping.** "Never send" means *no send tool*.
+  "Read-only inspection" means the inspect phase gets read tools only.
+- **Every deterministic rule gets an LLM fallback.** If a deterministic check
+  fails, branch to a `prompt/N`/`task/N` that reviews, repairs, or asks the user
+  — do not hard-fail.
+- **Ask, don't assume.** Confirm scope, the decomposition, and tool choices with
+  the user before authoring.
+
+## Workflow
+
+1. **Ingest** the handbook to Markdown (PDF→`pdftotext`, DOCX/HTML→`pandoc`).
+   Keep the source under `examples/handbook-md/<handbook_slug>/`.
+2. **Clarify** with the user: which workflows to convert, what the input looks
+   like (a request? a file? a case id?), and what the skill may change.
+3. **Decompose** by the document's own workflows and **propose the
+   section → skill map; get approval** before writing files.
+4. **Tool audit** (below): list each procedure's required capabilities, check
+   what is available, flag gaps, and ask whether dummy tools are acceptable.
+5. **Author** one DML per workflow from the template below.
+6. **Run and iterate** with the user's real input:
+   `/dc-run skills/<handbook_slug>/<slug>.dml "<request>" --debug`.
+7. **Write `INDEX.md`** and **add a row to the repo-root `AGENTS.md`** routing
+   table.
+
+## Input convention
+
+The skill takes a **generic natural-language request**, not positional args and
+not hardcoded case facts.
+
+- Default: parse the request with an LLM `task/N` into a typed `object/1` case.
+- Simpler skills: pass the request text directly into the first `task/N` and
+  skip the extraction step.
+
+```prolog
+extract_case(Request, Case) :-
+    format(string(Desc),
+        "Extract a structured case from this request: ~w. Store an object with keys <fields>. Use null for unknown fields.",
+        [Request]),
+    task(Desc, object(Case)).
+```
+
+`object(Var)` is a supported typed output; read it with `get_dict/3`. Then
+confirm with the user before acting. User interaction loops belong **inside a
+`task/N`**, with `ask_user` exposed as a tool:
+
+```prolog
+tool(user_feedback(Prompt, Response), "Ask the user one focused question and return their response") :-
+    exec(ask_user(prompt: Prompt), Result),
+    get_dict(user_response, Result, Response).
+
+confirm_case(Case, ConfirmedCase) :-
+    format(string(Instruction),
+        "Present this extracted case to the user with the user_feedback tool: ~w. Ask them to type 'ok' or describe corrections. If 'ok', return the case unchanged. Otherwise apply corrections and ask again, up to 3 rounds. Store the final case in ConfirmedCase.",
+        [Case]),
+    with_tools([user_feedback], (
+        task(Instruction, object(ConfirmedCase))
+    )).
+```
+
+## Tool audit
+
+Before authoring, list what the procedure needs and check what actually exists.
+
+| Procedure need | Real option | Dummy fallback |
+| --- | --- | --- |
+| email / Slack / calendar / Jira / Shopify | pi tools (if installed & active) or MCP | `tool/2` over `pi_bash` (files) or over facts |
+| read spreadsheets / PDFs / CSVs | pi tools or `pi_bash` | `tool/2` reading fixture files |
+| ask the user | `ask_user` (always available in DML) | — |
+| arbitrary shell | `pi_bash` (approval-gated) | — |
+
+The DML runtime itself exposes only `pi_workspace_list`, `pi_bash`, and
+`ask_user`. Everything else (email, Slack, calendar, structured file readers) is
+missing unless you add it as a DML `tool/2`. So:
+
+1. For each required capability, note whether a real tool exists.
+2. If it is missing, **tell the user and ask: "use a dummy tool for X?"** before
+   authoring. Do not silently build a dummy.
+3. Record the chosen substitution in the skill header and in `INDEX.md`.
+
+## Light template
+
+```prolog
+% POLICY: <slug>
+% Handbook : <handbook_slug> (<title>)
+% Trigger  : <when to run>
+% Input    : a natural-language request (parsed by the first task)
+% Effects  : <what state this changes, if any>
+% Tools    : <real or dummy, per the tool audit>
+%
+% Run: /dc-run skills/<handbook_slug>/<slug>.dml "<request>"
+
+% --- tools ----------------------------------------------------------------
+% Read tools for inspection; write tools for action; omit forbidden actions.
+tool(<read_...>(Args, Out), "Description") :- ... .
+tool(<write_...>(Args, Result), "Description") :- ... .
+tool(user_feedback(Prompt, Response), "Ask the user one focused question and return their response") :-
+    exec(ask_user(prompt: Prompt), Result),
+    get_dict(user_response, Result, Response).
+
+% --- deterministic helpers (mechanical only) -------------------------------
+<compute_or_check>(...).          % arithmetic/counts; keep small
+
+% --- entry point ------------------------------------------------------------
+agent_main(Request) :-
+    Request \= "",
+    system("Role and hard rules. Treat supplied policy text as governing."),
+    output("Parsing the request..."),
+    extract_case(Request, Case),                    % task/N -> object/1
+    output("Confirming..."),
+    confirm_case(Case, ConfirmedCase),              % user_feedback loop
+    output("Acting..."),
+    with_tools([<write tools>], (
+        task("Produce the required effects for this case: {ConfirmedCase}.", string(Summary))
+    )),
+    <optional verification with LLM fallback>,
+    answer(Final).
+
+agent_main(_) :-
+    answer("Supply a request describing the case.").
+```
+
+Notes:
+
+- `task/N` = agentic leaf (memory + DML tools). `prompt/N` = fresh-context
+  review. `with_tools/2` scopes capability per phase.
+- Build `task/N` descriptions with `format/3` (not `{Var}` interpolation) when
+  you embed dynamic values — avoids singleton-variable noise.
+- Mutable facts must be declared `:- dynamic` before `assertz`/`retract`.
+
+## Verification (optional, LLM-first)
+
+Only add checks when the procedure has observable post-conditions. Prefer model
+review; use deterministic checks only for mechanical facts, and always give a
+deterministic check an LLM fallback.
+
+```prolog
+% deterministic gate (mechanical only)
+holds(drafts_count) :- findall(_, draft(_,_,_,_), Ds), length(Ds, 2).
+
+verify_state(Failed, Report) :-
+    findall(Name, (postcondition(Name), \+ holds(Name)), Failed),
+    ( Failed = [] -> Report = "PASS" ; format(string(Report), "FAIL: ~w", [Failed]) ).
+
+% LLM fallback: never hard-fail on a deterministic check
+fallback_review(Failed, Verdict, Reason) :-
+    format(string(Prompt),
+        "These structural checks failed: ~w. Review the outcome and the requirement. Store 'acceptable' or 'needs-attention' in Verdict and a one-line reason in Reason.",
+        [Failed]),
+    prompt(Prompt, string(Verdict), string(Reason)).
+```
+
+In `agent_main`:
+
+```prolog
+verify_state(Failed, Report),
+(   Failed = [] -> V = "n/a", R = "deterministic checks passed"
+;   fallback_review(Failed, V, R)
+),
+answer(... report Report + V/R ...).
+```
+
+Choosing:
+
+- **Model review** (default): tone, completeness, correctness of free text,
+  "does this read right", and anything the rubric phrases as a judgment.
+- **Deterministic** (only when mechanical): counts, exact IDs, arithmetic. Keep
+  it tiny, and route failures to the model or the user instead of failing.
+
+## Dummy tools
+
+Only after the user approves. Prefer **file-backed** dummy tools (read/write
+fixture files through `pi_bash`) because they mirror real services and make the
+skill runnable against real data. Use **fact-backed** state only for pure
+in-memory cases. Keep the tool name/contract stable so a real `exec/2`/MCP
+implementation can replace the dummy body later.
+
+## The AGENTS.md policy router
+
+After the skills are written and tested, wire them into the repo-root
+`AGENTS.md` so pi calls them automatically. `AGENTS.md` is always in pi's
+context; `dc_run` (enabled once with `/dc-tool enable`) executes a named skill.
+
+```markdown
+## DeepClause policy routing
+
+When a request matches a procedure below, do **not** answer from memory or from
+the source handbook text. Call the `dc_run` tool with the mapped skill, then
+report its answer (including its verification result, if any).
+
+If `dc_run` is unavailable, tell the user to run `/dc-tool enable` (persists for
+the workspace) or to run the equivalent `/dc-run` command themselves.
+
+| Handbook | Procedure / trigger | Skill (`dc_run.skill`) | Args (`dc_run.args`) |
+| --- | --- | --- | --- |
+| <handbook_slug> | <short trigger phrase a request would match> | `skills/<handbook_slug>/<slug>.dml` | `["<the user's request>"]` |
+
+Conventions:
+
+- `args` carries the natural-language request; pass the user's message through.
+- If the skill has verification, do not claim success unless it passes.
+- Detailed maps live in each handbook's
+  `.pi/deepclause/skills/<handbook_slug>/INDEX.md`.
+```
+
+Rules:
+
+1. **`Handbook`** — the `<handbook_slug>` directory under `.pi/deepclause/skills/`.
+2. **`Procedure / trigger`** — a short phrase in the *user's* wording.
+3. **`Skill`** — path relative to `.pi/deepclause/`; must contain `/`.
+4. **`Args`** — the request text (the skill parses it). One row per procedure.
+5. Update the table in the same change; tell the user to `/reload` if pi is
+   already running.
+
+## Testing checklist
+
+For each generated skill:
+
+1. `/dc-run skills/<handbook_slug>/<slug>.dml "<request>" --context=isolated --debug`
+   — runs clean; verification (if any) passes or the fallback explains.
+2. Inspect phase is read-only, action phase is write-only, and a forbidden
+   action has no tool at all.
+3. Deterministic code is limited to arithmetic/counts; every deterministic
+   check has an LLM fallback branch.
+4. The tool audit was done and its result is recorded in the header/INDEX.
+5. Re-check the invalid-pattern table in `.pi/deepclause/AGENTS.md` (singleton
+   variables, `~` vs `{}` interpolation, `Result.field` vs `get_dict/3`, `->`
+   committing over generators, `answer/1` last, `:- dynamic` before
+   `assertz/retract`).
+
+`--context=isolated` keeps session text out of the run. The runtime still needs
+a model selected.
+
+## Reference shape
+
+A typical procedure skill has: `agent_main(Request)` that parses the request
+with `task/N`, confirms with the user via a `user_feedback` tool loop, acts
+through write tools, and reviews with `prompt/N` — with small deterministic
+helpers for arithmetic and a fallback branch instead of hard failures.
+
+For DML mechanics, see the bundled example skills in a fresh workspace
+(`example.dml`, `deep_research.dml`) and `.pi/deepclause/AGENTS.md`.