@zihanw/pi-forge 0.2.0 → 0.3.1

package/README.md

@@ -2,7 +2,9 @@
 
 [English](README.md) | [简体中文](README.zh-CN.md)
 
-**pi-forge** lets you customize how Pi thinks and behaves. It gives you prompt stacks — JSON files that can replace, append to, or prepend Pi's default system prompt while controlling the AI's personality, visible tools, conversation history layout, and cross-turn state.
+![pi-forge header](https://raw.githubusercontent.com/MacroSony/pi-forge/main/assets/pi-forge-header-concept-1.png)
+
+**pi-forge** lets you customize how Pi thinks and behaves. It gives you prompt stacks — JSON files that can replace, append to, or prepend Pi's default system prompt while controlling the AI's personality, visible tools, conversation history layout, template variables, and prompt transforms.
 
 Think of it as a character sheet for your AI agent.
 
@@ -11,7 +13,9 @@ Think of it as a character sheet for your AI agent.
 - **Give Pi a personality** — turn it into a creative writer, a roleplay partner, a strict code reviewer, or anything in between.
 - **Switch contexts instantly** — one command to swap between "coding mode", "writing mode", and "translation mode".
 - **Control what the AI sees** — choose which tools, skills, and project context appear in each prompt.
-- **Remember things across turns** — let the agent track progress, store notes, and recall user preferences throughout a session.
+- **Limit tools and skills per stack** — enforce active tool policy and filter skill visibility for focused modes.
+- **Use template variables** — define static values such as `{{char}}` / `{{user}}`, and use ST-style turn/session variable macros inside prompt text.
+- **Transform outgoing and finalized text** — run deterministic regex replacements on selected history, compiled prompt text, or finalized assistant messages.
 - **Import SillyTavern presets** — bring your existing ST character presets into Pi with one command.
 - **Debug your prompts** — intercept and inspect exactly what gets sent to the model.
 
@@ -25,43 +29,17 @@ pi install npm:@zihanw/pi-forge
 
 ### Your first prompt stack
 
-Create `.pi/prompt-stacks/default.json`:
+Create `.pi/forge/prompt-stacks/default.json` from [examples/default-prompt-stack.json](examples/default-prompt-stack.json).
 
-```json
-{
-  "schemaVersion": 1,
-  "type": "pi-forge.prompt-stack",
-  "id": "default",
-  "autoActivate": true,
-  "mode": "replace",
-  "items": [
-    {
-      "kind": "block",
-      "id": "role",
-      "name": "Main Role",
-      "enabled": true,
-      "role": "system",
-      "content": "You are a friendly and concise coding assistant. Prefer short answers with code examples."
-    },
-    {
-      "kind": "slot",
-      "id": "tools",
-      "name": "Available Tools",
-      "enabled": true,
-      "role": "system",
-      "slot": "tools"
-    },
-    {
-      "kind": "slot",
-      "id": "chat-history",
-      "name": "Chat History",
-      "enabled": true,
-      "slot": "chat-history"
-    }
-  ]
-}
+The default example mirrors Pi's own prompt builder from `@earendil-works/pi-coding-agent/dist/core/system-prompt.js`, but splits it into movable pi-forge slots: role, tools, guidelines, Pi docs guidance, appended system prompt text, project context, skills, date/cwd, and chat history.
+
+```bash
+mkdir -p .pi/forge/prompt-stacks
+$EDITOR .pi/forge/prompt-stacks/default.json
 ```
 
+Paste the example JSON into that file. If you are working inside this repository, you can copy it directly with `cp examples/default-prompt-stack.json .pi/forge/prompt-stacks/default.json`.
+
 That's it. Restart Pi or run `/preset reload`. If no stack is already selected, `default.json` auto-activates. If you previously chose another stack or `/preset use none`, run `/preset use default`.
 
 ### Visual editor
@@ -72,13 +50,15 @@ Prefer clicking over typing JSON? pi-forge has a built-in web editor:
 /preset ui
 ```
 
-Drag, drop, edit, validate, preview, import, export, fork, and delete stacks — all in your browser.
+Drag, drop, create, edit, validate, inspect full previews and captured payloads, manage variables/context/regex rules in tabs, switch dark mode, recover through raw stack JSON, import, export, fork, and delete stacks — all in your browser. New stacks start from the default Pi prompt mirror layout. Stack metadata is collapsible so the active editor stays in view. The policy tab shows registered tools and loaded skills with selected-pattern chips and filtering, so allow/deny rules can be built from exact names while still supporting wildcards.
 
 Import accepts native pi-forge stack JSON and SillyTavern preset JSON. SillyTavern presets are converted to prompt stacks automatically; if a preset contains multiple `character_id` configs, the editor asks which one to use.
 
-The editor runs on `127.0.0.1:41738` by default with a session token. Writes require a trusted project and stay inside `.pi/prompt-stacks`; successful save, import, fork, and delete actions reload into the current Pi session. Use `/preset ui restart` or `/preset ui stop` when needed.
+The editor runs on an available `127.0.0.1` port with a session token, so multiple Pi instances can run editors at the same time. If Pi reinitializes the extension after session navigation or a new session, `/preset ui` reuses the existing editor URL for the same project instead of orphaning the old server. Writes require a trusted project and stay inside prompt-stack storage. New stacks are written to `.pi/forge/prompt-stacks`; existing legacy stacks under `.pi/prompt-stacks` remain readable and editable. Successful save, import, fork, and delete actions reload into the current Pi session. Use `/preset ui restart` or `/preset ui stop` when needed.
+
+To copy old stacks into the new location, run `/preset migrate-stacks`. Add `--dry-run` to preview, `--overwrite` to replace existing target files, and `--delete-legacy` to remove old files after successful copy.
 
-To use a different fixed port, create `.pi/forge/config.json`:
+To prefer a specific port, create `.pi/forge/config.json`. If that port is busy, pi-forge falls back to another available port and shows the actual URL:
 
 ```json
 {
@@ -102,11 +82,11 @@ Useful pattern:
 
 This keeps the latest request clear and avoids duplicating it.
 
-For a copyable starter stack, see [examples/default-prompt-stack.json](examples/default-prompt-stack.json).
+For a baseline stack to fork before turning Pi into a character, start from [examples/default-prompt-stack.json](examples/default-prompt-stack.json).
 
 ### 🧑‍💻 Focused code review
 
-Create a `reviewer.json` stack with a strict review block: "prioritize correctness, regressions, security, and missing tests." Keep the `tools`, `project-context`, `variables`, and `chat-history` slots enabled so Pi can still inspect the repo and remember review state.
+Create a `reviewer.json` stack with a strict review block: "prioritize correctness, regressions, security, and missing tests." Keep the `tools`, `project-context`, `variables`, and `chat-history` slots enabled so Pi can still inspect the repo and see any template variables you expose.
 
 Use `mode: "append"` if you want to keep Pi's normal coding behavior and only add the sharper review lens.
 
@@ -119,7 +99,7 @@ Create a small `translator.json` stack with one system block for tone and target
 Create separate stacks for different tasks:
 
 ```
-.pi/prompt-stacks/
+.pi/forge/prompt-stacks/
   coder.json       # strict coding assistant
   writer.json      # creative writing partner
   translator.json  # bilingual translator
@@ -127,29 +107,35 @@ Create separate stacks for different tasks:
 
 Switch with `/preset use coder`, `/preset use writer`, etc.
 
-### 🧠 Cross-turn memory
+### 🧪 Presets that show off pi-forge
 
-Define state the agent can read and write:
+- **Pi mirror** — start from [examples/default-prompt-stack.json](examples/default-prompt-stack.json). It preserves normal Pi behavior while making every runtime section movable and inspectable.
+- **Focused reviewer** — see [examples/reviewer-prompt-stack.json](examples/reviewer-prompt-stack.json). It denies file-writing tools, wraps prior chat history as background, removes the latest user message from history, then reinserts `{{lastUserMessage}}` as the explicit review target.
+- **Read-only scout** — use `tools.allow` for `read`, `grep`, `find`, and `ls`; omit editing tools; cap `chat-history` with `maxChars`. Good for exploration turns where the model should report findings without changing files.
+- **Surgical patcher** — keep the Pi mirror, require `read`, `edit`, and `bash`, strip assistant thinking from inserted history, and move `project-context` near the final user turn. Good for focused implementation passes.
+- **SillyTavern DM writer** — see [examples/sillytavern-dm-writer-prompt-stack.json](examples/sillytavern-dm-writer-prompt-stack.json). It defines a Dungeon Master character with `{{char}}` / `{{user}}`, wraps prior adventure history, reinserts `{{lastUserMessage}}` as the current player action, and uses regex cleanup for OOC notes, secret-roll markers, dice notation, and `Player:` prefixes.
+- **Payload lab** — include `active-model`, `date-cwd`, and variables slots, then add `compiled` regex rules for deterministic redaction or formatting. Pair it with `/payload next` or the web editor's capture view to audit exactly what changed.
+- **Docs-only Pi expert** — allow only read/search tools, enable the `pi-docs` slot, and keep project context. Useful when you want answers grounded in the installed Pi docs instead of general memory.
+
+### 🔧 Template variables
 
 ```json
-"state": {
-  "definitions": {
-    "agent.progress": {
-      "type": "string",
-      "scope": "session",
-      "description": "What we're working on",
-      "agentWritable": true
-    }
-  }
+"variables": {
+  "char": "Konata",
+  "user": "User"
 }
 ```
 
-The agent updates it with `forge_state_set`. You can also set state manually:
+Use static variables for stable prompt constants, and ST-style macros for local prompt-time mutation:
 
 ```
-/state set user.preference "use TypeScript, not JavaScript"
+{{setvar::mood::focused}}
+{{getvar::mood}}
+{{setsessionvar::topic::compiler cleanup}}
 ```
 
+For durable project memory, use normal files in the repo rather than pi-forge prompt variables.
+
 ### 📦 SillyTavern migration
 
 Bring your ST presets into Pi:
@@ -160,6 +146,8 @@ Bring your ST presets into Pi:
 
 pi-forge converts the preset to a prompt stack and generates a migration report showing what was handled and what needs manual tweaking.
 
+Deterministic SillyTavern `promptOnly` regex scripts are converted to pi-forge `regex.rules` as history-stage rules when they can be represented safely, including full-match token conversion, trim strings, depth fields, and clear user/assistant placements. Display-only, mixed prompt/display, DOM/browser, CSS/HTML decoration, JavaScript, unsupported placements, and invalid regex scripts stay report-only for manual review.
+
 ### 🔍 Prompt debugging
 
 See exactly what gets sent to the model:
@@ -168,6 +156,8 @@ See exactly what gets sent to the model:
 /payload next save=.pi/forge/payloads/last.json
 ```
 
+Or open `/preset ui`, click **Arm payload**, send the next Pi prompt, and inspect the redacted provider payload in the browser.
+
 Or preview your compiled prompt without sending anything:
 
 ```
@@ -188,6 +178,9 @@ Items are arranged in order. When the stack is active, pi-forge:
 1. Builds a system prompt from your `system`-role blocks and slots, then applies it with the stack's `mode`.
 2. Inserts `user`/`assistant` blocks and slots around the conversation history.
 3. Expands `{{macros}}` like `{{lastUserMessage}}`, `{{date}}`, and custom variables.
+4. Applies stack tool policy to Pi's active tool set and filters pi-forge-rendered tool/skill slots.
+5. Applies enabled outgoing regex rules for the `history` and `compiled` stages.
+6. Optionally applies destructive `finalize` regex rules when an assistant message finishes.
 
 ### Slots at a glance
 
@@ -198,8 +191,8 @@ Items are arranged in order. When the stack is active, pi-forge:
 | `tool-guidelines` | Tool usage instructions |
 | `skills` | Loaded Pi skills |
 | `project-context` | Project instructions and context files |
-| `variables` | Agent and user state (progress, preferences, notes) |
-| `date` / `cwd` / `date-cwd` | Current date and working directory |
+| `variables` | Static/session/turn template variables |
+| `date` / `cwd` / `date-cwd` | Current date, optional current time, and working directory |
 | `active-model` | Which model is being used |
 | `append-system-prompt` | User's appended system prompt text |
 | `pi-docs` | Pi documentation guidance |
@@ -224,26 +217,16 @@ Items are arranged in order. When the stack is active, pi-forge:
 | `/preset status` | Show the active stack and diagnostics summary |
 | `/preset diagnostics` | Show runtime diagnostics |
 | `/preset reload` | Reload stacks from disk |
+| `/preset migrate-stacks [--dry-run] [--overwrite] [--delete-legacy]` | Copy legacy `.pi/prompt-stacks` files into `.pi/forge/prompt-stacks` |
 | `/preset ui [stop\|restart]` | Open, stop, or restart the web editor |
 
-### State management
-
-| Command | What it does |
-|---------|-------------|
-| `/state list` | Show all session state |
-| `/state status` | Show state definitions and current values |
-| `/state set <name> <value>` | Set a state variable |
-| `/state get <name>` | Read a state variable |
-| `/state clear [name]` | Clear state (all or by name) |
-| `/preset vars ...` | Legacy variable commands kept for older stacks |
-
 ### Import & debug
 
 | Command | What it does |
 |---------|-------------|
 | `/preset import-silly <path>` | Import a SillyTavern preset |
 | `/intercept` | Show the next provider payload |
-| `/payload next [save=<path>]` | Show and optionally save the next payload |
+| `/payload next [save=<path>]` | Show, save, and expose the next payload to the web editor |
 
 ## Common macros
 
@@ -274,6 +257,72 @@ Use these in block content to insert dynamic values:
 {{clearsessionvar::name}}     clear a session variable
 ```
 
+### Filter and conditional macros
+
+Nested macros are supported, and `::` separators are parsed only at the current macro depth.
+
+| Macro | Expands to |
+|-------|-----------|
+| `{{trim::value}}` | `value` with leading/trailing whitespace removed |
+| `{{upper::value}}` | Uppercase `value` |
+| `{{lower::value}}` | Lowercase `value` |
+| `{{json::value}}` | JSON string literal for `value` |
+| `{{xml::value}}` | XML-escaped `value` |
+| `{{ifvar::name::then::else}}` | `then` when a variable exists, otherwise `else` |
+| `{{ifeq::name::expected::then::else}}` | `then` when a variable equals `expected`, otherwise `else` |
+| `{{iftools::tool::then::else}}` | `then` when the selected tool list includes `tool`, otherwise `else` |
+| `{{ifslot::slot::then::else}}` | `then` when the enabled stack items include `slot`, otherwise `else` |
+
+Conditional branches are lazy: only the selected branch is expanded, so skipped branches cannot set or clear variables. The final `else` argument is optional and defaults to empty text.
+
+### Trusted custom macros and slots
+
+Custom macros and slots are registered by trusted extension code, not embedded in prompt-stack JSON. A stack can reference the registered name and pass declarative options; the executable `render` body lives in a Pi extension or another explicitly installed package.
+
+```ts
+import { registerMacro, registerSlot } from "@zihanw/pi-forge";
+
+registerMacro({
+  name: "ticketId",
+  description: "Current ticket id from session variables.",
+  render: (ctx) => ctx.variables.toMacroText(ctx.variables.get("ticket.id")),
+});
+
+registerSlot({
+  name: "ticket-context",
+  description: "Render ticket context for the current task.",
+  options: {
+    heading: { type: "string", default: "Ticket context" },
+  },
+  render: (ctx) => [
+    String(ctx.options.heading ?? "Ticket context") + ":",
+    "- Ticket: " + ctx.variables.toMacroText(ctx.variables.get("ticket.id")),
+    "- Project: " + ctx.helpers.normalizePath(ctx.runtime.options.cwd),
+  ].join("\n"),
+});
+```
+
+The stack remains declarative:
+
+```json
+{
+  "kind": "slot",
+  "id": "ticket-context",
+  "enabled": true,
+  "role": "system",
+  "slot": "ticket-context",
+  "options": {
+    "heading": "Current ticket"
+  }
+}
+```
+
+Missing custom slots are validation warnings until the registering extension is installed and loaded. Built-in macros and slots use the same registry internally, so `getRegisteredMacros()` and `getRegisteredSlots()` can be used as implementation references.
+
+For a complete copyable extension and stack, see [examples/custom-system-status-extension](examples/custom-system-status-extension). It registers a `{{cpuLoad}}` macro and a `machine-status` slot from a trusted Pi extension file.
+
+When pi-forge is installed as a package, custom extensions can import `@zihanw/pi-forge`. When developing from a local checkout loaded through Pi settings, set an explicit local path as shown in the example README, such as `PI_FORGE_MODULE=/path/to/pi-forge/src/index.ts`. pi-forge keeps the macro/slot registry process-global so compatible local module copies share registrations.
+
 ## Stack reference
 
 ### Full item types
@@ -313,77 +362,150 @@ Valid roles: `system`, `user`, `assistant`, `custom`.
 
 ```json
 "options": {
-  "includeLastUserMessage": false
+  "includeLastUserMessage": false,
+  "stripAssistantThinking": true,
+  "includeSummaries": true,
+  "toolMode": "keep",
+  "roles": ["user", "assistant"],
+  "maxMessages": 40,
+  "maxChars": 20000
 }
 ```
 
 Set to `false` when you use `{{lastUserMessage}}` after the history — prevents the user's message from appearing twice.
 
-### Variables slot options
+Set `stripAssistantThinking` to `true` to remove prior assistant thinking blocks from inserted chat history. Visible assistant text, tool calls, and tool result messages are preserved. This only affects history inserted by that slot and does not alter the current agent loop or stored transcript.
+
+Use `includeSummaries: false` to omit Pi branch/compaction summary messages, `roles` to keep only specific message roles, `toolMode: "drop"` to remove prior tool-call/tool-result history, and `maxMessages` / `maxChars` to keep only recent history. When filters or limits can break tool-call pairs, pi-forge removes dangling tool calls/results instead of sending inconsistent tool history.
+
+### Date slot options
+
+Set `"includeTime": true` on a `date` or `date-cwd` slot to include the current time in `HH:MM:SS` after the current date.
+
+### Structured slot format options
+
+Structured runtime slots default to XML-style wrappers. Add `"format": "plain"` to `tools`, `tool-guidelines`, `skills`, `project-context`, or `variables` slots for compact newline-separated output.
 
 ```json
 {
   "kind": "slot",
-  "id": "state",
+  "id": "tools",
   "enabled": true,
-  "role": "user",
-  "slot": "variables",
+  "role": "system",
+  "slot": "tools",
+  "options": {
+    "format": "plain"
+  }
+}
+```
+
+The default Pi mirror uses a few extra slot options:
+
+```json
+{
+  "slot": "tools",
   "options": {
-    "includeScopes": ["session"],
-    "includeNamespaces": ["user.*", "agent.*"],
-    "includeMetadata": true,
-    "format": "xml",
-    "maxValueChars": 1200
+    "format": "plain",
+    "onlyWithSnippets": true
   }
 }
 ```
 
-### State definitions
+`tools.onlyWithSnippets` matches Pi's default "Available tools" section by hiding tools that do not provide prompt snippets. `tool-guidelines.heading`, `tool-guidelines.includePiDefaultGuidelines`, and `tool-guidelines.piStyle` make the guidelines slot match Pi's default heading and bullets. `skills.requireReadTool` hides skills unless the read tool is active, matching Pi's default behavior.
+
+### Tool and skill policy
+
+Prompt stacks can constrain tools and skills with stack-level `allow` or `deny` lists. Patterns are exact by default and support `*` wildcards.
+
+```json
+{
+  "tools": {
+    "allow": ["read", "bash"]
+  },
+  "skills": {
+    "deny": ["browser-danger"]
+  }
+}
+```
+
+Use `allow` when only matching tools or skills should remain active. Use `deny` when everything except matching tools or skills should remain active. A single resource policy cannot contain both non-empty lists; mixed `allow` and `deny` entries are validation errors.
+
+Tool policy is enforced through Pi's active tool list while the stack is active. pi-forge remembers the previous active tools and restores them when prompt stacks are disabled or switched to an unrestricted stack.
+
+Skill policy filters skills rendered by pi-forge's `skills` slot. If a stack uses `mode: "append"` or `"prepend"`, Pi's base prompt may already contain unfiltered skills; use `mode: "replace"` when skill visibility must be controlled.
+
+### Regex transforms
+
+Prompt stacks can run deterministic regex replacements on model-bound prompt text and, optionally, finalized assistant messages. Outgoing rules support `history` and `compiled` stages. Destructive final-message cleanup uses `effect: "finalize"` at `stage: "compiled"` with the `messages` target. True display-only streaming transforms and provider-payload rewrites are not active yet.
 
 ```json
-"state": {
+"regex": {
   "schemaVersion": 1,
-  "definitions": {
-    "agent.progress": {
-      "type": "string",
-      "scope": "session",
-      "description": "Current task progress",
-      "agentWritable": true
-    },
-    "user.preference": {
-      "type": "string",
-      "scope": "session",
-      "description": "User's preference for this session",
-      "userWritable": true
+  "rules": [
+    {
+      "id": "trim-ooc",
+      "enabled": true,
+      "stage": "history",
+      "effect": "outgoing",
+      "pattern": "\\(OOC:[^)]+\\)",
+      "flags": "gi",
+      "replace": "",
+      "roles": ["assistant"],
+      "maxMessages": 20
     }
-  }
+  ]
 }
 ```
 
-Supported types: `string`, `number`, `boolean`, `null`, `object`, `array`, `string[]`, `number[]`, `boolean[]`, `unknown`, and unions like `string | null`.
+Use `stage: "history"` to transform messages inserted by the `chat-history` slot. Use `stage: "compiled"` with optional `targets: ["system"]`, `["messages"]`, or both to transform the final compiled prompt. Message rules can filter by `roles`, `maxMessages`, `maxChars`, `minDepth`, and `maxDepth`, where depth `0` is the latest message. Replacements use JavaScript syntax (`$&` for the full match, `$1` for captures; `$0` is also accepted as a full-match alias, and `$$` escapes a literal `$`). `trimStrings` removes literal strings from expanded replacement matches/captures, matching SillyTavern's Trim Out behavior. Supported regex flags are `g`, `i`, `m`, `s`, and `u`.
 
-## Agent tools
+To clean a completed assistant message after streaming, use `effect: "finalize"`:
 
-pi-forge registers two tools that the AI agent can call:
+```json
+{
+  "id": "finalize-ooc",
+  "enabled": true,
+  "stage": "compiled",
+  "effect": "finalize",
+  "targets": ["messages"],
+  "roles": ["assistant"],
+  "pattern": "\\s*\\(OOC:[^)]+\\)",
+  "flags": "gi",
+  "replace": ""
+}
+```
 
-### `forge_state_set`
+Warning: `finalize` runs at `message_end`, after raw output may already have streamed in the TUI. It returns a cleaned replacement message to Pi, so the original model output is not preserved in the stored transcript.
 
-Batch-update persistent state. Only `agent.*` names are writable. Use for cross-turn tracking:
+`effect: "outgoing"` changes model input. `effect: "finalize"` changes finalized assistant transcript content. `effect: "display"` and `"both"` validate with warnings but are ignored at runtime until true display transforms are implemented.
 
-- Task progress (`agent.progress`)
-- Open questions (`agent.openQuestions`)
-- Story state (`agent.storyState`)
-- User-requested notes (`agent.notes`)
+SillyTavern imports convert deterministic prompt-only `{{match}}` / `$0` full-match replacements to JavaScript `$&` (both `$0` and `$&` work in pi-forge), preserve original regex metadata in `source.sillytavern`, and run as history-stage rules so depth stays chat-relative. Display-only/browser/unsupported-placement scripts stay report-only. The web editor has a structured Regex dialog for these rule fields and preserves advanced unknown fields for raw JSON editing.
 
-### `forge_set_var`
+### Variables slot options
 
-Legacy alias for setting a single string value. Prefer `forge_state_set`.
+```json
+{
+  "kind": "slot",
+  "id": "variables",
+  "enabled": true,
+  "role": "user",
+  "slot": "variables",
+  "options": {
+    "includeStatic": true,
+    "includeSession": true,
+    "includeTurn": false,
+    "format": "xml"
+  }
+}
+```
 
 ## Package setup for development
 
 ```bash
 git clone <repo>
 cd pi-forge
+npm install
+npm run build
 # .pi/settings.json already points at the package root
 pi    # start Pi, trust the project, /reload if needed
 ```
@@ -400,6 +522,12 @@ Typecheck:
 npm run typecheck
 ```
 
+Build package output:
+
+```bash
+npm run build
+```
+
 ## License
 
 MIT