@hera-al/server 1.6.1

package/bundled/open-prose/skills/prose/state/in-context.md

@@ -0,0 +1,384 @@
+---
+role: in-context-state-management
+summary: |
+  In-context state management using the narration protocol with text markers.
+  This approach tracks execution state within the conversation history itself.
+  The OpenProse VM "thinks aloud" to persist state—what you say becomes what you remember.
+see-also:
+  - ../prose.md: VM execution semantics
+  - filesystem.md: File-system state management (alternative approach)
+  - sqlite.md: SQLite state management (experimental)
+  - postgres.md: PostgreSQL state management (experimental)
+  - ../primitives/session.md: Session context and compaction guidelines
+---
+
+# In-Context State Management
+
+This document describes how the OpenProse VM tracks execution state using **structured narration** in the conversation history. This is one of two state management approaches (the other being file-based state in `filesystem.md`).
+
+## Overview
+
+In-context state uses text-prefixed markers to persist state within the conversation. The VM "thinks aloud" about execution—what you say becomes what you remember.
+
+**Key principle:** Your conversation history IS the VM's working memory.
+
+---
+
+## When to Use In-Context State
+
+In-context state is appropriate for:
+
+| Factor            | In-Context      | Use File-Based Instead |
+| ----------------- | --------------- | ---------------------- |
+| Statement count   | < 30 statements | >= 30 statements       |
+| Parallel branches | < 5 concurrent  | >= 5 concurrent        |
+| Imported programs | 0-2 imports     | >= 3 imports           |
+| Nested depth      | <= 2 levels     | > 2 levels             |
+| Expected duration | < 5 minutes     | >= 5 minutes           |
+
+Announce your state mode at program start:
+
+```
+OpenProse Program Start
+   State mode: in-context (program is small, fits in context)
+```
+
+---
+
+## The Narration Protocol
+
+Use text-prefixed markers for each state change:
+
+| Marker     | Category       | Usage                                   |
+| ---------- | -------------- | --------------------------------------- |
+| [Program]  | Program        | Start, end, definition collection       |
+| [Position] | Position       | Current statement being executed        |
+| [Binding]  | Binding        | Variable assignment or update           |
+| [Input]    | Input          | Receiving inputs from caller            |
+| [Output]   | Output         | Producing outputs for caller            |
+| [Import]   | Import         | Fetching and invoking imported programs |
+| [Success]  | Success        | Session or block completion             |
+| [Warning]  | Error          | Failures and exceptions                 |
+| [Parallel] | Parallel       | Entering, branch status, joining        |
+| [Loop]     | Loop           | Iteration, condition evaluation         |
+| [Pipeline] | Pipeline       | Stage progress                          |
+| [Try]      | Error handling | Try/catch/finally                       |
+| [Flow]     | Flow           | Condition evaluation results            |
+| [Frame+]   | Call Stack     | Push new frame (block invocation)       |
+| [Frame-]   | Call Stack     | Pop frame (block completion)            |
+
+---
+
+## Narration Patterns by Construct
+
+### Session Statements
+
+```
+[Position] Executing: session "Research the topic"
+   [Task tool call]
+[Success] Session complete: "Research found that..."
+[Binding] let research = <result>
+```
+
+### Parallel Blocks
+
+```
+[Parallel] Entering parallel block (3 branches, strategy: all)
+   - security: pending
+   - perf: pending
+   - style: pending
+   [Multiple Task calls]
+[Parallel] Parallel complete:
+   - security = "No vulnerabilities found..."
+   - perf = "Performance is acceptable..."
+   - style = "Code follows conventions..."
+[Binding] security, perf, style bound
+```
+
+### Loop Blocks
+
+```
+[Loop] Starting loop until **task complete** (max: 5)
+
+[Loop] Iteration 1 of max 5
+   [Position] session "Work on task"
+   [Success] Session complete
+   [Loop] Evaluating: **task complete**
+   [Flow] Not satisfied, continuing
+
+[Loop] Iteration 2 of max 5
+   [Position] session "Work on task"
+   [Success] Session complete
+   [Loop] Evaluating: **task complete**
+   [Flow] Satisfied!
+
+[Loop] Loop exited: condition satisfied at iteration 2
+```
+
+### Error Handling
+
+```
+[Try] Entering try block
+[Position] session "Risky operation"
+[Warning] Session failed: connection timeout
+[Binding] err = {message: "connection timeout"}
+[Try] Executing catch block
+[Position] session "Handle error" with context: err
+[Success] Recovery complete
+[Try] Executing finally block
+[Position] session "Cleanup"
+[Success] Cleanup complete
+```
+
+### Variable Bindings
+
+```
+[Binding] let research = "AI safety research covers..." (mutable)
+[Binding] const config = {model: "opus"} (immutable)
+[Binding] research = "Updated research..." (reassignment, was: "AI safety...")
+```
+
+### Input/Output Bindings
+
+```
+[Input] Inputs received:
+   topic = "quantum computing" (from caller)
+   depth = "deep" (from caller)
+
+[Output] output findings = "Research shows..." (will return to caller)
+[Output] output sources = ["arxiv:2401.1234", ...] (will return to caller)
+```
+
+### Block Invocation and Call Stack
+
+Track block invocations with frame markers:
+
+```
+[Position] do process(data, 5)
+[Frame+] Entering block: process (execution_id: 1, depth: 1)
+   Arguments: chunk=data, depth=5
+
+   [Position] session "Split into parts"
+      [Task tool call]
+   [Success] Session complete
+   [Binding] let parts = <result> (execution_id: 1)
+
+   [Position] do process(parts[0], 4)
+   [Frame+] Entering block: process (execution_id: 2, depth: 2)
+      Arguments: chunk=parts[0], depth=4
+      Parent: execution_id 1
+
+      [Position] session "Split into parts"
+         [Task tool call]
+      [Success] Session complete
+      [Binding] let parts = <result> (execution_id: 2)  # Shadows parent's 'parts'
+
+      ... (continues recursively)
+
+   [Frame-] Exiting block: process (execution_id: 2)
+
+   [Position] session "Combine results"
+      [Task tool call]
+   [Success] Session complete
+
+[Frame-] Exiting block: process (execution_id: 1)
+```
+
+**Key points:**
+
+- Each `[Frame+]` must have a matching `[Frame-]`
+- `execution_id` uniquely identifies each invocation
+- `depth` shows call stack depth (1 = first level)
+- Bindings include `(execution_id: N)` to indicate scope
+- Nested frames show `Parent: execution_id N` for the scope chain
+
+### Scoped Binding Narration
+
+When inside a block invocation, always include the execution_id:
+
+```
+[Binding] let result = "computed value" (execution_id: 43)
+```
+
+For variable resolution across scopes:
+
+```
+[Binding] Resolving 'config': found in execution_id 41 (parent scope)
+```
+
+### Program Imports
+
+```
+[Import] Importing: @alice/research
+   Fetching from: https://p.prose.md/@alice/research
+   Inputs expected: [topic, depth]
+   Outputs provided: [findings, sources]
+   Registered as: research
+
+[Import] Invoking: research(topic: "quantum computing")
+   [Input] Passing inputs:
+      topic = "quantum computing"
+
+   [... imported program execution ...]
+
+   [Output] Received outputs:
+      findings = "Quantum computing uses..."
+      sources = ["arxiv:2401.1234"]
+
+[Import] Import complete: research
+[Binding] result = { findings: "...", sources: [...] }
+```
+
+---
+
+## Context Serialization
+
+**In-context state passes values, not references.** This is the key difference from file-based and PostgreSQL state. The VM holds binding values directly in conversation history.
+
+When passing context to sessions, format appropriately:
+
+| Context Size    | Strategy                |
+| --------------- | ----------------------- |
+| < 2000 chars    | Pass verbatim           |
+| 2000-8000 chars | Summarize to key points |
+| > 8000 chars    | Extract essentials only |
+
+**Format:**
+
+```
+Context provided:
+---
+research: "Key findings about AI safety..."
+analysis: "Risk assessment shows..."
+---
+```
+
+**Limitation:** In-context state cannot support RLM-style "environment as variable" patterns where agents query arbitrarily large bindings. For programs with large intermediate values, use file-based or PostgreSQL state instead.
+
+---
+
+## Complete Execution Trace Example
+
+```prose
+agent researcher:
+  model: sonnet
+
+let research = session: researcher
+  prompt: "Research AI safety"
+
+parallel:
+  a = session "Analyze risk A"
+  b = session "Analyze risk B"
+
+loop until **analysis complete** (max: 3):
+  session "Synthesize"
+    context: { a, b, research }
+```
+
+**Narration:**
+
+```
+[Program] Program Start
+   Collecting definitions...
+   - Agent: researcher (model: sonnet)
+
+[Position] Statement 1: let research = session: researcher
+   Spawning with prompt: "Research AI safety"
+   Model: sonnet
+   [Task tool call]
+[Success] Session complete: "AI safety research covers alignment..."
+[Binding] let research = <result>
+
+[Position] Statement 2: parallel block
+[Parallel] Entering parallel (2 branches, strategy: all)
+   [Task: "Analyze risk A"] [Task: "Analyze risk B"]
+[Parallel] Parallel complete:
+   - a = "Risk A: potential misalignment..."
+   - b = "Risk B: robustness concerns..."
+[Binding] a, b bound
+
+[Position] Statement 3: loop until **analysis complete** (max: 3)
+[Loop] Starting loop
+
+[Loop] Iteration 1 of max 3
+   [Position] session "Synthesize" with context: {a, b, research}
+   [Task with serialized context]
+   [Success] Result: "Initial synthesis shows..."
+   [Loop] Evaluating: **analysis complete**
+   [Flow] Not satisfied (synthesis is preliminary)
+
+[Loop] Iteration 2 of max 3
+   [Position] session "Synthesize" with context: {a, b, research}
+   [Task with serialized context]
+   [Success] Result: "Comprehensive analysis complete..."
+   [Loop] Evaluating: **analysis complete**
+   [Flow] Satisfied!
+
+[Loop] Loop exited: condition satisfied at iteration 2
+
+[Program] Program Complete
+```
+
+---
+
+## State Categories
+
+The VM must track these state categories in narration:
+
+| Category                | What to Track                             | Example                                      |
+| ----------------------- | ----------------------------------------- | -------------------------------------------- |
+| **Import Registry**     | Imported programs and aliases             | `research: @alice/research`                  |
+| **Agent Registry**      | All agent definitions                     | `researcher: {model: sonnet, prompt: "..."}` |
+| **Block Registry**      | All block definitions (hoisted)           | `review: {params: [topic], body: [...]}`     |
+| **Input Bindings**      | Inputs received from caller               | `topic = "quantum computing"`                |
+| **Output Bindings**     | Outputs to return to caller               | `findings = "Research shows..."`             |
+| **Variable Bindings**   | Name -> value mapping (with execution_id) | `result = "..." (execution_id: 3)`           |
+| **Variable Mutability** | Which are `let` vs `const` vs `output`    | `research: let, findings: output`            |
+| **Execution Position**  | Current statement index                   | Statement 3 of 7                             |
+| **Loop State**          | Counter, max, condition                   | Iteration 2 of max 5                         |
+| **Parallel State**      | Branches, results, strategy               | `{a: complete, b: pending}`                  |
+| **Error State**         | Exception, retry count                    | Retry 2 of 3, error: "timeout"               |
+| **Call Stack**          | Stack of execution frames                 | See below                                    |
+
+### Call Stack State
+
+For block invocations, track the full call stack:
+
+```
+[CallStack] Current stack (depth: 3):
+   execution_id: 5 | block: process | depth: 3 | status: executing
+   execution_id: 3 | block: process | depth: 2 | status: waiting
+   execution_id: 1 | block: process | depth: 1 | status: waiting
+```
+
+Each frame tracks:
+
+- `execution_id`: Unique ID for this invocation
+- `block`: Name of the block
+- `depth`: Position in call stack
+- `status`: executing, waiting, or completed
+
+---
+
+## Independence from File-Based State
+
+In-context state and file-based state (`filesystem.md`) are **independent approaches**. You choose one or the other based on program complexity.
+
+- **In-context**: State lives in conversation history
+- **File-based**: State lives in `.prose/runs/{id}/`
+
+They are not designed to be complementary—pick the appropriate mode at program start.
+
+---
+
+## Summary
+
+In-context state management:
+
+1. Uses **text-prefixed markers** to track state changes
+2. Persists state in **conversation history**
+3. Is appropriate for **smaller, simpler programs**
+4. Requires **consistent narration** throughout execution
+5. Makes state **visible** in the conversation itself
+
+The narration protocol ensures that the VM can recover its execution state by reading its own prior messages. What you say becomes what you remember.