@rcrsr/rill 0.1.0

package/docs/10_parsing.md

@@ -0,0 +1,366 @@
+# Content Parsing
+
+Built-in functions for extracting structured data from text, optimized for LLM output.
+
+## Overview
+
+| Function | Purpose |
+|----------|---------|
+| `parse_auto` | Auto-detect format and extract structured content |
+| `parse_json` | Parse JSON with error repair |
+| `parse_xml` | Extract content from XML tags |
+| `parse_fence` | Extract fenced code block content |
+| `parse_fences` | Extract all fenced code blocks |
+| `parse_frontmatter` | Parse YAML frontmatter and body |
+| `parse_checklist` | Parse markdown checkbox items |
+
+## Auto-Detection with `parse_auto`
+
+Automatically detects format and extracts structured content:
+
+```rill
+$response -> parse_auto :> $result
+
+# $result contains:
+#   type: "json" | "xml" | "yaml" | "frontmatter" | "fence" | "checklist" | "text"
+#   data: <parsed structured data>
+#   raw: <original extracted content>
+#   confidence: <0.0-1.0 detection confidence>
+#   repaired: <true if error recovery applied>
+```
+
+### Detection Priority
+
+| Priority | Format | Detection Signal |
+|----------|--------|------------------|
+| 1 | frontmatter | Starts with `---\n` |
+| 2 | fence (json/yaml) | ` ```json ` or ` ```yaml ` blocks |
+| 3 | fence (other) | ` ```lang ` blocks |
+| 4 | xml | `<tag>...</tag>` structure |
+| 5 | json | `{...}` or `[...]` with balanced braces |
+| 6 | checklist | `- [ ]` or `- [x]` patterns |
+| 7 | yaml | `key: value` line patterns (2+ lines) |
+| 8 | text | Fallback (returns trimmed input) |
+
+### Examples
+
+```rill
+"{{\"count\": 42}}" -> parse_auto
+# type: "json", data: [count: 42]
+```
+
+```rill
+"<answer>42</answer>" -> parse_auto
+# type: "xml", data: [answer: "42"]
+```
+
+```rill
+"- [ ] Todo\n- [x] Done" -> parse_auto
+# type: "checklist", data: [[false, "Todo"], [true, "Done"]]
+
+# Frontmatter
+"---\ntitle: Doc\n---\nBody" -> parse_auto
+# type: "frontmatter", data: [meta: [title: "Doc"], body: "Body"]
+```
+
+### Confidence Checking
+
+Use confidence scores to handle ambiguous responses:
+
+```rill
+prompt("Analyze this data") -> parse_auto :> $parsed
+
+($parsed.confidence < 0.8) ? {
+  "Low confidence parse: {$parsed.type}" -> log
+  # Fall back to manual handling
+}
+```
+
+## JSON Parsing
+
+### `parse_json`
+
+Parses JSON with automatic error repair for common LLM formatting issues:
+
+```rill
+"{{\"name\": \"test\", \"count\": 42}}" -> parse_json
+# Returns [name: "test", count: 42]
+```
+
+### Error Repair
+
+| Error | Repair |
+|-------|--------|
+| Trailing commas | Removed |
+| Single quotes | Converted to double quotes |
+| Unquoted keys | Quoted |
+| Unclosed braces | Closed (best effort) |
+
+### Practical Usage
+
+```rill
+# LLM returns JSON with common errors
+prompt("Return user data as JSON") -> parse_json :> $user
+
+# Safe access with defaults
+$user.name ?? "Unknown"
+$user.age ?? 0
+```
+
+## XML Parsing
+
+### `parse_xml`
+
+Extract content from XML tags. Useful for chain-of-thought prompting:
+
+```rill
+# Extract single tag
+$response -> parse_xml("thinking")
+# Returns content between <thinking>...</thinking>
+
+# Extract and parse nested content
+$response -> parse_xml("answer") -> parse_json
+# Extract <answer> tag, parse its content as JSON
+
+# Extract all tags as dict
+$response -> parse_xml
+# Returns [tag1: "content1", tag2: "content2", ...]
+```
+
+### Chain-of-Thought Pattern
+
+```rill
+prompt(<<EOF
+Analyze this problem step by step.
+
+<thinking>
+Show your reasoning here
+</thinking>
+
+<answer>
+Return your final answer as JSON
+</answer>
+EOF
+) :> $response
+
+# Log reasoning for debugging
+$response -> parse_xml("thinking") -> log
+
+# Parse structured answer
+$response -> parse_xml("answer") -> parse_json :> $result
+```
+
+### Tool Calling Pattern
+
+```rill
+prompt(<<EOF
+You have access to tools. To call a tool, use:
+<tool>
+  <name>tool_name</name>
+  <args>{"param": "value"}</args>
+</tool>
+EOF
+) :> $response
+
+$response -> parse_xml("tool") :> $tool
+$tool -> parse_xml("name") :> $fn_name
+$tool -> parse_xml("args") -> parse_json :> $fn_args
+
+# Call the function dynamically
+call($fn_name, $fn_args)
+```
+
+## Fenced Code Blocks
+
+### `parse_fence`
+
+Extract content from markdown fenced code blocks:
+
+```rill
+# Extract by language
+$response -> parse_fence("json") -> parse_json
+# Extracts ```json block, parses as JSON
+
+# Extract first fence (any language)
+$response -> parse_fence
+# Returns content of first fenced block
+```
+
+### `parse_fences`
+
+Extract all fenced code blocks:
+
+```rill
+prompt("Show examples in Python and JavaScript") -> parse_fences -> each {
+  "{$.lang}:" -> log
+  $.content -> log
+}
+# Returns list of [lang: "python", content: "..."], [lang: "javascript", content: "..."]
+```
+
+### Practical Usage
+
+```rill
+# ...
+# Parse fenced JSON from LLM response
+# prompt("Generate JSON") -> parse_fence("json") -> parse_json :> $config
+# $config.host -> log
+```
+
+## Frontmatter Parsing
+
+### `parse_frontmatter`
+
+Parse YAML frontmatter delimited by `---`:
+
+```rill
+"---\ntitle: Hello\n---\nBody text" :> $doc
+$doc -> parse_frontmatter
+# Returns [meta: [title: "Hello"], body: "Body text"]
+```
+
+Destructure into variables:
+
+```rill
+"---\ntitle: Hello\n---\nBody text" :> $doc
+$doc -> parse_frontmatter -> *<meta: $m, body: $b>
+$m.title -> log
+```
+
+### Practical Usage
+
+```rill
+# Parse frontmatter from document
+"---\ntitle: My Doc\nstatus: draft\n---\nContent here" -> parse_frontmatter :> $doc
+
+$doc.meta.title -> log
+($doc.meta.status ?? "") -> .eq("draft") ? { "Document is still a draft" -> log }
+$doc.body -> log
+```
+
+## Checklist Parsing
+
+### `parse_checklist`
+
+Parse markdown task list items:
+
+```rill
+"- [ ] Buy milk\n- [x] Call mom" -> parse_checklist
+# Returns [[false, "Buy milk"], [true, "Call mom"]]
+```
+
+Each item is a tuple: `[completed: bool, text: string]`
+
+### Practical Usage
+
+```rill
+"- [ ] Deploy\n- [x] Test" -> parse_checklist :> $tasks
+$tasks -> .len
+```
+
+## Validation Patterns
+
+### Type Checking
+
+```text
+app::prompt("Return JSON with status and items") -> parse_auto :> $result
+
+($result.type != "json") ? {
+  app::error("Expected JSON response, got {$result.type}")
+}
+
+$result.data -> app::process()
+```
+
+### Required Fields
+
+```text
+app::prompt("Return user profile as JSON") -> parse_json :> $user
+
+# Validate required fields exist
+($user.?name && $user.?email) ? {
+  app::create_account($user)
+} ! {
+  app::error("Missing required fields")
+}
+```
+
+### Retry on Parse Failure
+
+```text
+^(limit: 3) @ {
+  app::prompt("Generate valid JSON for a user profile")
+} ? (parse_json($) -> type != "dict")
+
+# Loop exits when valid dict is returned
+parse_json($) :> $profile
+```
+
+## Combining Parsers
+
+Chain parsers for complex extraction:
+
+```rill
+# ...
+# Extract JSON from a fenced block
+# $response -> parse_fence("json") -> parse_json :> $data
+
+# Extract XML answer, parse as JSON
+# $response -> parse_xml("answer") -> parse_json :> $result
+
+# Parse frontmatter, then parse body as checklist
+# $doc -> parse_frontmatter :> $parsed
+# $parsed.body -> parse_checklist :> $tasks
+```
+
+## Best Practices
+
+1. **Use `parse_auto` for unknown formats** — Let it detect the structure
+2. **Use specific parsers when format is known** — More predictable results
+3. **Check confidence for ambiguous content** — Handle low-confidence parses specially
+4. **Chain parsers for nested structures** — XML containing JSON, frontmatter with checklists
+5. **Validate before using** — Check type and required fields exist
+
+## Limitations
+
+These parsers handle common LLM output patterns with zero external dependencies. For full-featured parsing (nested YAML, XML attributes, streaming JSON), host applications can register their own functions via `RuntimeContext`.
+
+### YAML Parser
+
+The YAML parser handles **flat `key: value` format only**:
+
+```text
+# Supported
+title: My Document
+count: 42
+enabled: true
+
+# NOT supported (returns raw string or fails)
+nested:
+  child: value
+items:
+  - one
+  - two
+multiline: |
+  long text
+  across lines
+```
+
+### JSON Repair
+
+Repairs common LLM errors but **cannot fix**:
+
+| Repaired | Not Repaired |
+|----------|--------------|
+| Trailing commas | Missing commas between items |
+| Single quotes on values | Malformed string escapes |
+| Unquoted keys | Truncated mid-string |
+| Unclosed braces | Invalid Unicode |
+
+### XML Parser
+
+Extracts simple `<tag>content</tag>` but **does not support**:
+
+- Attribute extraction (`<tag attr="val">` — attr ignored)
+- Nested same-name tags (`<item><item>...</item></item>`)
+- CDATA sections, namespaces, self-closing tags

package/docs/11_reference.md

@@ -0,0 +1,350 @@
+# rill Core Language Specification v0.1.0
+
+*From prompts to workflows*
+
+rill is a pipe-based scripting language for orchestrating workflows.
+
+> **Experimental (v0.1.0).** Active development. Breaking changes until v1.0.
+
+## Overview
+
+rill is an imperative scripting language that is dynamically typed and type-safe. Types are checked at runtime, but type errors are always caught—there are no implicit conversions. Type annotations are optional, but variables lock their type on first assignment. The language is value-based: no references, all copies are deep, all comparisons are by value. Empty values are valid (empty strings, lists, dicts), but null and undefined do not exist. Control flow is singular: no exceptions, no try/catch. Data flows through pipes (`->`), not assignment.
+
+## Design Principles
+
+1. **Pipes over assignment** — Data flows via `->`. No `=` operator exists.
+2. **Pattern-driven decisions** — Check response patterns, branch accordingly.
+3. **Singular control flow** — No try/catch. Errors halt execution. Recovery requires explicit conditionals.
+4. **Value-based** — All values compare by content. No references, no object identity.
+5. **No null/undefined** — Empty values are valid, but "no value" cannot be represented.
+6. **No magic** — No truthiness, no automatic type conversions. Explicit behavior only.
+7. **Expressive and consistent** — Small syntax, coherent semantics. Every construct composes predictably.
+
+---
+
+## Quick Reference Tables
+
+### Operators
+
+| Category | Operators |
+|----------|-----------|
+| Arithmetic | `+`, `-`, `*`, `/`, `%` |
+| Comparison | `==`, `!=`, `<`, `>`, `<=`, `>=` |
+| Comparison | `.eq`, `.ne`, `.lt`, `.gt`, `.le`, `.ge` methods |
+| Logical | `!` (unary), `&&`, `\|\|` |
+| Pipe | `->` |
+| Capture | `:>` |
+| Spread | `@` (sequential), `*` (tuple) |
+| Extraction | `*<>` (destructure), `/<>` (slice) |
+| Type | `:type` (assert), `:?type` (check) |
+| Member | `.field`, `[index]` |
+| Default | `?? value` |
+| Existence | `.?field`, `.?field&type` |
+
+See [Operators](04_operators.md) for detailed documentation.
+
+### Control Flow
+
+| Syntax | Description |
+|--------|-------------|
+| `cond ? then ! else` | Conditional (if-else) |
+| `$val -> ? then ! else` | Piped conditional ($ as condition) |
+| `(cond) @ body` | While loop |
+| `@ body ? cond` | Do-while |
+| `break` / `$val -> break` | Exit loop |
+| `return` / `$val -> return` | Exit block or script |
+
+See [Control Flow](05_control-flow.md) for detailed documentation. Script-level exit functions must be host-provided.
+
+### Collection Operators
+
+| Syntax | Description |
+|--------|-------------|
+| `-> each { body }` | Sequential iteration, all results |
+| `-> each(init) { body }` | Sequential with accumulator (`$@`) |
+| `-> map { body }` | Parallel iteration, all results |
+| `-> filter { cond }` | Parallel filter, matching elements |
+| `-> fold(init) { body }` | Sequential reduction, final result |
+
+See [Collections](07_collections.md) for detailed documentation.
+
+### Types
+
+| Type | Syntax | Example |
+|------|--------|---------|
+| String | `"text"` | `"hello"` |
+| Number | `123`, `0.5` | `42`, `0.9` |
+| Bool | `true`, `false` | `true` |
+| List | `[a, b]` | `["file.ts", 42]` |
+| Dict | `[k: v]` | `[output: "text", code: 0]` |
+| Tuple | `*[...]` | `*[1, 2]`, `*[x: 1, y: 2]` |
+| Closure | `\|\|{ }` | `\|x\|($x * 2)` |
+
+See [Types](02_types.md) for detailed documentation.
+
+### Functions
+
+| Syntax | Description |
+|--------|-------------|
+| `\|p: type\|{ } :> $fn` | Define and capture function |
+| `\|p = default\|{ }` | Parameter with default |
+| `$fn(arg)` | Call function directly |
+| `arg -> $fn()` | Call function with pipe value |
+| `arg -> $fn` | Pipe-style invoke |
+
+See [Closures](06_closures.md) for detailed documentation.
+
+### Special Variables
+
+| Variable | Contains | Source |
+|----------|----------|--------|
+| `$` | Piped value (current scope) | Grammar |
+| `$ARGS` | CLI positional args (list) | Runtime |
+| `$ENV.NAME` | Environment variable | Runtime |
+| `$name` | Named variable | Runtime |
+
+See [Variables](03_variables.md) for detailed documentation.
+
+### `$` Binding by Context
+
+| Context | `$` contains |
+|---------|--------------|
+| Inline block `-> { }` | Piped value |
+| Each loop `-> each { }` | Current item |
+| While-loop `(cond) @ { }` | Accumulated value |
+| Do-while `@ { } ? cond` | Accumulated value |
+| Conditional `cond ? { }` | Tested value |
+| Piped conditional `-> ? { }` | Piped value |
+| Stored closure `\|x\|{ }` | N/A — use params |
+| Dict closure `\|\|{ $.x }` | Dict self |
+
+### Property Access
+
+| Syntax | Description |
+|--------|-------------|
+| `$data.field` | Literal field access |
+| `$data[0]`, `$data[-1]` | Index access |
+| `$data.$key` | Variable as key |
+| `$data.($i + 1)` | Computed key |
+| `$data.(a \|\| b)` | Alternative keys |
+| `$data.field ?? default` | Default if missing |
+| `$data.?field` | Existence check |
+
+### Parsing Functions
+
+| Function | Description |
+|----------|-------------|
+| `parse_auto` | Auto-detect and extract structured content |
+| `parse_json` | Parse JSON with error repair |
+| `parse_xml(tag?)` | Extract content between XML tags |
+| `parse_fence(lang?)` | Extract fenced code block content |
+| `parse_fences` | Extract all fenced blocks as list |
+| `parse_frontmatter` | Parse `---` delimited YAML frontmatter |
+| `parse_checklist` | Parse `- [ ]` and `- [x]` items |
+
+See [Parsing](10_parsing.md) for detailed documentation.
+
+### Core Methods
+
+| Method | Input | Output | Description |
+|--------|-------|--------|-------------|
+| `.str` | Any | String | Convert to string |
+| `.num` | Any | Number | Convert to number |
+| `.len` | Any | Number | Length |
+| `.trim` | String | String | Remove whitespace |
+| `.head` | String/List | Any | First element |
+| `.tail` | String/List | Any | Last element |
+| `.at(idx)` | String/List | Any | Element at index |
+| `.split(sep)` | String | List | Split by separator |
+| `.join(sep)` | List | String | Join with separator |
+| `.lines` | String | List | Split on newlines |
+| `.lower` | String | String | Lowercase |
+| `.upper` | String | String | Uppercase |
+| `.replace(pat, repl)` | String | String | Replace first match |
+| `.replace_all(pat, repl)` | String | String | Replace all matches |
+| `.contains(text)` | String | Bool | Contains substring |
+| `.starts_with(pre)` | String | Bool | Starts with prefix |
+| `.ends_with(suf)` | String | Bool | Ends with suffix |
+| `.match(regex)` | String | Dict | First match info |
+| `.is_match(regex)` | String | Bool | Regex matches |
+| `.empty` | Any | Bool | Is empty |
+| `.keys` | Dict | List | All keys |
+| `.values` | Dict | List | All values |
+| `.entries` | Dict | List | Key-value pairs |
+
+See [Strings](09_strings.md) for detailed string method documentation.
+
+### Global Functions
+
+| Function | Description |
+|----------|-------------|
+| `type` | Returns type name of value |
+| `log` | Print to console, pass through |
+| `json` | Convert to JSON string |
+| `identity` | Returns input unchanged |
+| `range(start, end, step?)` | Generate number sequence |
+| `repeat(value, count)` | Repeat value n times |
+| `enumerate(collection)` | Add index to elements |
+
+See [Iterators](08_iterators.md) for `range` and `repeat` documentation.
+
+---
+
+## Implied `$`
+
+When constructs appear without explicit input, `$` is used implicitly:
+
+| Written | Equivalent to |
+|---------|---------------|
+| `? { }` | `$ -> ? { }` |
+| `.method()` | `$ -> .method()` |
+| `$fn()` | `$fn($)` (when no args, no default) |
+
+---
+
+## Extraction Operators
+
+### Destructure `*<>`
+
+Extract elements from lists or dicts into variables:
+
+```rill
+[1, 2, 3] -> *<$a, $b, $c>
+# $a = 1, $b = 2, $c = 3
+
+[name: "test", count: 42] -> *<name: $n, count: $c>
+# $n = "test", $c = 42
+```
+
+### Slice `/<>`
+
+Extract portions using Python-style `start:stop:step`:
+
+```rill
+[0, 1, 2, 3, 4] -> /<0:3>        # [0, 1, 2]
+[0, 1, 2, 3, 4] -> /<-2:>        # [3, 4]
+[0, 1, 2, 3, 4] -> /<::-1>       # [4, 3, 2, 1, 0]
+"hello" -> /<1:4>                # "ell"
+```
+
+See [Operators](04_operators.md) for detailed extraction operator documentation.
+
+---
+
+## Tuples
+
+Tuples package values for argument unpacking:
+
+```rill
+|a, b, c|"{$a}-{$b}-{$c}" :> $fmt
+*[1, 2, 3] -> $fmt()             # "1-2-3"
+*[c: 3, a: 1, b: 2] -> $fmt()    # "1-2-3" (named)
+```
+
+See [Types](02_types.md) for detailed tuple documentation.
+
+---
+
+## Runtime Limits
+
+### Iteration Limits
+
+Loops have a default maximum of **10,000 iterations**. Override with `^(limit: N)`:
+
+```rill
+^(limit: 100) 0 -> ($ < 50) @ { $ + 1 }
+```
+
+### Concurrency Limits
+
+The `^(limit: N)` annotation also controls parallel concurrency in `map`:
+
+```text
+^(limit: 3) $items -> map { slow_process($) }
+```
+
+See [Host Integration](14_host-integration.md) for timeout and cancellation configuration.
+
+---
+
+## Host-Provided Functions
+
+rill is a vanilla language. The host application registers domain-specific functions via `RuntimeContext`:
+
+```typescript
+const ctx = createRuntimeContext({
+  functions: {
+    prompt: async (args) => await callLLM(args[0]),
+    'io::read': async (args) => fs.readFile(String(args[0]), 'utf-8'),
+  },
+  variables: {
+    config: { apiUrl: 'https://api.example.com' },
+  },
+});
+```
+
+Scripts call these as `prompt("text")` or `io::read("file.txt")`.
+
+See [Host Integration](14_host-integration.md) for complete API documentation.
+
+---
+
+## Script Frontmatter
+
+Optional YAML frontmatter between `---` markers. **Frontmatter is opaque to rill**—the host interprets it:
+
+```text
+---
+timeout: 00:10:00
+args: file: string
+---
+
+process($file)
+```
+
+---
+
+## Script Return Values
+
+Scripts return their last expression:
+
+| Return | Exit Code |
+|--------|-----------|
+| `true` / non-empty string | 0 |
+| `false` / empty string | 1 |
+| `[0, "message"]` | 0 with message |
+| `[1, "message"]` | 1 with message |
+
+---
+
+## Comments
+
+Single-line comments start with `#`:
+
+```rill
+# This is a comment
+"hello"  # inline comment
+```
+
+---
+
+## Grammar
+
+The complete formal grammar is in [grammar.ebnf](15_grammar.ebnf).
+
+---
+
+## Topic Guides
+
+For detailed documentation on specific topics:
+
+- [Types](02_types.md) — Type system, type assertions
+- [Variables](03_variables.md) — Declaration, scope, `$` binding
+- [Control Flow](05_control-flow.md) — Conditionals, loops
+- [Operators](04_operators.md) — All operators
+- [Closures](06_closures.md) — Late binding, dict closures
+- [Collections](07_collections.md) — `each`, `map`, `filter`, `fold`
+- [Iterators](08_iterators.md) — `range`, `repeat`, `.first()`
+- [Strings](09_strings.md) — String methods
+- [Parsing](10_parsing.md) — Content parsing
+- [Host Integration](14_host-integration.md) — Embedding API