self_agency 0.0.1

data/docs/api/errors.md

@@ -0,0 +1,166 @@
+# Errors
+
+SelfAgency defines a hierarchy of errors under `SelfAgency::Error`.
+
+## Hierarchy
+
+```
+StandardError
+  └── SelfAgency::Error
+        ├── SelfAgency::GenerationError
+        ├── SelfAgency::ValidationError
+        └── SelfAgency::SecurityError
+```
+
+All SelfAgency errors inherit from `SelfAgency::Error`, which inherits from `StandardError`. This means you can catch all SelfAgency errors with a single `rescue`:
+
+```ruby
+rescue SelfAgency::Error => e
+```
+
+---
+
+## `SelfAgency::Error`
+
+Base error class. Also raised directly when configuration is missing.
+
+**Raised when:**
+
+- `_()` is called before `SelfAgency.configure`
+- `_save!` is called with no generated methods
+- `_save!` is called on an anonymous class
+
+```ruby
+SelfAgency.reset!
+Widget.new._("a method")
+#=> SelfAgency::Error: SelfAgency.configure has not been called
+```
+
+---
+
+## `SelfAgency::GenerationError`
+
+Raised when the LLM fails to produce output or when an LLM communication failure occurs.
+
+**Attributes:**
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `stage` | `Symbol` or `nil` | `:shape` or `:generate` -- which pipeline stage failed |
+| `attempt` | `Integer` or `nil` | The attempt number (during retry loop) |
+
+**Raised when:**
+
+- The shape stage returns `nil` -- message: `"Prompt shaping failed (LLM returned nil)"`
+- The generate stage returns `nil` -- message: `"Code generation failed (LLM returned nil)"`
+- An LLM communication failure occurs -- message: `"LLM request failed (ExceptionClass: details)"`
+
+!!! note
+    LLM communication failures (network errors, timeouts, provider API errors) are wrapped and re-raised as `GenerationError`. The original exception class and message are preserved in the error message. If generation consistently fails, verify your LLM provider is running and the configuration (provider, model, api_base) is correct.
+
+```ruby
+rescue SelfAgency::GenerationError => e
+  puts "LLM failed at #{e.stage} stage: #{e.message}"
+end
+```
+
+---
+
+## `SelfAgency::ValidationError`
+
+Raised when the generated code fails structural or syntactic validation.
+
+**Attributes:**
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `generated_code` | `String` or `nil` | The code that failed validation |
+| `attempt` | `Integer` or `nil` | The attempt number (during retry loop) |
+
+**Raised when:**
+
+- Generated code is empty after sanitization
+- Generated code does not contain a `def...end` structure
+- Generated code has a syntax error (`RubyVM::InstructionSequence.compile` fails)
+
+!!! note
+    During automatic retries, `ValidationError` is only raised to the caller after all `generation_retries` attempts are exhausted. The `attempt` attribute indicates which attempt produced the final failure.
+
+```ruby
+# Empty code
+widget.send(:self_agency_validate!, "")
+#=> SelfAgency::ValidationError: code is empty
+
+# Missing def...end
+widget.send(:self_agency_validate!, "puts 'hello'")
+#=> SelfAgency::ValidationError: missing def...end structure
+
+# Syntax error
+widget.send(:self_agency_validate!, "def broken\n  if true\nend")
+#=> SelfAgency::ValidationError: syntax error: ...
+```
+
+---
+
+## `SelfAgency::SecurityError`
+
+Raised when the generated code contains a dangerous pattern.
+
+**Attributes:**
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `matched_pattern` | `String` or `nil` | The specific pattern text that was matched |
+| `generated_code` | `String` or `nil` | The code that triggered the error |
+
+**Raised when:**
+
+- The code matches `SelfAgency::DANGEROUS_PATTERNS` (static analysis)
+
+The error message includes the specific matched pattern, e.g., `"dangerous pattern detected: system"`.
+
+!!! note
+    This is `SelfAgency::SecurityError`, distinct from Ruby's built-in `::SecurityError`. The runtime sandbox raises `::SecurityError` (the Ruby built-in), while the static validator raises `SelfAgency::SecurityError`.
+
+```ruby
+# System call
+widget.send(:self_agency_validate!, "def hack\n  system('ls')\nend")
+#=> SelfAgency::SecurityError: dangerous pattern detected: system
+
+# File access
+widget.send(:self_agency_validate!, "def hack\n  File.read('/etc/passwd')\nend")
+#=> SelfAgency::SecurityError: dangerous pattern detected: File.
+
+# Eval
+widget.send(:self_agency_validate!, "def hack\n  eval('1+1')\nend")
+#=> SelfAgency::SecurityError: dangerous pattern detected: eval
+```
+
+---
+
+## Error Handling Patterns
+
+### Catch All SelfAgency Errors
+
+```ruby
+begin
+  obj._("a method description")
+rescue SelfAgency::Error => e
+  puts "#{e.class}: #{e.message}"
+end
+```
+
+### Catch Specific Errors
+
+```ruby
+begin
+  obj._("a method description")
+rescue SelfAgency::GenerationError => e
+  puts "LLM failed at #{e.stage} stage (attempt #{e.attempt}): #{e.message}"
+rescue SelfAgency::ValidationError => e
+  puts "Validation failed on attempt #{e.attempt}: #{e.message}"
+  puts "Code was: #{e.generated_code}" if e.generated_code
+rescue SelfAgency::SecurityError => e
+  puts "Security: matched '#{e.matched_pattern}' in generated code"
+end
+```

data/docs/api/index.md

@@ -0,0 +1,37 @@
+# API Reference
+
+Complete reference for SelfAgency's public API.
+
+## Modules and Classes
+
+- [**SelfAgency Module**](self-agency-module.md) -- The main mixin module with `_()`, `_source_for`, `_save!`, and `on_method_generated`
+- [**Configuration**](configuration.md) -- `SelfAgency::Configuration` class and singleton methods (`configure`, `reset!`, `ensure_configured!`)
+- [**Errors**](errors.md) -- Error hierarchy: `Error`, `GenerationError`, `ValidationError`, `SecurityError`
+
+## Quick Reference
+
+### Instance Methods (from `include SelfAgency`)
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `_(description, scope:)` | `Array<Symbol>` | Generate and install methods from a description |
+| `self_agency_generate(description, scope:)` | `Array<Symbol>` | Alias for `_()` |
+| `_source_for(method_name)` | `String` or `nil` | Retrieve source code for a method |
+| `_save!(as:, path:)` | `String` | Save generated methods as a subclass file |
+| `on_method_generated(name, scope, code)` | - | Lifecycle hook (override in your class) |
+
+### Class Methods (from `extend ClassMethods`)
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `_source_for(method_name)` | `String` or `nil` | Retrieve source code at the class level |
+| `_source_versions_for(method_name)` | `Array<Hash>` | Version history for a generated method |
+
+### Module-Level Methods
+
+| Method | Returns | Description |
+|--------|---------|-------------|
+| `SelfAgency.configure { \|c\| ... }` | `Configuration` | Configure the gem (required) |
+| `SelfAgency.configuration` | `Configuration` | Access current configuration |
+| `SelfAgency.reset!` | - | Restore defaults |
+| `SelfAgency.ensure_configured!` | - | Raise if not configured |

data/docs/api/self-agency-module.md

@@ -0,0 +1,198 @@
+# SelfAgency Module
+
+The main mixin module. Include it in any class to enable LLM-powered method generation.
+
+```ruby
+class MyClass
+  include SelfAgency
+end
+```
+
+Including `SelfAgency` adds instance methods to the class and extends it with `SelfAgency::ClassMethods`.
+
+---
+
+## Instance Methods
+
+### `_(description, scope: :instance)`
+
+Generate and install one or more methods from a natural language description.
+
+**Parameters:**
+
+| Name | Type | Default | Description |
+|------|------|---------|-------------|
+| `description` | `String` | *(required)* | Natural language description of the method(s) |
+| `scope` | `Symbol` | `:instance` | One of `:instance`, `:singleton`, `:class` |
+
+**Returns:** `Array<Symbol>` -- names of the newly defined methods.
+
+**Raises:**
+
+| Exception | Condition |
+|-----------|-----------|
+| `SelfAgency::Error` | `SelfAgency.configure` has not been called |
+| `SelfAgency::GenerationError` | LLM returned `nil` at shape or generate stage |
+| `SelfAgency::ValidationError` | Generated code is empty, malformed, or has syntax errors |
+| `SelfAgency::SecurityError` | Generated code contains a dangerous pattern |
+
+**Example:**
+
+```ruby
+names = obj._("an instance method to add two integers")
+#=> [:add]
+
+names = obj._("a class method named 'self.ping' that returns 'pong'", scope: :class)
+#=> [:ping]
+```
+
+---
+
+### `self_agency_generate(description, scope: :instance)`
+
+Alias for `_()`. Provides a named alternative when `_` conflicts with other conventions (e.g., i18n):
+
+```ruby
+names = obj.self_agency_generate("a method to add two integers")
+#=> [:add]
+```
+
+---
+
+### `_source_for(method_name)`
+
+Return the source code for a method, or `nil` if unavailable.
+
+For LLM-generated methods, returns the code with the original description as a comment header. For file-defined methods, falls back to the `method_source` gem.
+
+**Parameters:**
+
+| Name | Type | Description |
+|------|------|-------------|
+| `method_name` | `Symbol` or `String` | The method to look up |
+
+**Returns:** `String` or `nil`.
+
+**Example:**
+
+```ruby
+puts obj._source_for(:add)
+# >> # an instance method to add two integers
+# >> def add(a, b)
+# >>   a + b
+# >> end
+```
+
+---
+
+### `_save!(as:, path: nil)`
+
+Save the object's generated methods as a subclass in a Ruby source file.
+
+**Parameters:**
+
+| Name | Type | Default | Description |
+|------|------|---------|-------------|
+| `as` | `String` or `Symbol` | *(required)* | Subclass name (snake_case converted to CamelCase) |
+| `path` | `String` or `nil` | `nil` | Output file path (defaults to snake_cased name + `.rb`) |
+
+**Returns:** `String` -- the file path written to.
+
+**Raises:**
+
+| Exception | Condition |
+|-----------|-----------|
+| `ArgumentError` | `as:` is not a String or Symbol |
+| `SelfAgency::Error` | No generated methods to save |
+| `SelfAgency::Error` | Parent class is anonymous |
+
+**Example:**
+
+```ruby
+path = obj._save!(as: :calculator)
+#=> "calculator.rb"
+
+path = obj._save!(as: :calculator, path: "lib/calculator.rb")
+#=> "lib/calculator.rb"
+```
+
+---
+
+### `on_method_generated(method_name, scope, code)`
+
+Lifecycle hook called once per generated method. Override in your class to persist or log generated methods.
+
+**Parameters:**
+
+| Name | Type | Description |
+|------|------|-------------|
+| `method_name` | `Symbol` | Name of the generated method |
+| `scope` | `Symbol` | `:instance`, `:singleton`, or `:class` |
+| `code` | `String` | The generated source code |
+
+**Default behavior:** No-op.
+
+**Example:**
+
+```ruby
+def on_method_generated(method_name, scope, code)
+  File.write("generated/#{method_name}.rb", code)
+end
+```
+
+---
+
+## Class Methods (via ClassMethods)
+
+### `_source_for(method_name)`
+
+Class-level version of `_source_for`. Works identically to the instance method but is called on the class.
+
+**Parameters:**
+
+| Name | Type | Description |
+|------|------|-------------|
+| `method_name` | `Symbol` or `String` | The method to look up |
+
+**Returns:** `String` or `nil`.
+
+**Example:**
+
+```ruby
+puts MyClass._source_for(:add)
+```
+
+---
+
+### `_source_versions_for(method_name)`
+
+Return the version history for a generated method. Each entry records the code, description, generating instance, and timestamp.
+
+**Parameters:**
+
+| Name | Type | Description |
+|------|------|-------------|
+| `method_name` | `Symbol` or `String` | The method to look up |
+
+**Returns:** `Array<Hash>` -- each Hash contains:
+
+| Key | Type | Description |
+|-----|------|-------------|
+| `:code` | `String` | The generated source code |
+| `:description` | `String` | The description passed to `_()` |
+| `:instance_id` | `Integer` | `object_id` of the instance that generated it |
+| `:at` | `Time` | When the method was generated |
+
+Returns an empty array if no versions exist.
+
+**Example:**
+
+```ruby
+obj._("add two integers")
+obj._("add two integers, raise ArgumentError if either is negative")
+
+versions = MyClass._source_versions_for(:add)
+versions.size  #=> 2
+versions.last[:at]          #=> 2025-01-31 12:34:56 -0500
+versions.last[:description] #=> "add two integers, raise ArgumentError if either is negative"
+```

data/docs/architecture/overview.md

@@ -0,0 +1,181 @@
+# Architecture Overview
+
+SelfAgency uses a two-stage LLM pipeline with multi-layer security to generate and install methods at runtime.
+
+## Pipeline
+
+```mermaid
+flowchart TD
+    A["User calls _('description')"] --> B[Acquire per-class mutex]
+    B --> C[ensure_configured!]
+    C --> D[Shape Stage]
+    D --> E{Shaped spec nil?}
+    E -->|Yes| F[Raise GenerationError]
+    E -->|No| G[Generate Stage]
+    G --> H{Raw code nil?}
+    H -->|Yes| F
+    H -->|No| I[Sanitize]
+    I --> J[Validate]
+    J --> K{Valid?}
+    K -->|No| L{Retries left?}
+    L -->|Yes| M[Feed error + code back to LLM]
+    M --> G
+    L -->|No| N[Raise ValidationError or SecurityError]
+    K -->|Yes| O[Sandbox Eval]
+    O --> P[Split Methods]
+    P --> Q[Store Source + Version History]
+    Q --> R[Fire on_method_generated Hook]
+    R --> S["Return Array<Symbol>"]
+```
+
+## Stage 1: Shape
+
+The shape stage rewrites a casual language description into a precise Ruby method specification. It uses ERB templates from the `shape/` directory.
+
+The LLM receives class context:
+
+- **Class name** -- e.g., `Calculator`
+- **Instance variables** -- e.g., `@data, @name`
+- **Public methods** -- e.g., `add, subtract, mean`
+- **Scope instruction** -- e.g., "This will be an instance method available on all instances of the class."
+
+The shape stage does **not** produce code. It produces a refined natural language specification that the generate stage can work with reliably.
+
+## Stage 2: Generate
+
+The generate stage takes the shaped specification and produces a `def...end` block. It uses templates from the `generate/` directory.
+
+The LLM receives the same class context plus the shaped specification from stage 1.
+
+If validation or security checks fail, the generate stage retries up to `generation_retries` times (default: 3). On each retry, the previous error message and failed code are injected into the generate template via `previous_error` and `previous_code` variables, allowing the LLM to self-correct.
+
+## Post-Processing
+
+After generation, the raw LLM output goes through three steps:
+
+### Sanitize
+
+Strips artifacts from the LLM response:
+
+- Markdown code fences (` ```ruby ... ``` `)
+- `<think>` blocks (used by some models for chain-of-thought reasoning)
+- Leading/trailing whitespace
+
+### Validate
+
+Four checks run in sequence:
+
+1. **Non-empty** -- Code must not be blank
+2. **Structure** -- Must contain at least one `def...end` block
+3. **Security** -- Must not match any `DANGEROUS_PATTERNS`
+4. **Syntax** -- Must compile via `RubyVM::InstructionSequence.compile`
+
+### Sandbox Eval
+
+The validated code is evaluated inside a sandboxed module that includes `SelfAgency::Sandbox`. This module shadows dangerous Kernel methods, placing them ahead of Kernel in Ruby's method resolution order (MRO).
+
+Sandbox modules are **cached per scope** to prevent ancestor chain accumulation across multiple `_()` calls:
+
+| Scope | Prepend Target | Cache Level |
+|-------|---------------|-------------|
+| `:instance` | `self.class` | Per class |
+| `:singleton` | `singleton_class` | Per instance |
+| `:class` | `self.class.singleton_class` | Per class |
+
+On the first `_()` call for a given scope, a new anonymous module is created, prepended, and cached. Subsequent calls reuse the same module, defining new methods into it rather than creating additional anonymous modules.
+
+## Module Structure
+
+```mermaid
+classDiagram
+    class SelfAgency {
+        +_(description, scope) Array~Symbol~
+        +self_agency_generate(description, scope) Array~Symbol~
+        +_source_for(method_name) String?
+        +_save!(as, path) String
+        +on_method_generated(name, scope, code)
+    }
+
+    class ClassMethods {
+        +_source_for(method_name) String?
+        +_source_versions_for(method_name) Array~Hash~
+    }
+
+    class Configuration {
+        +provider Symbol
+        +model String
+        +api_base String
+        +request_timeout Integer
+        +max_retries Integer
+        +retry_interval Float
+        +template_directory String
+        +generation_retries Integer
+        +logger Proc/Logger/nil
+    }
+
+    class Sandbox {
+        -system(*) raises SecurityError
+        -exec(*) raises SecurityError
+        -spawn(*) raises SecurityError
+        -fork(*) raises SecurityError
+        -backticks(*) raises SecurityError
+        -open(*) raises SecurityError
+    }
+
+    class Validator {
+        +DANGEROUS_PATTERNS Regexp
+        -self_agency_sanitize(raw) String
+        -self_agency_validate!(code)
+    }
+
+    class Generator {
+        -self_agency_ask_with_template(name, **vars) String?
+        -self_agency_shape(prompt, scope) String?
+        -self_agency_generation_vars() Hash
+    }
+
+    class Saver {
+        -self_agency_to_class_name(value) String
+        -self_agency_to_snake_case(name) String
+        -self_agency_relative_require(output, source) String
+        -self_agency_build_subclass_source(...) String
+    }
+
+    SelfAgency --> ClassMethods : extends including class
+    SelfAgency --> Configuration : uses
+    SelfAgency --> Sandbox : includes in eval module
+    SelfAgency --> Validator : validates code
+    SelfAgency --> Generator : calls LLM
+    SelfAgency --> Saver : persists methods
+```
+
+## Thread Safety
+
+SelfAgency uses two mutexes to ensure thread-safe operation:
+
+- **`CONFIG_MUTEX`** (module-level) -- Serializes `SelfAgency.configure` and `SelfAgency.reset!` calls so that concurrent configuration changes do not interleave.
+- **Per-class mutex** (`@self_agency_mutex`) -- Initialized when a class includes `SelfAgency`. Serializes the entire `_()` pipeline per class so that concurrent method generation calls do not interfere with each other.
+
+The per-class mutex wraps the full pipeline: shape, generate, validate (with retries), eval, source storage, and lifecycle hook. This means only one thread can generate methods for a given class at a time, but different classes can generate concurrently.
+
+## File Layout
+
+```
+lib/
+  self_agency.rb            # Main module, public API, eval logic
+  self_agency/
+    version.rb              # VERSION constant
+    errors.rb               # Error hierarchy
+    configuration.rb        # Configuration class and singleton methods
+    sandbox.rb              # Runtime sandbox module
+    validator.rb            # DANGEROUS_PATTERNS, sanitize, validate!
+    generator.rb            # LLM communication and prompt shaping
+    saver.rb                # _save! helpers
+    prompts/
+      shape/
+        system.txt.erb      # Shape stage system prompt
+        user.txt.erb        # Shape stage user prompt
+      generate/
+        system.txt.erb      # Generate stage system prompt
+        user.txt.erb        # Generate stage user prompt
+```

data/docs/architecture/security.md

@@ -0,0 +1,101 @@
+# Security
+
+SelfAgency employs a two-layer security model to prevent generated code from performing dangerous operations. Both layers must pass before any code is installed.
+
+## Layer 1: Static Analysis
+
+Before code is evaluated, it is checked against `DANGEROUS_PATTERNS`, a compiled regular expression that matches known dangerous constructs:
+
+| Pattern | What It Catches |
+|---------|----------------|
+| `\b(system\|exec\|spawn\|fork\|abort\|exit)\b` | Process execution and termination |
+| `` `[^`]*` `` | Backtick shell execution |
+| `%x\{`, `%x\[`, `%x\(` | `%x` shell execution syntax |
+| `\bFile\.\b` | File system access |
+| `\bIO\.\b` | I/O operations |
+| `\bKernel\.\b` | Direct Kernel calls |
+| `\bOpen3\.\b` | Advanced process spawning |
+| `\bProcess\.\b` | Process management |
+| `\brequire\b` | Loading external code |
+| `\bload\b` | Loading external code |
+| `\b__send__\b` | Method dispatch bypass |
+| `\beval\b` | Dynamic code evaluation |
+| `\bsend\b` | Method dispatch (`send`) |
+| `\bpublic_send\b` | Method dispatch (`public_send`) |
+| `\bmethod\s*\(` | Method object retrieval |
+| `\bconst_get\b` | Constant lookup bypass |
+| `\bclass_eval\b` | Class-level eval |
+| `\bmodule_eval\b` | Module-level eval |
+| `\binstance_eval\b` | Instance-level eval |
+| `\binstance_variable_set\b` | Direct ivar write |
+| `\binstance_variable_get\b` | Direct ivar read |
+| `\bdefine_method\b` | Dynamic method definition |
+| `\bBinding\b` | Binding access |
+| `\bBasicObject\b` | BasicObject escape hatch |
+| `\bremove_method\b` | Method removal |
+| `\bundef_method\b` | Method undefinition |
+
+If any pattern matches, a `SelfAgency::SecurityError` is raised and the code is **not evaluated**.
+
+```ruby
+# These all raise SelfAgency::SecurityError:
+"def hack\n  system('ls')\nend"
+"def hack\n  File.read('/etc/passwd')\nend"
+"def hack\n  eval('1+1')\nend"
+"def hack\n  require 'socket'\nend"
+```
+
+## Layer 2: Runtime Sandbox
+
+Even if static analysis were bypassed, the runtime sandbox provides a second line of defense. Every generated method is evaluated inside an anonymous module that includes `SelfAgency::Sandbox`:
+
+```ruby
+module SelfAgency::Sandbox
+  private
+
+  def system(*)  = raise(::SecurityError, "system() blocked by SelfAgency sandbox")
+  def exec(*)    = raise(::SecurityError, "exec() blocked by SelfAgency sandbox")
+  def spawn(*)   = raise(::SecurityError, "spawn() blocked by SelfAgency sandbox")
+  def fork(*)    = raise(::SecurityError, "fork() blocked by SelfAgency sandbox")
+  def `(*)       = raise(::SecurityError, "backtick execution blocked by SelfAgency sandbox")
+  def open(*)    = raise(::SecurityError, "open() blocked by SelfAgency sandbox")
+end
+```
+
+Because the sandbox module is included in the anonymous module that wraps the generated code, its methods appear **ahead of Kernel** in Ruby's method resolution order (MRO). Any call to `system`, `exec`, `spawn`, `fork`, backticks, or `open` from within a generated method raises `::SecurityError` at runtime.
+
+## Validation Pipeline
+
+The full validation sequence runs in order. The first failure stops evaluation:
+
+```mermaid
+flowchart LR
+    A[Raw LLM Output] --> B[Sanitize]
+    B --> C{Empty?}
+    C -->|Yes| D[ValidationError]
+    C -->|No| E{"Has def...end?"}
+    E -->|No| D
+    E -->|Yes| F{Dangerous pattern?}
+    F -->|Yes| G[SecurityError]
+    F -->|No| H{Syntax valid?}
+    H -->|No| D
+    H -->|Yes| I[Sandbox Eval]
+```
+
+1. **Sanitize** -- Strip markdown fences, `<think>` blocks, whitespace
+2. **Empty check** -- Raise `ValidationError` if code is blank
+3. **Structure check** -- Raise `ValidationError` if no `def...end` block found
+4. **Pattern check** -- Raise `SecurityError` if `DANGEROUS_PATTERNS` matches
+5. **Syntax check** -- Raise `ValidationError` if `RubyVM::InstructionSequence.compile` fails
+6. **Sandbox eval** -- Evaluate inside sandboxed anonymous module
+
+## Limitations
+
+The security model is designed for defense-in-depth against accidental or LLM-hallucinated dangerous code. It is **not** a full sandboxing solution:
+
+- Static patterns can potentially be bypassed through creative obfuscation
+- The runtime sandbox only shadows six specific Kernel methods
+- Generated code has access to the full Ruby standard library (except blocked methods)
+- Network access (e.g., `Net::HTTP`) is not blocked by default
+
+For production use, consider additional controls such as network-level restrictions, process isolation, or reviewing generated code before deployment (see [`_save!`](../guide/saving-methods.md)).