silas 0.5.0 → 0.6.1

data/docs/evals.md

@@ -0,0 +1,95 @@
+# Evals
+
+Agent evals answer the question tests can't: *given these model decisions, does
+the durable machinery do the right thing?* A scenario scripts the **model's
+decisions** step by step; everything else is real — the real registry resolves
+your real tools, the real Ledger enforces effect modes and approvals, and the
+assertions read the genuine durable transcript, not a mock.
+
+They run keyless and deterministically, which makes them a **deploy gate**:
+
+```sh
+bin/rails silas:eval     # loads <eval_dir>/**/*_eval.rb; exits 1 on failure
+```
+
+The installer wires this into `bin/ci` (or tells you to add it to yours), and
+the application template generates a working example in
+`test/agent_evals/refund_desk_eval.rb`.
+
+## A scenario
+
+```ruby
+Silas::Eval.scenario "over the gate: a £64 refund holds at the signal" do
+  input "The walnut monitor stand (order R-1002) arrived cracked."
+
+  on_step 0, call: { name: "lookup_order", arguments: { number: "R-1002" } }
+  on_step 1, call: { name: "issue_refund",
+                     arguments: { number: "R-1002", amount_pence: 6400, reason: "arrived cracked" } }
+
+  expect do
+    assert_parked tool: "issue_refund"   # the turn holds at zero compute
+    assert_no_tool_called "issue_refund" # and the refund row does NOT exist
+  end
+end
+```
+
+`on_step index, ...` is the script: what the "model" returns on that model
+call. `text:` is an assistant text block, `call:`/`calls:` are tool calls. A
+step index with no entry ends the scripted conversation.
+
+## The DSL
+
+| Call | Meaning |
+|---|---|
+| `input "…"` | The turn's input. Required. |
+| `on_step i, text:, call:, calls:, data:` | The scripted model output for step `i`. `data:` scripts a **structured answer** (the `final_answer` schema case) — it becomes the same block the real adapter persists, so `assert_answer_data` reads it back. |
+| `approve tool: "name"` | When the turn parks on this tool, approve it (recorded as approved by `"eval"` — the same `approve!` the inbox uses) and resume. This is how you assert exactly-once **across** a hold. |
+| `stub_tool "name", effect_mode:, approval: { \|**args\| … }` | Replace a real tool with a stub for a side-effect-free scenario. Unstubbed tools stay real. |
+| `mode :real` | Drive a real model instead of the script. **Skipped automatically when `ANTHROPIC_API_KEY` is unset**, so the gate stays green offline. |
+| `max_steps n` | Override the per-turn step cap for this scenario. |
+| `metadata h` / tags on `scenario "…", tags: [...]` | Stamped on the session / used for filtering. |
+| `expect { … }` | The assertion block. Required. Runs after the turn reaches its resting state. |
+
+## Assertions
+
+All assertions read only the durable rows, and they **collect** failures — one
+run reports every miss, not just the first.
+
+| Assertion | Checks |
+|---|---|
+| `assert_tool_called "name", times: 1` | The tool actually executed (`times:` for exactly-N — the exactly-once assertion). |
+| `assert_no_tool_called "name"` | No execution happened (e.g. because the turn parked first). |
+| `assert_tool_arg "name", :key, value` (or a block predicate) | An argument the model passed. |
+| `assert_parked tool: "name"` | The turn is waiting, held on that tool's approval (or in-doubt). |
+| `assert_approved tool: "name"` | The invocation's approval state is `approved`. |
+| `assert_turn_completed` / `assert_turn_failed(reason: "…")` | Terminal state. |
+| `assert_final_matches(/…/)` | The final answer text. |
+| `assert_answer_data(key: :verdict, value: "approve")` | The structured `final_answer` payload (whole-Hash and predicate forms too). |
+| `assert_no_hallucinated_price(allowed: [])` | Every money amount in the final answer traces to a number the agent actually saw (tool results or the input), allowing pence↔pounds scaling. |
+| `assert_rubric "…" ` | LLM-graded check against `config.eval_grader` — **skips offline** rather than failing the gate (set `SILAS_EVAL_STRICT=1` to make skips fail). |
+
+## How a scenario runs (so you can trust it)
+
+The driver starts a real session (`Silas.agent.start`), runs the real
+`AgentLoopJob` inline on the `:test` queue adapter, and — for each `approve
+tool:` — performs the same `approve!` an operator would, then resumes the
+loop. Scripted mode swaps only the inference adapter; `mode :real` uses your
+configured one.
+
+Two consequences worth knowing:
+
+- **Rows persist.** Scenarios commit ordinary durable rows to the database of
+  the environment you run them in (they need real seeds for tools that read
+  your tables — the template's CI runs `db:seed silas:eval`). Run against a
+  scratch database if residue bothers you.
+- **Config is restored per run**, but scenarios execute sequentially in one
+  process — don't have two scenarios fight over global state you set yourself.
+
+## Configuration
+
+```ruby
+Silas.configure do |c|
+  c.eval_dir = "test/agent_evals"   # where *_eval.rb files live
+  # c.eval_grader = ->(prompt) { … } # custom LLM grader for assert_rubric
+end
+```

data/docs/guarantees.md

@@ -0,0 +1,76 @@
+# Guarantees
+
+Durability in Silas is a contract, not a slide. Everything on this page is
+verified by the in-repo chaos harness (`chaos_host/bin/chaos`), which
+`kill -9`s live agents hundreds of times per release — the current gate is
+100/100 completions per mode across a 295-run matrix, zero duplicate effects,
+byte-identical replay, on SQLite and Postgres. Results live in
+`chaos_host/results/` and every run is reproducible.
+
+## Turns survive hard process death
+
+Worker `kill -9`, whole-tree `kill -9`, SIGTERM deploys — a turn resumes from
+its last completed step. Each step checkpoints through Active Job
+Continuations; the resume replays completed work **from rows**, never by
+re-calling the model or re-running settled tools.
+
+## `transactional!` tools execute exactly once
+
+The tool's database writes and the ledger's record of "this ran" commit or
+roll back **in one transaction**:
+
+- **Crash before commit** → both roll back; resume runs the tool from a clean
+  slate. No orphan effect.
+- **Crash after commit** → the ledger row says `completed`; resume skips the
+  step. No second effect.
+
+No idempotency key is required, because the dedup lives in your database — the
+same transaction as the effect. This is the guarantee that requires the ledger
+to live *inside* your app: a runtime outside your database can retry and
+reconcile, but it can't make its record and your effect one atomic commit.
+
+## Ambiguity parks instead of guessing
+
+The default effect mode is `at_most_once!`: when a crash makes an execution
+ambiguous ("did the email send?"), the invocation parks **in doubt** for a
+human verdict rather than re-firing blind. `idempotent!` is the explicit
+opt-in to automatic re-runs. Never double-pay; sometimes ask.
+
+## Approvals park at zero compute
+
+A turn awaiting a human exits the worker entirely — no held thread, no polling
+loop. Approving enqueues a fresh job that replays from rows. Parks expire
+(default 7 days, `config.approval_ttl`) rather than ghosting forever.
+
+## Errors can't strand a turn
+
+Transient model errors (rate limits, overloads, timeouts) back off and retry
+from the checkpoint. Exhausted retries and permanent rejections (bad key, bad
+request) expire pending approvals and fail the turn loudly. And a turn can
+never sit in `running` forever: the recurring `DeadJobRescuerJob` retries jobs
+failed by dead-worker reaping and fails turns stranded by a loop job that died
+outside the retry list. The rescuer is part of the contract — keep it in
+`config/recurring.yml`, and monitor worker liveness (the rescuer can requeue
+work; it cannot conjure a consumer).
+
+## Deploys can't corrupt a run
+
+Instructions are snapshotted per turn. Tools, skills, connections, and the
+final-answer schema are model-visible state, captured in a definitions digest —
+a deploy that changes them while a turn is parked fails that turn loudly on
+resume (`NondeterminismError`) instead of quietly resuming into a different
+agent. Settle parked turns before shipping agent changes.
+
+## Compaction can't corrupt a replay
+
+Long conversations summarise past `config.compact_at` — but a summary is a
+**persisted, exactly-once effect** (claimed compare-and-swap, written once),
+never a rebuild-time computation. The message array a resumed turn sees is
+byte-identical to the one the crashed turn saw.
+
+## The one rule you owe the contract
+
+Run agents on Solid Queue (or `:inline` for scripts) — never ActiveJob's
+in-process `:async` adapter, which runs a re-enqueued continuation
+concurrently with the original and double-executes steps. Silas raises on it
+in production; `bin/rails silas:doctor` flags it everywhere.

data/docs/inbox-and-api.md

@@ -0,0 +1,91 @@
+# Inbox & API
+
+Everything an operator needs ships in the gem — mount the engine (the
+installer does this) and it's there. No dashboard to build, no second console
+to deploy.
+
+## The inbox — `/silas/inbox`
+
+A session rail grouped **Held / Working / Filed**, web chat (start a session
+or reply from the browser — same durable loop), a live step-trace that
+streams tokens over Turbo as the agent runs, approval and question cards
+hoisted to the top of the session, a full audit trail (every tool call's
+arguments and result, who cleared what and why), cancel, raise-budget, and
+per-session token/cost accounting priced from RubyLLM's model registry.
+
+<img src="https://raw.githubusercontent.com/danielstpaul/silas/main/docs/img/silas-inbox-held.png" width="740"
+     alt="A session held at the signal: the amber approval card for issue_refund hoisted above the trace, with the turn marked HELD and a held-at-the-signal stub in place">
+
+A £64 refund holding at the signal: the card carries the exact arguments, the
+turn costs nothing while it waits, and the trace below keeps a one-line stub
+where the invocation held.
+
+It's **deny-by-default** — invisible until you wire auth:
+
+```ruby
+Silas.configure do |c|
+  # The lambda DENIES by rendering (or head-ing) and PASSES by not rendering.
+  c.inbox_auth = ->(controller) { controller.head :not_found unless controller.current_user&.admin? }
+  # c.inbox_public_read = true   # read-only demo mode; writes stay gated
+  # c.model_prices["your-fine-tune"] = { in: 3000, out: 15_000 }  # price overrides
+end
+```
+
+`config.inbox_actor` names the identity recorded on approvals (defaults to
+`current_user&.email || "inbox"`). Live streaming activates automatically when
+the host app has `turbo-rails` (every default Rails app does); without it the
+trace falls back to a polling refresh. The gem takes no turbo dependency.
+
+## The JSON API — `/silas/api/v1`
+
+The same surface over HTTP, also deny-by-default (`config.api_auth`, same
+contract as the inbox; `config.api_actor` names the API identity):
+
+```sh
+curl -X POST .../silas/api/v1/sessions -d "input=Refund order 42, £12.50"
+curl .../silas/api/v1/sessions/1?trace=1                # turns + steps + tool calls
+curl .../silas/api/v1/sessions/1/approvals              # what's held
+curl -X POST .../silas/api/v1/approvals/7/approve       # the same approve! as the inbox
+curl -X POST .../silas/api/v1/approvals/9/answer -d "answer=Use the June invoice"
+curl -X POST .../silas/api/v1/sessions/1/turns -d "input=Now email them"   # 409 if busy
+curl -X POST .../silas/api/v1/turns/9/cancel
+curl -N .../silas/api/v1/sessions/1/stream              # server-sent events
+```
+
+The stream is SSE at **row granularity** — turn, completed-step, and
+invocation changes, at-least-once with `Last-Event-ID` resume (ids are
+epoch-ms watermarks). `?poll=1` returns the backlog and closes, curl-friendly;
+streams close themselves after `config.api_stream_max_duration` and clients
+reconnect. Per-token streaming is deliberately the browser/Turbo feature:
+deltas live in the worker process, and the gem requires no cross-process bus.
+
+## Streaming — decoration over durable rows
+
+Turns stream everywhere it helps: `silas:chat` prints tokens as they arrive,
+and the inbox renders them live (coalesced to ~10Hz). Deltas are never
+persisted, never fed back to the model, and absent from replays — a replayed
+step renders from its row — so streaming adds zero risk to the durability
+contract. Custom sinks subscribe to the `"delta.silas"` notification
+(`{ session_id:, turn_id:, step_id:, step_index:, text: }`, where `text` is
+the accumulated string so far; filter by ids — notifications are
+process-global).
+
+## Structured answers
+
+Give the turn's final answer a schema in `agent.yml`:
+
+```yaml
+final_answer:
+  type: object
+  properties:
+    verdict: { type: string }
+    amount_pence: { type: integer }
+  required: [verdict]
+```
+
+`Turn#answer_data` returns the parsed Hash (`answer_text` stays for prose
+agents); the API carries it as `answer_data`, and evals assert on it with
+`assert_answer_data(key: :verdict, value: "approve")`. Rendered through
+RubyLLM's `with_schema`, so each provider's native structured-output mode is
+used. The schema is model-visible state — changing it mid-turn fails the turn
+loudly rather than resuming into a different contract.

data/docs/memory.md

@@ -0,0 +1,35 @@
+# Memory
+
+Silas memory is **graph-shaped, not a graph database**: facts stored as
+`subject · attribute · content` triples with provenance and supersession —
+"author:jane · report_format: prefers CSV", and a new value retires the old
+one instead of piling up beside it.
+
+## Approval-gated by default
+
+The first time the model calls `remember`, nothing persists — the memory
+**parks as a card in your inbox** and you approve or decline it
+(`config.memory_approval = :always`, the default). An agent that can silently
+write its own long-term memory is an agent that can silently drift;
+the gate keeps a person in that loop. `:never` auto-approves once you trust
+an agent's judgment, and `config.memory = false` removes the tools entirely.
+
+## Recall
+
+A handful of the most recent relevant memories (default 8,
+`config.memory_injection_limit`) are injected into each turn automatically;
+the `recall` tool digs deeper on demand. Memories are private per agent, or
+`shared: true` for the whole staff.
+
+## What belongs here — and what doesn't
+
+Memory is for the fuzzy residue with no natural home: preferences, standing
+context, things a colleague would jot in a notebook. Your **domain data does
+not belong here** — it belongs in your own tables, which your tools already
+read. If it has a schema, it's a model; if it's a remark, it's a memory.
+
+## The fine print
+
+Memory tools are model-visible state, so toggling `config.memory` (like any
+builtin) changes the definitions digest — settle parked turns before flipping
+it, or they fail loudly on resume ([guarantees](guarantees.md)).

data/docs/sandbox.md

@@ -0,0 +1,44 @@
+# Sandbox
+
+Code execution is **off by default** (`config.sandbox = :none`). Configure a
+sandbox and the `run_code` tool is advertised to the model automatically —
+always `at_most_once!`, because an exec is an external effect.
+
+## Built-in: `:docker`
+
+A hardened container seam — resource-capped, network-off by default, with
+knobs for image, memory, CPUs, pids, and timeout
+([configuration](configuration.md)). It's honest-but-interim: a container is
+weaker than a microVM, and it runs on the same host as your ledger and
+`RAILS_MASTER_KEY`. Treat it as suitable for code **you** wrote, not code the
+model wrote.
+
+## Real isolation: hermetic
+
+For untrusted or model-generated code, the companion gem
+[hermetic](https://github.com/danielstpaul/hermetic) drops straight in:
+
+```ruby
+# Gemfile: gem "hermetic"   (zero runtime deps)
+Silas.configure do |c|
+  c.sandbox = Hermetic.gvisor(image: "python:3.12-slim")
+  # or .docker / .firecracker(kernel:, rootfs:) / .hosted(:e2b, api_key:)
+end
+```
+
+Two properties carry through the seam:
+
+- **The trust axis is visible.** Every hermetic backend exposes `trust`
+  (`:vendor` / `:remote` / `:vm` / `:host`) and `off_host?`, so you can refuse
+  to run untrusted code on the box that holds your secrets — and pair any
+  local backend with `executor:` to push execution to a dedicated sandbox
+  host.
+- **The ledger guard is auto-armed.** Configuring a hermetic backend loads its
+  Silas shim, so a sandbox exec attempted inside a ledger transaction fails
+  loudly (sandbox-backed tools must be `at_most_once!`, never
+  `transactional!`).
+
+## Bring your own
+
+`config.sandbox` accepts any object responding to `#run` — the seam is the
+contract, the backends are interchangeable.

data/docs/tools.md

@@ -0,0 +1,77 @@
+# Tools & approvals
+
+A tool is one file in `app/agent/tools/`. The filename is the tool's identity;
+the keyword signature of `#call` is the schema the model sees. No registry, no
+wiring — add a file, restart, and the model can use it.
+
+```ruby
+# app/agent/tools/issue_refund.rb  ->  the "issue_refund" tool
+class Agent::Tools::IssueRefund < Silas::Tool
+  description "Refund part or all of an order."
+  param :amount_pence, :integer, desc: "Amount in pence (1800 = £18.00)"
+  approval ->(session:, input:) { input[:amount_pence] > 2_500 ? :user_approval : :approved }
+  transactional!
+
+  def call(number:, amount_pence:, reason:)
+    order = Order.find_by!(number: number)
+    refund = order.refunds.create!(amount_pence:, reason:)
+    { refunded_pence: refund.amount_pence, order: order.number }
+  end
+end
+```
+
+The rules:
+
+- **Keywords only** — the keyword signature *is* the schema; `param` refines
+  types and descriptions.
+- **Return a Hash** (anything else is wrapped as `{"value" => ...}`). Raising
+  records a failed invocation the model sees — don't rescue-and-swallow.
+- `session` is available inside `call` (the `Silas::Session` row), so tools
+  can read channel metadata or scope queries.
+
+## Effect modes — decide by where the side effect lives
+
+| The tool… | Declare | What you get |
+|---|---|---|
+| writes this app's database | `transactional!` | effect + ledger commit atomically → **exactly-once**, even through `kill -9` |
+| calls anything external (email, HTTP, Slack…) | `at_most_once!` (the default) | a crash mid-call leaves it **in doubt** → parks for a human verdict, never re-fires blind |
+| only reads, safe to repeat | `idempotent!` | crash replays may re-run it freely |
+
+Never mark an external call `transactional!` — the ledger cannot roll back a
+sent email. Model money as rows in your own database wherever you can; that's
+what upgrades the guarantee to exactly-once
+([guarantees](guarantees.md)).
+
+## Approval — who holds the lever
+
+```ruby
+approval :never    # default — runs without asking
+approval :always   # every call parks for a human
+approval :once     # one approval per identical (tool, arguments) pair per session
+approval ->(session:, input:) { ... }  # decide from the arguments
+```
+
+A lambda returns `:user_approval`, `:approved`, `:not_applicable`, or
+`{denied: "reason"}` (the denial rides back to the model as the tool result).
+Parking costs nothing — the turn **holds at the signal** at zero compute until
+someone clears it from the inbox, Slack, a signed email link, or the JSON API,
+all calling the same `approve!`/`decline!`. Parks expire after
+`config.approval_ttl` (default 7 days).
+
+The built-in **`ask_question`** tool is the reverse direction: the agent parks
+to ask the operator something — information, not permission — and their typed
+answer resumes the turn as the tool result.
+
+## Skills — playbooks, loaded on demand
+
+A skill is a markdown file in `app/agent/skills/` with a `description:`
+frontmatter line. The description is always visible to the model as a routing
+hint; the body loads only when the model asks for it (`load_skill`), keeping
+the always-on prompt small. Put procedures in skills; put identity in
+`instructions.md`.
+
+## Remote tools
+
+`app/agent/connections/*.yml` plugs a remote MCP server's tools in under the
+same ledger, namespaced `<connection>__<tool>` — see
+[connections](connections.md).