silas 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +211 -0
- data/DEPLOY.md +111 -0
- data/README.md +87 -250
- data/app/controllers/silas/api/v1/approvals_controller.rb +10 -0
- data/app/controllers/silas/inbox/invocations_controller.rb +8 -0
- data/app/helpers/silas/inbox/trace_helper.rb +20 -5
- data/app/jobs/silas/channel_delivery_job.rb +15 -0
- data/app/models/concerns/silas/inbox/broadcastable.rb +12 -0
- data/app/models/silas/compaction.rb +32 -0
- data/app/models/silas/tool_invocation.rb +29 -3
- data/app/views/layouts/silas/inbox.html.erb +89 -30
- data/app/views/silas/inbox/invocations/_approval_card.html.erb +36 -13
- data/app/views/silas/inbox/invocations/_invocation.html.erb +17 -5
- data/app/views/silas/inbox/sessions/_row.html.erb +14 -0
- data/app/views/silas/inbox/sessions/index.html.erb +16 -15
- data/app/views/silas/inbox/sessions/show.html.erb +9 -0
- data/config/routes.rb +2 -0
- data/db/migrate/20260725000002_create_silas_compactions.rb +26 -0
- data/docs/agents.md +81 -0
- data/docs/budgets.md +67 -0
- data/docs/cancellation.md +41 -0
- data/docs/channels.md +290 -0
- data/docs/configuration.md +106 -0
- data/docs/connections.md +55 -0
- data/docs/conventions.md +161 -0
- data/docs/evals.md +95 -0
- data/docs/guarantees.md +76 -0
- data/docs/inbox-and-api.md +84 -0
- data/docs/memory.md +35 -0
- data/docs/sandbox.md +44 -0
- data/docs/tools.md +77 -0
- data/docs/tutorial.md +268 -0
- data/docs/vs-eve.md +93 -0
- data/docs/why-silas.md +87 -0
- data/lib/generators/silas/channel/channel_generator.rb +72 -0
- data/lib/generators/silas/channel/templates/channel.rb.tt +48 -0
- data/lib/generators/silas/channel/templates/controller.rb.tt +66 -0
- data/lib/generators/silas/install/install_generator.rb +11 -1
- data/lib/generators/silas/install/templates/claude_skill.md +136 -0
- data/lib/generators/silas/install/templates/ruby_llm.rb +4 -1
- data/lib/silas/adapters/ruby_llm.rb +102 -46
- data/lib/silas/channel.rb +35 -0
- data/lib/silas/compactor.rb +178 -0
- data/lib/silas/configuration.rb +16 -0
- data/lib/silas/eval/dsl.rb +7 -2
- data/lib/silas/instrumentation.rb +7 -3
- data/lib/silas/ledger.rb +2 -2
- data/lib/silas/log_subscriber.rb +5 -0
- data/lib/silas/message_builder.rb +19 -0
- data/lib/silas/registry.rb +5 -2
- data/lib/silas/schedule.rb +44 -15
- data/lib/silas/slack.rb +8 -5
- data/lib/silas/step_runner.rb +5 -0
- data/lib/silas/tools/ask_question.rb +26 -0
- data/lib/silas/version.rb +1 -1
- data/lib/silas/webhook.rb +47 -0
- data/lib/silas.rb +3 -0
- metadata +30 -1
data/docs/evals.md
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
# Evals
|
|
2
|
+
|
|
3
|
+
Agent evals answer the question tests can't: *given these model decisions, does
|
|
4
|
+
the durable machinery do the right thing?* A scenario scripts the **model's
|
|
5
|
+
decisions** step by step; everything else is real — the real registry resolves
|
|
6
|
+
your real tools, the real Ledger enforces effect modes and approvals, and the
|
|
7
|
+
assertions read the genuine durable transcript, not a mock.
|
|
8
|
+
|
|
9
|
+
They run keyless and deterministically, which makes them a **deploy gate**:
|
|
10
|
+
|
|
11
|
+
```sh
|
|
12
|
+
bin/rails silas:eval # loads <eval_dir>/**/*_eval.rb; exits 1 on failure
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The installer wires this into `bin/ci` (or tells you to add it to yours), and
|
|
16
|
+
the application template generates a working example in
|
|
17
|
+
`test/agent_evals/refund_desk_eval.rb`.
|
|
18
|
+
|
|
19
|
+
## A scenario
|
|
20
|
+
|
|
21
|
+
```ruby
|
|
22
|
+
Silas::Eval.scenario "over the gate: a £64 refund holds at the signal" do
|
|
23
|
+
input "The walnut monitor stand (order R-1002) arrived cracked."
|
|
24
|
+
|
|
25
|
+
on_step 0, call: { name: "lookup_order", arguments: { number: "R-1002" } }
|
|
26
|
+
on_step 1, call: { name: "issue_refund",
|
|
27
|
+
arguments: { number: "R-1002", amount_pence: 6400, reason: "arrived cracked" } }
|
|
28
|
+
|
|
29
|
+
expect do
|
|
30
|
+
assert_parked tool: "issue_refund" # the turn holds at zero compute
|
|
31
|
+
assert_no_tool_called "issue_refund" # and the refund row does NOT exist
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
`on_step index, ...` is the script: what the "model" returns on that model
|
|
37
|
+
call. `text:` is an assistant text block, `call:`/`calls:` are tool calls. A
|
|
38
|
+
step index with no entry ends the scripted conversation.
|
|
39
|
+
|
|
40
|
+
## The DSL
|
|
41
|
+
|
|
42
|
+
| Call | Meaning |
|
|
43
|
+
|---|---|
|
|
44
|
+
| `input "…"` | The turn's input. Required. |
|
|
45
|
+
| `on_step i, text:, call:, calls:, data:` | The scripted model output for step `i`. `data:` scripts a **structured answer** (the `final_answer` schema case) — it becomes the same block the real adapter persists, so `assert_answer_data` reads it back. |
|
|
46
|
+
| `approve tool: "name"` | When the turn parks on this tool, approve it (recorded as approved by `"eval"` — the same `approve!` the inbox uses) and resume. This is how you assert exactly-once **across** a hold. |
|
|
47
|
+
| `stub_tool "name", effect_mode:, approval: { \|**args\| … }` | Replace a real tool with a stub for a side-effect-free scenario. Unstubbed tools stay real. |
|
|
48
|
+
| `mode :real` | Drive a real model instead of the script. **Skipped automatically when `ANTHROPIC_API_KEY` is unset**, so the gate stays green offline. |
|
|
49
|
+
| `max_steps n` | Override the per-turn step cap for this scenario. |
|
|
50
|
+
| `metadata h` / tags on `scenario "…", tags: [...]` | Stamped on the session / used for filtering. |
|
|
51
|
+
| `expect { … }` | The assertion block. Required. Runs after the turn reaches its resting state. |
|
|
52
|
+
|
|
53
|
+
## Assertions
|
|
54
|
+
|
|
55
|
+
All assertions read only the durable rows, and they **collect** failures — one
|
|
56
|
+
run reports every miss, not just the first.
|
|
57
|
+
|
|
58
|
+
| Assertion | Checks |
|
|
59
|
+
|---|---|
|
|
60
|
+
| `assert_tool_called "name", times: 1` | The tool actually executed (`times:` for exactly-N — the exactly-once assertion). |
|
|
61
|
+
| `assert_no_tool_called "name"` | No execution happened (e.g. because the turn parked first). |
|
|
62
|
+
| `assert_tool_arg "name", :key, value` (or a block predicate) | An argument the model passed. |
|
|
63
|
+
| `assert_parked tool: "name"` | The turn is waiting, held on that tool's approval (or in-doubt). |
|
|
64
|
+
| `assert_approved tool: "name"` | The invocation's approval state is `approved`. |
|
|
65
|
+
| `assert_turn_completed` / `assert_turn_failed(reason: "…")` | Terminal state. |
|
|
66
|
+
| `assert_final_matches(/…/)` | The final answer text. |
|
|
67
|
+
| `assert_answer_data(key: :verdict, value: "approve")` | The structured `final_answer` payload (whole-Hash and predicate forms too). |
|
|
68
|
+
| `assert_no_hallucinated_price(allowed: [])` | Every money amount in the final answer traces to a number the agent actually saw (tool results or the input), allowing pence↔pounds scaling. |
|
|
69
|
+
| `assert_rubric "…" ` | LLM-graded check against `config.eval_grader` — **skips offline** rather than failing the gate (set `SILAS_EVAL_STRICT=1` to make skips fail). |
|
|
70
|
+
|
|
71
|
+
## How a scenario runs (so you can trust it)
|
|
72
|
+
|
|
73
|
+
The driver starts a real session (`Silas.agent.start`), runs the real
|
|
74
|
+
`AgentLoopJob` inline on the `:test` queue adapter, and — for each `approve
|
|
75
|
+
tool:` — performs the same `approve!` an operator would, then resumes the
|
|
76
|
+
loop. Scripted mode swaps only the inference adapter; `mode :real` uses your
|
|
77
|
+
configured one.
|
|
78
|
+
|
|
79
|
+
Two consequences worth knowing:
|
|
80
|
+
|
|
81
|
+
- **Rows persist.** Scenarios commit ordinary durable rows to the database of
|
|
82
|
+
the environment you run them in (they need real seeds for tools that read
|
|
83
|
+
your tables — the template's CI runs `db:seed silas:eval`). Run against a
|
|
84
|
+
scratch database if residue bothers you.
|
|
85
|
+
- **Config is restored per run**, but scenarios execute sequentially in one
|
|
86
|
+
process — don't have two scenarios fight over global state you set yourself.
|
|
87
|
+
|
|
88
|
+
## Configuration
|
|
89
|
+
|
|
90
|
+
```ruby
|
|
91
|
+
Silas.configure do |c|
|
|
92
|
+
c.eval_dir = "test/agent_evals" # where *_eval.rb files live
|
|
93
|
+
# c.eval_grader = ->(prompt) { … } # custom LLM grader for assert_rubric
|
|
94
|
+
end
|
|
95
|
+
```
|
data/docs/guarantees.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# Guarantees
|
|
2
|
+
|
|
3
|
+
Durability in Silas is a contract, not a slide. Everything on this page is
|
|
4
|
+
verified by the in-repo chaos harness (`chaos_host/bin/chaos`), which
|
|
5
|
+
`kill -9`s live agents hundreds of times per release — the current gate is
|
|
6
|
+
100/100 completions per mode across a 295-run matrix, zero duplicate effects,
|
|
7
|
+
byte-identical replay, on SQLite and Postgres. Results live in
|
|
8
|
+
`chaos_host/results/` and every run is reproducible.
|
|
9
|
+
|
|
10
|
+
## Turns survive hard process death
|
|
11
|
+
|
|
12
|
+
Worker `kill -9`, whole-tree `kill -9`, SIGTERM deploys — a turn resumes from
|
|
13
|
+
its last completed step. Each step checkpoints through Active Job
|
|
14
|
+
Continuations; the resume replays completed work **from rows**, never by
|
|
15
|
+
re-calling the model or re-running settled tools.
|
|
16
|
+
|
|
17
|
+
## `transactional!` tools execute exactly once
|
|
18
|
+
|
|
19
|
+
The tool's database writes and the ledger's record of "this ran" commit or
|
|
20
|
+
roll back **in one transaction**:
|
|
21
|
+
|
|
22
|
+
- **Crash before commit** → both roll back; resume runs the tool from a clean
|
|
23
|
+
slate. No orphan effect.
|
|
24
|
+
- **Crash after commit** → the ledger row says `completed`; resume skips the
|
|
25
|
+
step. No second effect.
|
|
26
|
+
|
|
27
|
+
No idempotency key is required, because the dedup lives in your database — the
|
|
28
|
+
same transaction as the effect. This is the guarantee that requires the ledger
|
|
29
|
+
to live *inside* your app: a runtime outside your database can retry and
|
|
30
|
+
reconcile, but it can't make its record and your effect one atomic commit.
|
|
31
|
+
|
|
32
|
+
## Ambiguity parks instead of guessing
|
|
33
|
+
|
|
34
|
+
The default effect mode is `at_most_once!`: when a crash makes an execution
|
|
35
|
+
ambiguous ("did the email send?"), the invocation parks **in doubt** for a
|
|
36
|
+
human verdict rather than re-firing blind. `idempotent!` is the explicit
|
|
37
|
+
opt-in to automatic re-runs. Never double-pay; sometimes ask.
|
|
38
|
+
|
|
39
|
+
## Approvals park at zero compute
|
|
40
|
+
|
|
41
|
+
A turn awaiting a human exits the worker entirely — no held thread, no polling
|
|
42
|
+
loop. Approving enqueues a fresh job that replays from rows. Parks expire
|
|
43
|
+
(default 7 days, `config.approval_ttl`) rather than ghosting forever.
|
|
44
|
+
|
|
45
|
+
## Errors can't strand a turn
|
|
46
|
+
|
|
47
|
+
Transient model errors (rate limits, overloads, timeouts) back off and retry
|
|
48
|
+
from the checkpoint. Exhausted retries and permanent rejections (bad key, bad
|
|
49
|
+
request) expire pending approvals and fail the turn loudly. And a turn can
|
|
50
|
+
never sit in `running` forever: the recurring `DeadJobRescuerJob` retries jobs
|
|
51
|
+
failed by dead-worker reaping and fails turns stranded by a loop job that died
|
|
52
|
+
outside the retry list. The rescuer is part of the contract — keep it in
|
|
53
|
+
`config/recurring.yml`, and monitor worker liveness (the rescuer can requeue
|
|
54
|
+
work; it cannot conjure a consumer).
|
|
55
|
+
|
|
56
|
+
## Deploys can't corrupt a run
|
|
57
|
+
|
|
58
|
+
Instructions are snapshotted per turn. Tools, skills, connections, and the
|
|
59
|
+
final-answer schema are model-visible state, captured in a definitions digest —
|
|
60
|
+
a deploy that changes them while a turn is parked fails that turn loudly on
|
|
61
|
+
resume (`NondeterminismError`) instead of quietly resuming into a different
|
|
62
|
+
agent. Settle parked turns before shipping agent changes.
|
|
63
|
+
|
|
64
|
+
## Compaction can't corrupt a replay
|
|
65
|
+
|
|
66
|
+
Long conversations summarise past `config.compact_at` — but a summary is a
|
|
67
|
+
**persisted, exactly-once effect** (claimed compare-and-swap, written once),
|
|
68
|
+
never a rebuild-time computation. The message array a resumed turn sees is
|
|
69
|
+
byte-identical to the one the crashed turn saw.
|
|
70
|
+
|
|
71
|
+
## The one rule you owe the contract
|
|
72
|
+
|
|
73
|
+
Run agents on Solid Queue (or `:inline` for scripts) — never ActiveJob's
|
|
74
|
+
in-process `:async` adapter, which runs a re-enqueued continuation
|
|
75
|
+
concurrently with the original and double-executes steps. Silas raises on it
|
|
76
|
+
in production; `bin/rails silas:doctor` flags it everywhere.
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Inbox & API
|
|
2
|
+
|
|
3
|
+
Everything an operator needs ships in the gem — mount the engine (the
|
|
4
|
+
installer does this) and it's there. No dashboard to build, no second console
|
|
5
|
+
to deploy.
|
|
6
|
+
|
|
7
|
+
## The inbox — `/silas/inbox`
|
|
8
|
+
|
|
9
|
+
A session rail grouped **Held / Working / Filed**, web chat (start a session
|
|
10
|
+
or reply from the browser — same durable loop), a live step-trace that
|
|
11
|
+
streams tokens over Turbo as the agent runs, approval and question cards
|
|
12
|
+
hoisted to the top of the session, a full audit trail (every tool call's
|
|
13
|
+
arguments and result, who cleared what and why), cancel, raise-budget, and
|
|
14
|
+
per-session token/cost accounting priced from RubyLLM's model registry.
|
|
15
|
+
|
|
16
|
+
It's **deny-by-default** — invisible until you wire auth:
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
Silas.configure do |c|
|
|
20
|
+
# The lambda DENIES by rendering (or head-ing) and PASSES by not rendering.
|
|
21
|
+
c.inbox_auth = ->(controller) { controller.head :not_found unless controller.current_user&.admin? }
|
|
22
|
+
# c.inbox_public_read = true # read-only demo mode; writes stay gated
|
|
23
|
+
# c.model_prices["your-fine-tune"] = { in: 3000, out: 15_000 } # price overrides
|
|
24
|
+
end
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
`config.inbox_actor` names the identity recorded on approvals (defaults to
|
|
28
|
+
`current_user&.email || "inbox"`). Live streaming activates automatically when
|
|
29
|
+
the host app has `turbo-rails` (every default Rails app does); without it the
|
|
30
|
+
trace falls back to a polling refresh. The gem takes no turbo dependency.
|
|
31
|
+
|
|
32
|
+
## The JSON API — `/silas/api/v1`
|
|
33
|
+
|
|
34
|
+
The same surface over HTTP, also deny-by-default (`config.api_auth`, same
|
|
35
|
+
contract as the inbox; `config.api_actor` names the API identity):
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
curl -X POST .../silas/api/v1/sessions -d "input=Refund order 42, £12.50"
|
|
39
|
+
curl .../silas/api/v1/sessions/1?trace=1 # turns + steps + tool calls
|
|
40
|
+
curl .../silas/api/v1/sessions/1/approvals # what's held
|
|
41
|
+
curl -X POST .../silas/api/v1/approvals/7/approve # the same approve! as the inbox
|
|
42
|
+
curl -X POST .../silas/api/v1/approvals/9/answer -d "answer=Use the June invoice"
|
|
43
|
+
curl -X POST .../silas/api/v1/sessions/1/turns -d "input=Now email them" # 409 if busy
|
|
44
|
+
curl -X POST .../silas/api/v1/turns/9/cancel
|
|
45
|
+
curl -N .../silas/api/v1/sessions/1/stream # server-sent events
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The stream is SSE at **row granularity** — turn, completed-step, and
|
|
49
|
+
invocation changes, at-least-once with `Last-Event-ID` resume (ids are
|
|
50
|
+
epoch-ms watermarks). `?poll=1` returns the backlog and closes, curl-friendly;
|
|
51
|
+
streams close themselves after `config.api_stream_max_duration` and clients
|
|
52
|
+
reconnect. Per-token streaming is deliberately the browser/Turbo feature:
|
|
53
|
+
deltas live in the worker process, and the gem requires no cross-process bus.
|
|
54
|
+
|
|
55
|
+
## Streaming — decoration over durable rows
|
|
56
|
+
|
|
57
|
+
Turns stream everywhere it helps: `silas:chat` prints tokens as they arrive,
|
|
58
|
+
and the inbox renders them live (coalesced to ~10Hz). Deltas are never
|
|
59
|
+
persisted, never fed back to the model, and absent from replays — a replayed
|
|
60
|
+
step renders from its row — so streaming adds zero risk to the durability
|
|
61
|
+
contract. Custom sinks subscribe to the `"delta.silas"` notification
|
|
62
|
+
(`{ session_id:, turn_id:, step_id:, step_index:, text: }`, where `text` is
|
|
63
|
+
the accumulated string so far; filter by ids — notifications are
|
|
64
|
+
process-global).
|
|
65
|
+
|
|
66
|
+
## Structured answers
|
|
67
|
+
|
|
68
|
+
Give the turn's final answer a schema in `agent.yml`:
|
|
69
|
+
|
|
70
|
+
```yaml
|
|
71
|
+
final_answer:
|
|
72
|
+
type: object
|
|
73
|
+
properties:
|
|
74
|
+
verdict: { type: string }
|
|
75
|
+
amount_pence: { type: integer }
|
|
76
|
+
required: [verdict]
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
`Turn#answer_data` returns the parsed Hash (`answer_text` stays for prose
|
|
80
|
+
agents); the API carries it as `answer_data`, and evals assert on it with
|
|
81
|
+
`assert_answer_data(key: :verdict, value: "approve")`. Rendered through
|
|
82
|
+
RubyLLM's `with_schema`, so each provider's native structured-output mode is
|
|
83
|
+
used. The schema is model-visible state — changing it mid-turn fails the turn
|
|
84
|
+
loudly rather than resuming into a different contract.
|
data/docs/memory.md
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Memory
|
|
2
|
+
|
|
3
|
+
Silas memory is **graph-shaped, not a graph database**: facts stored as
|
|
4
|
+
`subject · attribute · content` triples with provenance and supersession —
|
|
5
|
+
"author:jane · report_format: prefers CSV", and a new value retires the old
|
|
6
|
+
one instead of piling up beside it.
|
|
7
|
+
|
|
8
|
+
## Approval-gated by default
|
|
9
|
+
|
|
10
|
+
The first time the model calls `remember`, nothing persists — the memory
|
|
11
|
+
**parks as a card in your inbox** and you approve or decline it
|
|
12
|
+
(`config.memory_approval = :always`, the default). An agent that can silently
|
|
13
|
+
write its own long-term memory is an agent that can silently drift;
|
|
14
|
+
the gate keeps a person in that loop. `:never` auto-approves once you trust
|
|
15
|
+
an agent's judgment, and `config.memory = false` removes the tools entirely.
|
|
16
|
+
|
|
17
|
+
## Recall
|
|
18
|
+
|
|
19
|
+
A handful of the most recent relevant memories (default 8,
|
|
20
|
+
`config.memory_injection_limit`) are injected into each turn automatically;
|
|
21
|
+
the `recall` tool digs deeper on demand. Memories are private per agent, or
|
|
22
|
+
`shared: true` for the whole staff.
|
|
23
|
+
|
|
24
|
+
## What belongs here — and what doesn't
|
|
25
|
+
|
|
26
|
+
Memory is for the fuzzy residue with no natural home: preferences, standing
|
|
27
|
+
context, things a colleague would jot in a notebook. Your **domain data does
|
|
28
|
+
not belong here** — it belongs in your own tables, which your tools already
|
|
29
|
+
read. If it has a schema, it's a model; if it's a remark, it's a memory.
|
|
30
|
+
|
|
31
|
+
## The fine print
|
|
32
|
+
|
|
33
|
+
Memory tools are model-visible state, so toggling `config.memory` (like any
|
|
34
|
+
builtin) changes the definitions digest — settle parked turns before flipping
|
|
35
|
+
it, or they fail loudly on resume ([guarantees](guarantees.md)).
|
data/docs/sandbox.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Sandbox
|
|
2
|
+
|
|
3
|
+
Code execution is **off by default** (`config.sandbox = :none`). Configure a
|
|
4
|
+
sandbox and the `run_code` tool is advertised to the model automatically —
|
|
5
|
+
always `at_most_once!`, because an exec is an external effect.
|
|
6
|
+
|
|
7
|
+
## Built-in: `:docker`
|
|
8
|
+
|
|
9
|
+
A hardened container seam — resource-capped, network-off by default, with
|
|
10
|
+
knobs for image, memory, CPUs, pids, and timeout
|
|
11
|
+
([configuration](configuration.md)). It's honest-but-interim: a container is
|
|
12
|
+
weaker than a microVM, and it runs on the same host as your ledger and
|
|
13
|
+
`RAILS_MASTER_KEY`. Treat it as suitable for code **you** wrote, not code the
|
|
14
|
+
model wrote.
|
|
15
|
+
|
|
16
|
+
## Real isolation: hermetic
|
|
17
|
+
|
|
18
|
+
For untrusted or model-generated code, the companion gem
|
|
19
|
+
[hermetic](https://github.com/danielstpaul/hermetic) drops straight in:
|
|
20
|
+
|
|
21
|
+
```ruby
|
|
22
|
+
# Gemfile: gem "hermetic" (zero runtime deps)
|
|
23
|
+
Silas.configure do |c|
|
|
24
|
+
c.sandbox = Hermetic.gvisor(image: "python:3.12-slim")
|
|
25
|
+
# or .docker / .firecracker(kernel:, rootfs:) / .hosted(:e2b, api_key:)
|
|
26
|
+
end
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Two properties carry through the seam:
|
|
30
|
+
|
|
31
|
+
- **The trust axis is visible.** Every hermetic backend exposes `trust`
|
|
32
|
+
(`:vendor` / `:remote` / `:vm` / `:host`) and `off_host?`, so you can refuse
|
|
33
|
+
to run untrusted code on the box that holds your secrets — and pair any
|
|
34
|
+
local backend with `executor:` to push execution to a dedicated sandbox
|
|
35
|
+
host.
|
|
36
|
+
- **The ledger guard is auto-armed.** Configuring a hermetic backend loads its
|
|
37
|
+
Silas shim, so a sandbox exec attempted inside a ledger transaction fails
|
|
38
|
+
loudly (sandbox-backed tools must be `at_most_once!`, never
|
|
39
|
+
`transactional!`).
|
|
40
|
+
|
|
41
|
+
## Bring your own
|
|
42
|
+
|
|
43
|
+
`config.sandbox` accepts any object responding to `#run` — the seam is the
|
|
44
|
+
contract, the backends are interchangeable.
|
data/docs/tools.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Tools & approvals
|
|
2
|
+
|
|
3
|
+
A tool is one file in `app/agent/tools/`. The filename is the tool's identity;
|
|
4
|
+
the keyword signature of `#call` is the schema the model sees. No registry, no
|
|
5
|
+
wiring — add a file, restart, and the model can use it.
|
|
6
|
+
|
|
7
|
+
```ruby
|
|
8
|
+
# app/agent/tools/issue_refund.rb -> the "issue_refund" tool
|
|
9
|
+
class Agent::Tools::IssueRefund < Silas::Tool
|
|
10
|
+
description "Refund part or all of an order."
|
|
11
|
+
param :amount_pence, :integer, desc: "Amount in pence (1800 = £18.00)"
|
|
12
|
+
approval ->(session:, input:) { input[:amount_pence] > 2_500 ? :user_approval : :approved }
|
|
13
|
+
transactional!
|
|
14
|
+
|
|
15
|
+
def call(number:, amount_pence:, reason:)
|
|
16
|
+
order = Order.find_by!(number: number)
|
|
17
|
+
refund = order.refunds.create!(amount_pence:, reason:)
|
|
18
|
+
{ refunded_pence: refund.amount_pence, order: order.number }
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The rules:
|
|
24
|
+
|
|
25
|
+
- **Keywords only** — the keyword signature *is* the schema; `param` refines
|
|
26
|
+
types and descriptions.
|
|
27
|
+
- **Return a Hash** (anything else is wrapped as `{"value" => ...}`). Raising
|
|
28
|
+
records a failed invocation the model sees — don't rescue-and-swallow.
|
|
29
|
+
- `session` is available inside `call` (the `Silas::Session` row), so tools
|
|
30
|
+
can read channel metadata or scope queries.
|
|
31
|
+
|
|
32
|
+
## Effect modes — decide by where the side effect lives
|
|
33
|
+
|
|
34
|
+
| The tool… | Declare | What you get |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| writes this app's database | `transactional!` | effect + ledger commit atomically → **exactly-once**, even through `kill -9` |
|
|
37
|
+
| calls anything external (email, HTTP, Slack…) | `at_most_once!` (the default) | a crash mid-call leaves it **in doubt** → parks for a human verdict, never re-fires blind |
|
|
38
|
+
| only reads, safe to repeat | `idempotent!` | crash replays may re-run it freely |
|
|
39
|
+
|
|
40
|
+
Never mark an external call `transactional!` — the ledger cannot roll back a
|
|
41
|
+
sent email. Model money as rows in your own database wherever you can; that's
|
|
42
|
+
what upgrades the guarantee to exactly-once
|
|
43
|
+
([guarantees](guarantees.md)).
|
|
44
|
+
|
|
45
|
+
## Approval — who holds the lever
|
|
46
|
+
|
|
47
|
+
```ruby
|
|
48
|
+
approval :never # default — runs without asking
|
|
49
|
+
approval :always # every call parks for a human
|
|
50
|
+
approval :once # one approval per identical (tool, arguments) pair per session
|
|
51
|
+
approval ->(session:, input:) { ... } # decide from the arguments
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
A lambda returns `:user_approval`, `:approved`, `:not_applicable`, or
|
|
55
|
+
`{denied: "reason"}` (the denial rides back to the model as the tool result).
|
|
56
|
+
Parking costs nothing — the turn **holds at the signal** at zero compute until
|
|
57
|
+
someone clears it from the inbox, Slack, a signed email link, or the JSON API,
|
|
58
|
+
all calling the same `approve!`/`decline!`. Parks expire after
|
|
59
|
+
`config.approval_ttl` (default 7 days).
|
|
60
|
+
|
|
61
|
+
The built-in **`ask_question`** tool is the reverse direction: the agent parks
|
|
62
|
+
to ask the operator something — information, not permission — and their typed
|
|
63
|
+
answer resumes the turn as the tool result.
|
|
64
|
+
|
|
65
|
+
## Skills — playbooks, loaded on demand
|
|
66
|
+
|
|
67
|
+
A skill is a markdown file in `app/agent/skills/` with a `description:`
|
|
68
|
+
frontmatter line. The description is always visible to the model as a routing
|
|
69
|
+
hint; the body loads only when the model asks for it (`load_skill`), keeping
|
|
70
|
+
the always-on prompt small. Put procedures in skills; put identity in
|
|
71
|
+
`instructions.md`.
|
|
72
|
+
|
|
73
|
+
## Remote tools
|
|
74
|
+
|
|
75
|
+
`app/agent/connections/*.yml` plugs a remote MCP server's tools in under the
|
|
76
|
+
same ledger, namespaced `<connection>__<tool>` — see
|
|
77
|
+
[connections](connections.md).
|
data/docs/tutorial.md
ADDED
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
# Tutorial: build the desk
|
|
2
|
+
|
|
3
|
+
One running app, eleven short chapters, one new primitive each. You start with
|
|
4
|
+
a working refund-desk agent and finish with a scheduled, Slack-connected,
|
|
5
|
+
memory-having, staff-employing one — with evals gating every change and every
|
|
6
|
+
consequential action holding at the signal until a person clears it.
|
|
7
|
+
|
|
8
|
+
No API key is needed until chapter 3 (and even then it's optional — everything
|
|
9
|
+
except "talk to a real model" works keyless).
|
|
10
|
+
|
|
11
|
+
## 1 · Open the desk
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
rails new desk -m https://raw.githubusercontent.com/danielstpaul/silas/main/templates/desk.rb
|
|
15
|
+
cd desk && bin/dev
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Open <http://localhost:3000> — the **signal board** — then *operator inbox →*
|
|
19
|
+
and start a session:
|
|
20
|
+
|
|
21
|
+
> The walnut monitor stand (order R-1002) arrived cracked.
|
|
22
|
+
|
|
23
|
+
Watch the trace: `lookup_order` clears, then `issue_refund` **holds** — an
|
|
24
|
+
amber card at the top of the session, and the turn costs nothing while it
|
|
25
|
+
waits. Approve it. The turn resumes exactly where it stopped, notifies the
|
|
26
|
+
customer, and answers. Now check the till:
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
bin/rails runner 'puts Refund.count' # => 1
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Try the £18 story too (`order R-1001`) — under the gate, it never holds.
|
|
33
|
+
|
|
34
|
+
What you just saw is the whole thesis: a **turn** (one request to its answer)
|
|
35
|
+
made of **steps** (model calls) whose tool effects go through a **ledger**;
|
|
36
|
+
anything gated **parks at zero compute** until a human verdict; and a crash
|
|
37
|
+
anywhere in that story resumes from the last completed step. Kill `bin/dev`
|
|
38
|
+
mid-run and restart it if you want to test that claim right now.
|
|
39
|
+
|
|
40
|
+
## 2 · Read the agent (it's a directory)
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
app/agent/
|
|
44
|
+
instructions.md # the persona — plain markdown, ERB allowed
|
|
45
|
+
agent.yml # model + per-turn limits; data only
|
|
46
|
+
tools/
|
|
47
|
+
lookup_order.rb # idempotent! — read-only, replays freely
|
|
48
|
+
issue_refund.rb # transactional! — DB effect: exactly-once, gated over £25
|
|
49
|
+
notify_customer.rb # at_most_once! — external effect: parks IN DOUBT on a crash
|
|
50
|
+
skills/ schedules/ channels/
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Open the three tools. Each declares an **effect mode**, and the mode is the
|
|
54
|
+
entire durability decision:
|
|
55
|
+
|
|
56
|
+
| The tool… | Declare | Because |
|
|
57
|
+
|---|---|---|
|
|
58
|
+
| writes this app's database | `transactional!` | effect + ledger commit atomically → **exactly-once** |
|
|
59
|
+
| calls anything external | `at_most_once!` (default) | an ambiguous crash **parks for a human**, never re-fires blind |
|
|
60
|
+
| only reads | `idempotent!` | replays may re-run it freely |
|
|
61
|
+
|
|
62
|
+
Never mark an external call `transactional!` — the ledger cannot roll back a
|
|
63
|
+
sent email. And note `issue_refund`'s approval lambda: policy lives **on the
|
|
64
|
+
tool**, next to the code it gates.
|
|
65
|
+
|
|
66
|
+
## 3 · Change it: your first tool
|
|
67
|
+
|
|
68
|
+
The desk can't answer "what did Ada buy this year?" Give it history. Create
|
|
69
|
+
`app/agent/tools/order_history.rb`:
|
|
70
|
+
|
|
71
|
+
```ruby
|
|
72
|
+
class Agent::Tools::OrderHistory < Silas::Tool
|
|
73
|
+
description "List a customer's orders by email, newest first."
|
|
74
|
+
idempotent!
|
|
75
|
+
|
|
76
|
+
def call(email:)
|
|
77
|
+
orders = Order.where(email: email.to_s.strip.downcase).order(created_at: :desc)
|
|
78
|
+
return { error: "no orders for #{email}" } if orders.none?
|
|
79
|
+
|
|
80
|
+
{ orders: orders.map { |o| { number: o.number, item: o.item,
|
|
81
|
+
amount_pence: o.amount_pence, status: o.status } } }
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The keyword signature of `#call` **is** the schema the model sees; the
|
|
87
|
+
filename is the tool's name. Restart `bin/dev` (files register at boot), then:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
bin/rails silas:doctor # "tools — 4 tool(s) validate"
|
|
91
|
+
bin/rails silas:chat # you> what has ada@example.com ordered?
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
(Keyless, the scripted stand-in won't improvise about your new tool — export
|
|
95
|
+
`ANTHROPIC_API_KEY` and restart to watch a real model use it. Everything else
|
|
96
|
+
in this tutorial stays keyless-friendly.)
|
|
97
|
+
|
|
98
|
+
## 4 · Prove it: evals as the deploy gate
|
|
99
|
+
|
|
100
|
+
Open `test/agent_evals/refund_desk_eval.rb` — three scenarios already assert
|
|
101
|
+
the desk's contract, including *the refund holds* and *clearing it executes
|
|
102
|
+
exactly once*. Add one for your tool:
|
|
103
|
+
|
|
104
|
+
```ruby
|
|
105
|
+
Silas::Eval.scenario "order history is grounded in rows" do
|
|
106
|
+
input "What has ada@example.com ordered?"
|
|
107
|
+
|
|
108
|
+
on_step 0, call: { name: "order_history", arguments: { email: "ada@example.com" } }
|
|
109
|
+
on_step 1, text: "Ada has two orders: the field notebook (£18.00) and the walnut monitor stand (£64.00)."
|
|
110
|
+
|
|
111
|
+
expect do
|
|
112
|
+
assert_tool_called "order_history", times: 1
|
|
113
|
+
assert_turn_completed
|
|
114
|
+
assert_no_hallucinated_price # every £ in the answer must trace to data the agent saw
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
bin/rails silas:eval # 4 scenarios, 0 failing
|
|
121
|
+
bin/ci # tests + evals — the deploy gate
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
You script the **model's decisions**; the **real ledger** runs your real
|
|
125
|
+
tools. That's why `assert_parked` and `times: 1` are trustworthy — they read
|
|
126
|
+
durable rows, not mocks. Details: [evals.md](evals.md).
|
|
127
|
+
|
|
128
|
+
## 5 · Give it a clock: schedules
|
|
129
|
+
|
|
130
|
+
A schedule is a markdown file whose body becomes the turn input. Create
|
|
131
|
+
`app/agent/schedules/daily_digest.md`:
|
|
132
|
+
|
|
133
|
+
```markdown
|
|
134
|
+
---
|
|
135
|
+
cron: "0 9 * * *"
|
|
136
|
+
---
|
|
137
|
+
Summarize yesterday's refunds: count, total pence, and anything still held at
|
|
138
|
+
the signal. Keep it to five lines.
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
bin/rails silas:schedules
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
That **compiles** schedules into `config/recurring.yml` — cron that fires real
|
|
146
|
+
work stays a reviewable git diff, and each tick is a normal durable turn (it
|
|
147
|
+
can hold for approval like any other). Prefer code? Drop a `.rb` subclassing
|
|
148
|
+
`Silas::Schedule::Handler` in the same directory.
|
|
149
|
+
|
|
150
|
+
## 6 · Give it a doorway: Slack
|
|
151
|
+
|
|
152
|
+
The installer already scaffolded `app/agent/channels/slack.rb`. Wire
|
|
153
|
+
credentials and it's live:
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
bin/rails credentials:edit # silas: { slack: { signing_secret: ..., bot_token: ... } }
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Point your Slack app's events at `/silas/channels/slack/events` (the mounted
|
|
160
|
+
engine verifies signatures). A new thread starts a session; replies continue
|
|
161
|
+
it; and a held refund renders as **Approve/Decline buttons in Slack** that
|
|
162
|
+
call the exact same `approve!` as the inbox. Any other transport:
|
|
163
|
+
|
|
164
|
+
```bash
|
|
165
|
+
bin/rails g silas:channel whatsapp
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
scaffolds the signature-verifying webhook and the outbound half. See
|
|
169
|
+
[channels.md](channels.md).
|
|
170
|
+
|
|
171
|
+
## 7 · Let it ask: `ask_question`
|
|
172
|
+
|
|
173
|
+
Approvals are the human saying yes/no. `ask_question` is the reverse — the
|
|
174
|
+
agent needs *information*, not permission. It's built in; instruct it in
|
|
175
|
+
`app/agent/instructions.md`:
|
|
176
|
+
|
|
177
|
+
```markdown
|
|
178
|
+
- If the customer's request is ambiguous (which order? partial or full?), use
|
|
179
|
+
ask_question to ask the operator before touching money.
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
When the model calls it, the turn parks the same way an approval does — zero
|
|
183
|
+
compute, amber card — but the card has a **text box**. Your typed answer
|
|
184
|
+
becomes the tool result and the turn resumes with it. Same TTL, same audit
|
|
185
|
+
trail, same API (`POST .../approvals/:id/answer`).
|
|
186
|
+
|
|
187
|
+
## 8 · Let it keep notes: memory
|
|
188
|
+
|
|
189
|
+
Memory is for the fuzzy residue with no natural home in your tables —
|
|
190
|
+
"ada@example.com prefers replacement over refund" — stored as
|
|
191
|
+
`subject · attribute · content` triples with provenance and supersession.
|
|
192
|
+
|
|
193
|
+
It's already on. Tell the agent when to use it (instructions again):
|
|
194
|
+
|
|
195
|
+
```markdown
|
|
196
|
+
- When a customer states a durable preference, remember it.
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
The first time the model calls `remember`, the memory **parks as a card in
|
|
200
|
+
your inbox** — nothing persists until you approve it
|
|
201
|
+
(`config.memory_approval = :always` is the default). Approved memories inject
|
|
202
|
+
into future turns automatically; `recall` digs deeper on demand. Your *domain*
|
|
203
|
+
data stays in your tables, where your tools already read it.
|
|
204
|
+
|
|
205
|
+
## 9 · Hire staff: named agents and handoffs
|
|
206
|
+
|
|
207
|
+
One desk, many specialists. A named agent is the same directory shape under
|
|
208
|
+
`app/agents/<name>/`:
|
|
209
|
+
|
|
210
|
+
```
|
|
211
|
+
app/agents/escalations/
|
|
212
|
+
instructions.md # "You handle disputes the desk hands you…"
|
|
213
|
+
agent.yml
|
|
214
|
+
tools/ # its own toolset — the desk's tools are NOT inherited
|
|
215
|
+
schedules/ # its own clock, ticking IT — not the root agent
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
Restart, and the desk can delegate durably: the built-in `handoff` tool files
|
|
219
|
+
a **self-contained brief** that starts a linked session for the named agent —
|
|
220
|
+
exactly-once-guarded and cycle-checked — instead of two models chatting
|
|
221
|
+
freely (a cost and audit hazard, deliberately unblessed). Talk to a
|
|
222
|
+
specialist directly with `Silas.agent("escalations").start(input: "…")` or
|
|
223
|
+
`bin/rails silas:chat AGENT=escalations`; the inbox filters by agent.
|
|
224
|
+
|
|
225
|
+
## 10 · Fit the governors: budgets and compaction
|
|
226
|
+
|
|
227
|
+
Open `app/agent/agent.yml`:
|
|
228
|
+
|
|
229
|
+
```yaml
|
|
230
|
+
limits:
|
|
231
|
+
max_steps: 8 # runaway guard — breaching this FAILS the turn
|
|
232
|
+
max_cost: 0.25 # dollars per turn — breaching PARKS it
|
|
233
|
+
timeout: 300 # active seconds — held time never counts
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
A parked budget breach is an amber card with a **raise-budget** control: the
|
|
237
|
+
top-up is recorded per turn, so one exception never loosens the standing
|
|
238
|
+
limits ([budgets.md](budgets.md)). And long conversations don't die at the
|
|
239
|
+
context window: past `config.compact_at` (default 90% of the model's window)
|
|
240
|
+
Silas summarises prior turns — as a **persisted, exactly-once effect**, so a
|
|
241
|
+
crash replay rebuilds byte-identical messages. You configure nothing; it's a
|
|
242
|
+
default.
|
|
243
|
+
|
|
244
|
+
## 11 · Ship it
|
|
245
|
+
|
|
246
|
+
```bash
|
|
247
|
+
bin/rails silas:doctor
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
Work through its checklist for production:
|
|
251
|
+
|
|
252
|
+
1. A real provider key in the production environment.
|
|
253
|
+
2. Real inbox auth — replace the template's dev-only lambda in
|
|
254
|
+
`config/initializers/silas.rb` with your `current_user` check.
|
|
255
|
+
3. Keep `silas_dead_job_rescuer` in `config/recurring.yml` — it's part of the
|
|
256
|
+
crash-recovery contract — and **monitor worker liveness**: the rescuer can
|
|
257
|
+
requeue work; it cannot conjure a consumer.
|
|
258
|
+
4. `bin/ci` in your pipeline — the evals you wrote are now the gate that stops
|
|
259
|
+
a deploy from changing the desk's behavior unnoticed.
|
|
260
|
+
5. Deploy with Kamal as usual. The hard-won operational notes (what the chaos
|
|
261
|
+
harness taught us about dead workers, why you never hand-delete
|
|
262
|
+
`solid_queue_processes` rows) are in
|
|
263
|
+
[DEPLOY.md](https://github.com/danielstpaul/silas/blob/main/DEPLOY.md).
|
|
264
|
+
|
|
265
|
+
Then delete the desk — models, tools, seeds — and build your own agent in the
|
|
266
|
+
hole it leaves. The shape you learned is the whole framework: **a directory of
|
|
267
|
+
plain files, a ledger that makes effects land exactly once, and a signal that
|
|
268
|
+
holds anything consequential until a person clears it.**
|