robot_lab-audit 0.2.6

data/docs/getting_started.md

@@ -0,0 +1,129 @@
+# Getting Started
+
+## Prerequisites
+
+- Ruby 3.2+ (per the gemspec's `required_ruby_version`)
+- `robot_lab` — `robot_lab/audit` requires `RobotLab::Hook` to already be defined, so `robot_lab` must be `require`d first.
+- SQLite — the gem depends on the `sqlite3` gem directly; no separate database server to run.
+
+## Installation
+
+Add to your `Gemfile`:
+
+```ruby
+gem "robot_lab"
+gem "robot_lab-audit"
+```
+
+Then:
+
+```sh
+bundle install
+```
+
+Or install directly:
+
+```sh
+gem install robot_lab-audit
+```
+
+## Enabling Globally
+
+```ruby
+require "robot_lab"
+require "robot_lab/audit"
+
+RobotLab::Audit.enable(db_path: "~/.robot_lab/audit.db")
+
+network = RobotLab.create_network(...)
+network.run(message: "do something")
+```
+
+`RobotLab::Audit.enable(db_path:)`:
+
+1. Instantiates `RobotLab::Audit::EventLog`, which opens (or creates) the SQLite
+   file at `db_path`, creating any missing parent directories along the way.
+2. Assigns the log to `RobotLab::Audit::Hook.event_log`.
+3. Registers the hook globally via `RobotLab.on(Hook)`.
+
+Calling `enable` again with a new `db_path` simply repoints the hook at a new
+`EventLog` — it is safe to call once at boot.
+
+If `robot_lab` has not been loaded yet, `enable` raises
+`RobotLab::Audit::Error` ("robot_lab must be loaded before calling
+RobotLab::Audit.enable").
+
+## What Gets Logged
+
+| Table | One row per | Key columns |
+|-------|--------------|-------------|
+| `network_runs` | `Network#run` call | `run_id`, `network_name`, `input`, `result`, `error_class`, `error_message`, `started_at`, `finished_at`, `duration_ms` |
+| `audit_events` | `Robot#run` or `Tool#call` | `run_id`, `event_type`, `robot_name`, `tool_name`, `input`, `output`, `error_class`, `error_message`, `started_at`, `finished_at`, `duration_ms` |
+
+`event_type` is `"robot_run"` or `"tool_call"`. All `input`/`output`/`result`
+values are stored as JSON text. Timestamps are UTC, ISO 8601, with millisecond
+precision.
+
+Every event produced while a `Network#run` is in flight shares that run's
+`run_id`, so you can pull the entire trace for one execution with
+`events_for(run_id)` (see [Querying the Log](querying.md)).
+
+Standalone `Robot#run` calls made outside of a network are recorded too —
+their `run_id` column is `nil`.
+
+## Minimal Example
+
+```ruby
+require "robot_lab"
+require "robot_lab/audit"
+require "tmpdir"
+
+RobotLab::Audit.enable(db_path: File.join(Dir.mktmpdir, "audit.db"))
+
+robot = RobotLab.build(name: "greeter", system_prompt: "You are a friendly greeter.")
+robot.run("Say hello")
+
+log = RobotLab::Audit::Hook.event_log
+log.events_for(nil).each do |event|
+  puts "#{event[:event_type]}  #{event[:robot_name]}  #{event[:duration_ms]}ms"
+end
+```
+
+The gem ships a fuller, self-contained demo with no API keys required — it
+uses deterministic stubbed robots and tools so the audit trail is
+reproducible:
+
+```sh
+bundle exec ruby examples/01_basic_usage.rb
+```
+
+It walks through five scenarios: enabling the hook, a two-robot network run,
+tool call capture, error capture, and scoping the hook to a single robot.
+
+## Scoping to a Robot or Network
+
+`RobotLab::Audit.enable` registers the hook globally — every robot and
+network in the process gets logged. To audit only specific objects, skip
+`enable` and register the hook directly instead:
+
+```ruby
+RobotLab::Audit::Hook.event_log = RobotLab::Audit::EventLog.new(db_path: "audit.db")
+
+robot.on(RobotLab::Audit::Hook)
+# or
+network.on(RobotLab::Audit::Hook)
+```
+
+Robots and networks that never call `.on(RobotLab::Audit::Hook)` produce no
+audit entries at all — the hook simply never fires for them.
+
+If you had previously called `RobotLab::Audit.enable`, clear the global hook
+registry first (`RobotLab.hooks.clear`) so the same events aren't logged
+twice — once globally and once for the specific object.
+
+## Key Constraints
+
+- `robot_lab` must be `require`d before `robot_lab/audit` — see Prerequisites above.
+- `EventLog` is not thread-safe by itself; SQLite handles concurrent writes via its own file locking.
+- Run IDs are correlated via `Thread.current` — they are **not** propagated across Fibers or Ractors. Work dispatched off-thread (e.g. via `robot_lab-ractor`) will not share a `run_id` with its parent network run.
+- `safe_json` (used internally to serialise `input`/`output`) silently falls back to `#inspect` for values that can't be JSON-encoded — don't assume every `input`/`output` column is valid JSON when a tool passes an unusual argument type.

data/docs/how_it_works.md

@@ -0,0 +1,193 @@
+# How It Works
+
+## Schema
+
+`EventLog#initialize` opens the SQLite file at `db_path` (expanding `~` and
+creating any missing parent directories via `FileUtils.mkdir_p`), then runs a
+`CREATE TABLE IF NOT EXISTS` schema batch — safe to call every time the
+process boots.
+
+```sql
+CREATE TABLE network_runs (
+  id            INTEGER PRIMARY KEY AUTOINCREMENT,
+  run_id        TEXT    NOT NULL UNIQUE,
+  network_name  TEXT,
+  input         TEXT,
+  result        TEXT,
+  error_class   TEXT,
+  error_message TEXT,
+  started_at    TEXT    NOT NULL,
+  finished_at   TEXT,
+  duration_ms   REAL
+);
+
+CREATE TABLE audit_events (
+  id            INTEGER PRIMARY KEY AUTOINCREMENT,
+  run_id        TEXT,
+  event_type    TEXT NOT NULL,
+  robot_name    TEXT,
+  tool_name     TEXT,
+  input         TEXT,
+  output        TEXT,
+  error_class   TEXT,
+  error_message TEXT,
+  started_at    TEXT NOT NULL,
+  finished_at   TEXT,
+  duration_ms   REAL
+);
+```
+
+Two tables, no foreign key constraint between them — `audit_events.run_id`
+is simply a shared correlation string, nullable for events that happen
+outside of any network run.
+
+| Column | Present on | Notes |
+|--------|-----------|-------|
+| `run_id` | both | `NOT NULL UNIQUE` on `network_runs` (one row per run); nullable, non-unique on `audit_events` (many rows share a run) |
+| `event_type` | `audit_events` only | `"robot_run"` or `"tool_call"` |
+| `robot_name` / `tool_name` | `audit_events` only | `tool_name` is `nil` for `"robot_run"` events |
+| `network_name` | `network_runs` only | taken from `ctx.network&.name` |
+| `input` / `output` / `result` | both | JSON-encoded text (see [JSON Serialisation](#json-serialisation-safe_json) below) |
+| `error_class` / `error_message` | both | populated from the exception object when the call raised, else `nil` |
+| `started_at` / `finished_at` | both | UTC, ISO 8601, millisecond precision (`Time#iso8601(3)`) |
+| `duration_ms` | both | `(finished - started) * 1000`, rounded to 2 decimal places |
+
+## Hook Callback Mapping
+
+`RobotLab::Audit::Hook` is a subclass of `RobotLab::Hook` with
+`self.namespace = :audit`. It implements three before/after callback pairs
+from RobotLab's [Hook system](https://github.com/MadBomber/robot_lab/blob/main/docs/guides/hooks.md):
+
+| Hook family | Callbacks | Writes |
+|-------------|-----------|--------|
+| `:network_run` | `before_network_run`, `after_network_run` | one `network_runs` row |
+| `:run` | `before_run`, `after_run` | one `audit_events` row, `event_type: "robot_run"` |
+| `:tool_call` | `before_tool_call`, `after_tool_call` | one `audit_events` row, `event_type: "tool_call"` |
+
+`before_network_run`, `after_network_run`, `after_run`, and `after_tool_call`
+all `return unless event_log` (or `event_log && current_run_id`) as their
+first line — they are true no-ops when `Hook.event_log` is `nil`. This is what
+makes it safe to `require "robot_lab/audit"` without calling `enable`, and
+what makes per-robot/per-network scoping work cleanly.
+
+`before_run` and `before_tool_call` are a partial exception: they run
+unconditionally, stashing `Time.now.utc` on `ctx.local.started_at` even when
+`event_log` is `nil`. That write has no observable effect on its own — it's
+local, namespace-scoped state (see below) that only gets read if the matching
+`after_*` callback goes on to record a row, and that callback still checks
+`event_log` before touching the database. So the *net* behavior is the same
+(nothing is written when auditing is off), it's just the `after_*` half of
+each pair that carries the guard, not the `before_*` half.
+
+### Network run lifecycle
+
+```
+before_network_run(ctx)
+  → generates a new run_id  (SecureRandom.uuid)
+  → stores run_id and started_at in Thread.current
+
+  ... network executes: robots run, tools are called ...
+
+after_network_run(ctx)
+  → writes the network_runs row using the thread-local run_id/started_at
+  → clears both thread-locals in an `ensure` block
+```
+
+`after_network_run` is a no-op if there is no `current_run_id` — this
+guards against `after_network_run` firing without a matching
+`before_network_run` (for example, if the hook was registered mid-flight).
+
+### Robot run and tool call lifecycle
+
+`before_run` / `before_tool_call` don't touch `Thread.current` at all —
+they stash `started_at` on `ctx.local`, which is namespace-scoped state that
+RobotLab's `HookContext` provides per-callback-pair (see
+`ctx.with_namespace(:audit)` in the test suite for how this is activated).
+
+`after_run` / `after_tool_call` then read `ctx.local.started_at`, look up the
+thread-local `run_id` set by an enclosing `before_network_run` (or `nil` if
+there is none), and write one `audit_events` row via the shared private
+`record_audit_event` helper.
+
+### Run ID correlation
+
+All events produced while a given `Network#run` is executing — the network
+run itself, every robot invocation, every tool call — share the same
+`run_id`, because that ID lives in `Thread.current` for the duration of the
+network run and every robot/tool callback reads it. This is what makes
+`EventLog#events_for(run_id)` a full execution trace rather than a fragment.
+
+Standalone `Robot#run` or `Tool#call` invocations made outside of any
+network run see `current_run_id` as `nil` — they still get logged, just
+without a correlating run.
+
+## Thread-Local State
+
+Two `Thread.current` keys carry state between the `before_*` and `after_*`
+halves of the network-run lifecycle:
+
+| Key | Set in | Cleared in | Purpose |
+|-----|--------|-----------|---------|
+| `:robot_lab_audit_run_id` | `before_network_run` | `after_network_run` (`ensure`) | correlates all events within one network run |
+| `:robot_lab_audit_network_started_at` | `before_network_run` | `after_network_run` (`ensure`) | computes the network run's `duration_ms` |
+
+Because this is `Thread.current` rather than `ctx.local`, run correlation
+does **not** cross Fiber or Ractor boundaries — parallel work dispatched via
+`robot_lab-ractor`, for instance, will not inherit the parent network run's
+`run_id`. `before_run`/`before_tool_call` state, by contrast, uses `ctx.local`
+and is scoped per-callback-pair regardless of thread.
+
+## JSON Serialisation (`safe_json`)
+
+`input`/`output` values (tool arguments, robot request/response payloads) are
+serialised with a private `safe_json` helper:
+
+```ruby
+def safe_json(value)
+  return nil if value.nil?
+
+  JSON.generate(value)
+rescue JSON::GeneratorError, TypeError
+  value.inspect
+end
+```
+
+If a value can't be JSON-encoded (circular references, unsupported types),
+the column falls back to Ruby's `#inspect` string instead of raising. Don't
+assume every `input`/`output` value is parseable JSON — check for `nil`
+first if you write tooling against the column, and be prepared for an
+`inspect`-style string on the rare unencodable payload.
+
+`network_runs.input` and `.result` use dedicated extractors
+(`extract_network_input`, `extract_network_result`) that look for a
+`:message`/`'message'` key on the network context, or — for the result — a
+`.reply` on `result.value` when the result object responds to `:value` (the
+shape `simple_flow`'s result wrapper returns), falling back to `safe_json`
+and finally to `#inspect` if anything raises.
+
+## Error Capture
+
+When a network run, robot run, or tool call raises, the hook's
+`error_fields` helper extracts `error_class` (the exception's class name as
+a string) and `error_message` (the exception's message) from whatever
+`ctx.error` / `ctx.tool_error` object RobotLab's Hook system populates. Both
+columns are `nil` on success. `EventLog#recent_errors` filters
+`audit_events` down to rows where `error_class IS NOT NULL`.
+
+## RobotLab Extension Registration
+
+`lib/robot_lab/audit.rb` ends with:
+
+```ruby
+require_relative 'audit/hook' if defined?(RobotLab::Hook)
+
+if defined?(RobotLab) && RobotLab.respond_to?(:register_extension)
+  RobotLab.register_extension(:audit, RobotLab::Audit)
+end
+```
+
+The `Hook` subclass is only loaded when `RobotLab::Hook` is already defined,
+and extension registration only runs when `RobotLab.register_extension`
+exists — both guards make `require "robot_lab/audit"` safe even in contexts
+where `robot_lab` hasn't been (fully) loaded, though calling `enable` in that
+state still raises (see [Getting Started](getting_started.md)).

data/docs/index.md

@@ -0,0 +1,44 @@
+# robot_lab-audit
+
+A SQLite-backed execution audit log for the [RobotLab](https://github.com/MadBomber/robot_lab) LLM agent framework.
+
+`RobotLab::Audit` plugs into RobotLab's [Hook system](https://github.com/MadBomber/robot_lab/blob/main/docs/guides/hooks.md) and writes one row per network run, robot invocation, and tool call to a local SQLite database — timestamps, JSON input/output, errors, and duration in milliseconds. Zero changes required to your robots, networks, or tools.
+
+```ruby
+require "robot_lab"
+require "robot_lab/audit"
+
+RobotLab::Audit.enable(db_path: "~/.robot_lab/audit.db")
+
+# All subsequent network and robot runs are recorded automatically.
+network = RobotLab.create_network(...)
+network.run(message: "do something")
+```
+
+Every event from a single `Network#run` shares the same `run_id`, so the full
+execution trace — which robots ran, which tools they called, what each one
+returned, and how long it took — can be reconstructed after the fact for
+post-mortem analysis or debugging.
+
+## Navigation
+
+- [Getting Started](getting_started.md) — installation, enabling the hook, a minimal example, scoping to one robot or network
+- [How It Works](how_it_works.md) — the SQLite schema, how each Hook callback maps to a row, run_id correlation, thread-local state, JSON serialisation
+- [Querying the Log](querying.md) — the `EventLog` query API and raw SQL examples for post-mortem analysis
+
+## At a Glance
+
+| | |
+|---|---|
+| **Storage** | SQLite (via the `sqlite3` gem) — one file, no server |
+| **Tables** | `network_runs`, `audit_events` |
+| **Wiring** | `RobotLab::Hook` subclass (`RobotLab::Audit::Hook`), namespace `:audit` |
+| **Enable globally** | `RobotLab::Audit.enable(db_path:)` |
+| **Enable per robot/network** | `robot.on(RobotLab::Audit::Hook)` / `network.on(RobotLab::Audit::Hook)` |
+| **Query API** | `EventLog#network_runs`, `#events_for(run_id)`, `#recent_errors` |
+
+## Links
+
+- [RobotLab Core](https://github.com/MadBomber/robot_lab)
+- [RubyGems](https://rubygems.org/gems/robot_lab-audit)
+- [GitHub](https://github.com/MadBomber/robot_lab-audit)

data/docs/querying.md

@@ -0,0 +1,126 @@
+# Querying the Log
+
+`RobotLab::Audit::EventLog` is the only public query surface. Get a handle
+to the active log via the hook (when using global `enable`):
+
+```ruby
+log = RobotLab::Audit::Hook.event_log
+```
+
+or hold onto the `EventLog` instance yourself if you constructed it directly
+for scoped auditing (see [Getting Started — Scoping](getting_started.md#scoping-to-a-robot-or-network)).
+
+Each query method returns an `Array` of `Hash` objects keyed by column name
+as symbols (e.g. `row[:run_id]`, `row[:duration_ms]`).
+
+## `network_runs(limit: 100)`
+
+Recent network runs, newest first (ordered by `started_at DESC`).
+
+```ruby
+log.network_runs(limit: 20).each do |run|
+  status = run[:error_class] ? "FAILED (#{run[:error_class]})" : "OK"
+  puts "#{run[:run_id]}  #{run[:network_name]}  #{status}  #{run[:duration_ms]}ms"
+end
+```
+
+Columns returned: `id`, `run_id`, `network_name`, `input`, `result`,
+`error_class`, `error_message`, `started_at`, `finished_at`, `duration_ms`.
+
+## `events_for(run_id)`
+
+Every `audit_events` row for one network run, ordered chronologically
+(`ORDER BY started_at`) — the full execution trace for that run: every robot
+invocation and every tool call, interleaved in the order they actually
+happened.
+
+```ruby
+run_id = log.network_runs.first[:run_id]
+
+log.events_for(run_id).each do |event|
+  label = event[:tool_name] || event[:robot_name]
+  puts "#{event[:event_type].ljust(10)} #{label.ljust(20)} #{event[:duration_ms]}ms"
+end
+```
+
+Columns returned: `id`, `run_id`, `event_type`, `robot_name`, `tool_name`,
+`input`, `output`, `error_class`, `error_message`, `started_at`,
+`finished_at`, `duration_ms`.
+
+Standalone `Robot#run`/`Tool#call` invocations made outside any network run
+are recorded with `run_id: nil` — pass `nil` to `events_for` to retrieve
+them:
+
+```ruby
+log.events_for(nil)
+```
+
+## `recent_errors(limit: 50)`
+
+`audit_events` rows where `error_class IS NOT NULL`, newest first — every
+failed robot run or tool call across all network runs (and standalone
+calls), regardless of `run_id`.
+
+```ruby
+log.recent_errors.each do |event|
+  puts "#{event[:started_at]}  #{event[:robot_name]}/#{event[:tool_name]}  " \
+       "#{event[:error_class]}: #{event[:error_message]}"
+end
+```
+
+This is the fastest way to answer "what broke recently?" without knowing a
+specific `run_id` up front.
+
+## Querying with Raw SQL
+
+`EventLog` doesn't try to be a full reporting layer — for anything beyond
+the three methods above, query the SQLite file directly. It's a plain
+single-file database; any SQL client works.
+
+```bash
+sqlite3 ~/.robot_lab/audit.db \
+  "SELECT event_type, robot_name, tool_name, duration_ms, error_class
+   FROM audit_events
+   WHERE run_id = 'some-run-id'
+   ORDER BY started_at;"
+```
+
+Useful ad hoc queries:
+
+```sql
+-- Slowest tool calls across all runs
+SELECT tool_name, robot_name, duration_ms, started_at
+FROM audit_events
+WHERE event_type = 'tool_call'
+ORDER BY duration_ms DESC
+LIMIT 20;
+
+-- Error rate by robot
+SELECT robot_name,
+       COUNT(*) AS total,
+       SUM(CASE WHEN error_class IS NOT NULL THEN 1 ELSE 0 END) AS errors
+FROM audit_events
+WHERE event_type = 'robot_run'
+GROUP BY robot_name;
+
+-- Network runs that failed
+SELECT run_id, network_name, error_class, error_message, started_at
+FROM network_runs
+WHERE error_class IS NOT NULL
+ORDER BY started_at DESC;
+```
+
+Because `input`/`output`/`result` are stored as JSON text, SQLite's
+[JSON1 functions](https://www.sqlite.org/json1.html) (`json_extract`, etc.)
+can reach into them directly:
+
+```sql
+SELECT json_extract(input, '$.text') AS text_arg, duration_ms
+FROM audit_events
+WHERE tool_name = 'word_count_tool';
+```
+
+Remember that a small fraction of `input`/`output` values may be a
+`#inspect` fallback string rather than valid JSON (see
+[How It Works — JSON Serialisation](how_it_works.md#json-serialisation-safe_json)) — `json_extract` will
+simply return `NULL` for those rows rather than raising.