insika 0.0.1 → 0.1.0

data/docs/WORKFLOWS.md

@@ -0,0 +1,225 @@
+---
+title: Workflows
+parent: Build an agent
+nav_order: 5
+permalink: /workflows/
+---
+
+# Workflows
+
+A single agent's tool-loop is one shape of work: the model decides, calls a tool,
+looks at the result, decides again. Plenty of real work has a shape you already
+know — draft then edit, classify then answer, three reviewers then a summary,
+try until it passes. For those, the choice of "what happens next" belongs in
+**Ruby**, not in a prompt.
+
+That is a **workflow**: deterministic orchestration around agent turns.
+
+## The one decision that matters
+
+**Who chooses the next step — your code, or the model?**
+
+| | **Workflow** | **Delegation (`subagents`)** |
+|---|---|---|
+| Chooses the next step | your Ruby | the model |
+| Cost of deciding | zero | a model call |
+| Repeatable | yes — same path every run | no |
+| Reach for it when | the shape is known in advance | the work depends on what the user said |
+| Surface | `workflow` in a system + `POST /v1/workflows/:name` | `subagents` on a profile + `spawn_subagent(s)` |
+
+They compose: a workflow step can `ask` an agent that itself delegates. Start
+with a workflow and hand the choice to the model only where the choice is
+genuinely open — see [Agents](AGENTS.md#delegation-subagents) for delegation.
+
+## Declaring one
+
+Workflows live in a **system** (`Insika.system`), because their steps address
+agents by id and those agents must resolve in the same runtime:
+
+```ruby
+newsroom = Insika.system do
+  provider :deepseek
+
+  agent("writer") { model "deepseek-chat"; instructions "Write ONE paragraph." }
+  agent("editor") { model "deepseek-chat"; instructions "Rewrite as ONE sentence." }
+
+  workflow "publish",
+           description: "Draft a paragraph, then tighten it.",
+           input:  { type: "object", properties: { topic: { type: "string" } }, required: ["topic"] },
+           output: { type: "object", properties: { headline: { type: "string" } } } do |input, ctx|
+    draft = ctx.ask("writer", "Topic: #{input['topic']}")
+    { "headline" => ctx.ask("editor", draft) }
+  end
+end
+
+newsroom.run("publish", input: { "topic" => "Ruby fibers" })
+# => { "headline" => "…" }
+```
+
+What the block receives:
+
+| | What it does |
+|---|---|
+| `input` | the validated input Hash (string keys) |
+| `ctx.ask(agent, message)` | one turn against one agent of the system → its text |
+| `ctx.gather(*blocks, max: 8)` | runs blocks **concurrently**, returns values **in order** |
+| `ctx.context` / `ctx.tools` | the turn's ContextPackage and its resolved tool instances — escape hatches, rarely needed |
+
+The workflow's return value **is** its output: whatever Hash (or value) you
+return is what `run` returns, what `:workflow_completed` carries, and what the
+`output` schema validates.
+
+## What you get for free
+
+- **A durable run.** The run id *is* a Task — checkpointed and recoverable, so a
+  crash mid-run does not lose the record. `run_id` comes back from every trigger.
+- **Schemas at the edges.** `input:` is validated **synchronously**: a bad input
+  is refused with **no run created** (a `422` over HTTP). `output:` is validated
+  when the workflow returns; a violation fails the run at the `workflow_schema`
+  stage instead of handing a malformed result downstream. Both take a JSON Schema
+  Hash or any dry-schema-compatible validator.
+- **Events.** `:workflow_started` and `:workflow_completed` (with the typed
+  output) on the turn's event stream, alongside every step's own turn events.
+- **Every step is visible.** Each `ctx.ask` is its own turn — its own Task, its
+  own trace in the Studio. A workflow is not a black box.
+
+## Over HTTP
+
+Workflows are exposed when the deployment injects the registry (a `serve`d system
+with at least one workflow does it automatically):
+
+```bash
+GET  /v1/workflows                      # discovery: names, descriptions, I/O schemas
+POST /v1/workflows/publish              # 202 { run_id, task_id } — fire and observe
+POST /v1/workflows/publish?stream=true  # SSE of the run, ending at workflow_completed
+```
+
+```jsonc
+// POST body
+{ "agent": "writer",                    // the profile the run executes under
+  "input": { "topic": "Ruby fibers" },
+  "session_id": null }                  // optional
+```
+
+Observe an async run with `GET /v1/tasks/:run_id` or
+`GET /v1/events?task_id=:run_id`. A bad input is a `422`; an unknown workflow or
+agent is a `404`.
+
+## The five patterns
+
+Each has a runnable script in
+[`examples/agentic-workflows/`](https://github.com/guizaols/insika/tree/main/examples/agentic-workflows/).
+
+### Sequential (prompt chaining)
+
+Each step's output feeds the next. The order is code, so it is identical on
+every run.
+
+```ruby
+draft = ctx.ask("writer", input["topic"])
+ctx.ask("editor", draft)
+```
+
+### Routing
+
+One cheap classification turn, then a specialist with a short, focused prompt —
+instead of one agent carrying every instruction it might ever need.
+
+```ruby
+label = ctx.ask("router", input["message"]).downcase       # constrain the label set
+lane  = label.include?("billing") ? "billing" : "technical"
+ctx.ask(lane, input["message"])
+```
+
+**Do the branching in Ruby, with a fallback.** A closed label set plus an `else`
+means an unexpected answer degrades instead of picking a random branch.
+
+### Parallel (fan-out / fan-in)
+
+An LLM turn is almost all *waiting* on the provider, so independent turns overlap
+on the fiber scheduler: wall-clock is the **slowest** branch, not the sum.
+
+```ruby
+security, performance = ctx.gather(-> { ctx.ask("security", code) },
+                                  -> { ctx.ask("performance", code) })
+ctx.ask("lead", "security: #{security}\nperformance: #{performance}")   # fan-in
+```
+
+`gather` returns values in declaration order and caps concurrency at `max:`
+(default 8) — the cap matters, because N simultaneous calls hit provider rate
+limits and per-agent token ceilings.
+
+### Evaluator-optimizer
+
+Produce, judge, revise. The judge's reason becomes the revision instruction.
+
+```ruby
+loop do
+  verdict = ctx.ask("critic", "Brief: #{brief}\nTagline: #{tagline}")
+  break if verdict.include?("PASS") || attempts >= MAX_ATTEMPTS   # the cap is CODE
+  tagline = ctx.ask("writer", "Fix this: #{verdict}")
+  attempts += 1
+end
+```
+
+**Two things must be code, not prompt:** the attempt cap (a loop that asks the
+model when to stop can run forever) and the honest report of whether it actually
+passed. Have the judge answer in a shape you can branch on (`VERDICT: PASS|FAIL`).
+
+### Orchestrator-workers
+
+The model chooses the workers. This is delegation, not a workflow — see
+[Agents](AGENTS.md#delegation-subagents).
+
+```ruby
+agent "lead" do
+  instructions "You have no expertise yourself — always delegate, then report."
+  subagents "security", "performance"
+end
+```
+
+## Gotchas
+
+- **`ctx.ask` is stateless.** A step is a unit of work, not a conversation: no
+  session is threaded, so the agent sees only the message you pass. Pass what it
+  needs.
+- **Nothing forces the model to delegate.** If a parent answers alone, the task
+  list shows only the parent — and it is the *prompt* that needs work: say
+  plainly that it has no expertise of its own. (The engine helps by naming the
+  spawnable agent ids in the tool contract, but it cannot make the model use them.)
+- **The `agent:` of a run is the profile it executes under** — its policy,
+  limits, and guardrails. It is not "the agent that does the work"; the steps
+  pick those themselves.
+- **A workflow turn assembles no chat.** Stage 5 is skipped: there is no
+  model call for the *workflow itself*, only for the turns its steps start.
+  One consequence worth naming: `queue_mode: "steer"`
+  ([Agents](AGENTS.md#steer--the-message-arrives-while-the-turn-is-already-running))
+  cannot append into a running workflow — there is no chat to append to, and no tool
+  batch of the engine's to use as a boundary. A message that arrives while a workflow
+  run is in flight becomes the next turn on the session, which is what `followup`
+  does. Steering is refused at the door, not silently dropped.
+- **A running workflow reaches no boundary until it returns.** The engine's stage
+  boundaries sit *around* the workflow call, not inside it, so `queue_mode:
+  "interrupt"` (and a plain `cancel_task`) is observed only once the workflow is done:
+  the steps run, and then the run is abandoned without publishing or persisting. If you
+  need a run to stop early, that decision belongs inside the workflow — it is ordinary
+  Ruby, so `return` on the condition you care about.
+- **Braces bite in Ruby.** `agent "x" { … }` is a syntax error (`{` binds to the
+  argument); write `agent("x") { … }` or `agent "x" do … end`.
+
+## Troubleshooting
+
+| Symptom | Cause |
+|---|---|
+| `GET /v1/workflows` 404s | no workflow declared, so the routes are not exposed (parity) |
+| `422` on trigger | the input violates `input:` — the message names the offending field |
+| `404` on trigger | unknown workflow name, or the `agent:` does not exist |
+| run fails at `workflow_schema` | your return value violates `output:` |
+| a step "does nothing" | the agent id is not in this system — `ask` raises with the list of valid ids |
+
+## See also
+
+- [Agents](AGENTS.md) — profiles, and delegation via `subagents`.
+- [Tools](TOOLS.md) — what a single agent can call inside one turn.
+- [Architecture](ARCHITECTURE.md) — the turn pipeline a run goes through, and checkpointing.
+- [`examples/agentic-workflows/`](https://github.com/guizaols/insika/tree/main/examples/agentic-workflows/) — the five patterns, runnable.

data/docs/build.md

@@ -0,0 +1,14 @@
+---
+title: Build an agent
+nav_order: 3
+has_children: true
+permalink: /build/
+---
+
+# Build an agent
+
+An agent is data: a profile, its tools, its skills, and what fills its prompt. Start
+with [Agents](AGENTS.md), then add capability one doc at a time — and when config is not
+enough, [Plugins](PLUGINS.md) covers the two ways to extend the engine itself. Every one of these has a
+runnable counterpart under
+[`examples/`](https://github.com/guizaols/insika/tree/main/examples/).

data/docs/index.md

@@ -0,0 +1,68 @@
+---
+title: Home
+nav_order: 1
+permalink: /
+---
+
+# Insika
+{: .fs-9 }
+
+Your agent is the idea. Insika is what holds it up in production.
+{: .fs-6 .fw-300 }
+
+[Build your first agent](RUNNING-LOCAL.md){: .btn .btn-primary .fs-5 .mb-4 .mb-md-0 .mr-2 }
+[View on GitHub](https://github.com/guizaols/insika){: .btn .fs-5 .mb-4 .mb-md-0 }
+
+---
+
+*Insika* is Zulu for the pillar that carries a structure — the part nobody admires and
+everything rests on. A turn that survives a crash, tools that cannot wander off, limits
+that hold under load, and an API your clients already speak. Build the agent; the
+scaffolding is already here.
+
+Concretely: a Ruby runtime for **LLM agents in production** — a durable, resumable turn
+pipeline behind an **OpenAI-Responses-compatible** HTTP API (`POST /v1/responses`), with
+tools, skills, cross-session memory, per-agent policy, content-safety guardrails, and a
+web control UI. Point an existing Responses client at it and serve many agents from one
+deployment.
+
+## Your first agent
+
+Ruby `>= 3.3` and a provider key (the demo uses DeepSeek). The whole program:
+
+```ruby
+require "insika"
+
+assistant = Insika.agent("assistant") do
+  model "deepseek-chat"
+  provider :deepseek
+  instructions "You are Bia, a concise and friendly assistant. Answer briefly."
+end
+
+puts assistant.reply("hi, what can you do?")   # one turn, in-process
+```
+
+Swap `reply` for `serve` and the same agent is a server — the control UI at `/studio`
+plus the drop-in API, on `:9292`. → [Running locally](RUNNING-LOCAL.md)
+
+## Or let your coding agent read the docs
+
+A running instance serves this same documentation as raw markdown, plus a
+skill-structured prompt that walks a coding agent through building your first agent:
+
+```
+Read http://localhost:9292/start.md then help me build my first agent
+```
+
+Alongside it, `GET /models.json` (configured providers, model ids, defaults — no
+secrets), `GET /docs` and `GET /docs/<name>.md`. Public and on by default when you
+`serve`; opt-in in production (`INSIKA_ONBOARDING=1`).
+
+## Where to go next
+
+- **[Understand the idea](understand.md)** — why a runtime rather than a DIY loop, and how a turn actually runs.
+- **[Build an agent](build.md)** — agents, tools, skills, context, and the local loop.
+- **[Ship it](ship.md)** — security, confined execution, deployment.
+- **[Operate & prove it](operate.md)** — observability, the benchmark, load testing, evals, refinement.
+
+Pre-release: APIs may still change and nothing is tagged yet. Licensed MIT.

data/docs/onboarding/start.md

@@ -0,0 +1,126 @@
+# Insika — build your first agent
+
+> **You are a coding agent** (Claude Code, Cursor, an IDE assistant, …) reading this
+> on behalf of a developer who just pointed you at a running Insika. Treat this file
+> as a **skill**: follow the steps in order, apply the RULES literally, and stop at the
+> self-check. Do **not** improvise beyond it.
+
+Insika is a Ruby runtime for LLM agents in production. Your job here is the smallest
+possible one: get the developer a **first working agent**, defined in Ruby with the
+public DSL (`Insika.agent { … }`), talking to a real model — nothing more.
+
+The machine-readable list of models this engine already knows about is at
+**{{MODELS_URL}}** — fetch it before you write any `model`/`provider` line. The full
+docs are mirrored as raw markdown under **{{DOCS_URL}}**.
+
+---
+
+## Step 0 — Gather context (do this first, silently)
+
+RULES — verify, do not assume:
+
+- **Ruby ≥ 3.3.** Run `ruby -v`. If lower, stop and tell the developer; do not try to
+  upgrade Ruby for them.
+- **The gem/library must be loadable.** In a project that already depends on Insika,
+  `require "insika"` works. Otherwise add it to the `Gemfile` (or `bundle add`) — do
+  not vendor or copy source files.
+- **A provider key must come from the developer, via the environment.** Look for one
+  already exported (e.g. `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`). If none is set, **ask
+  the developer for the provider and confirm the env var is exported.** See the hard
+  constraint on keys below.
+- **Fetch {{MODELS_URL}}.** It tells you which providers and model ids this engine is
+  configured for, the platform default, and the valid `thinking` levels. Use those
+  exact ids.
+
+## Step 1 — Decide (RULES, not taste)
+
+| Question | RULE |
+|---|---|
+| How many agents? | **Exactly one.** A first agent is a single `Insika.agent`. Resist adding a second. |
+| Which model/provider? | Use an id from **{{MODELS_URL}}**. If it lists a `default`, use that. Never guess a model id. |
+| Where does it live? | One new Ruby file (e.g. `my_agent.rb`), or extend `examples/quickstart.rb` if present. One file. |
+| Tools / skills / memory? | **None yet.** Ship a plain conversational agent first; add capability only after it replies. |
+| Reply or serve? | Start with `reply` (one in-process turn). Only switch to `serve` once `reply` works. |
+
+## Step 2 — Build (exact shape)
+
+Write **one** file. This is the whole program:
+
+```ruby
+require "insika"
+
+assistant = Insika.agent("assistant") do
+  provider :deepseek            # ← the provider slug from {{MODELS_URL}}
+  model "deepseek-chat"         # ← a model id from {{MODELS_URL}}
+  instructions "You are a concise, friendly assistant. Answer briefly."
+end
+
+puts assistant.reply(ARGV.join(" ").empty? ? "hi, what can you do?" : ARGV.join(" "))
+```
+
+Notes that are RULES, not options:
+
+- The block is **config that generates data** — `assistant.to_pack` is a plain
+  provisioning pack. Do not reach past the DSL into internal classes; everything a
+  first agent needs is a DSL method (`model`, `provider`, `instructions`, `tools`,
+  `skill`, `memory`, `temperature`, …).
+- The provider key is read from the environment by name (`<PROVIDER>_API_KEY`). Do
+  **not** write it into the file, the DSL, or a committed config.
+
+## Step 3 — Run it
+
+```bash
+bundle install
+DEEPSEEK_API_KEY=sk-... ruby my_agent.rb "hi, what can you do?"
+```
+
+Once that prints a reply, turning the same agent into a server is one line — swap
+`reply` for `serve`:
+
+```ruby
+assistant.serve   # control UI at /studio  +  drop-in POST /v1/responses on :9292
+```
+
+Over the drop-in API the `model` field is the **agent id**:
+
+```bash
+curl -N http://localhost:9292/v1/responses \
+  -H "Authorization: Bearer local-demo" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"assistant","user":"chat-1","stream":true,"input":"hi"}'
+```
+
+## Step 4 — Self-check before you report done
+
+- [ ] `ruby -v` is ≥ 3.3.
+- [ ] The `model`/`provider` you wrote appear in **{{MODELS_URL}}**.
+- [ ] The provider key is exported in the environment, **not** written into any file.
+- [ ] Running the file printed a real model reply (not an auth/model error).
+- [ ] Exactly one agent, one file. No tools, skills, workflows, or extra agents added
+      "to test things".
+
+If any box is unchecked, fix that one thing — do not add scope to work around it.
+
+---
+
+## Hard constraints (known failure modes — never do these)
+
+- **Never invent, guess, or hard-code an API key.** If no key is available, ask. A
+  placeholder like `sk-xxxx` is not a fix — it produces a confusing auth failure.
+- **Never guess a model id.** Use only ids from {{MODELS_URL}}. A wrong id fails at
+  the provider, not in the engine, and wastes the developer's time.
+- **Do not create a workflow, tool, or second agent just to test the first one.** The
+  test is `reply` / one `curl`. Extra machinery is scope you were not asked for.
+- **Do not bypass the DSL / config-over-code.** If something seems to need a private
+  class, it is almost certainly a DSL method you have not used yet — check the docs at
+  {{DOCS_URL}} before reaching deeper.
+- **Keep secrets in the environment.** No keys in source, in the pack, or in commits.
+
+## Where to go next
+
+- **{{MODELS_URL}}** — live list of configured models, the default, and `thinking` levels.
+- **{{DOCS_URL}}** — the docs index (README, running locally, deploy, guardrails, …), raw markdown.
+- Add capability only after the first reply works: `tools`, `skill`, `memory`,
+  `temperature`, `data_tool` — each is a DSL method documented in the README.
+
+This is `rails new` reimplemented as a prompt — and you are the generator.

data/docs/operate.md

@@ -0,0 +1,12 @@
+---
+title: Operate & prove it
+nav_order: 5
+has_children: true
+permalink: /operate/
+---
+
+# Operate & prove it
+
+Turns as traces and metrics, the engine's measured overhead, how to load-test it
+yourself, the cases that grade an agent, and reading a live agent's own traffic back as
+a report of what broke.

data/docs/ship.md

@@ -0,0 +1,10 @@
+---
+title: Ship it
+nav_order: 4
+has_children: true
+permalink: /ship/
+---
+
+# Ship it
+
+What stands between your agent and the open internet, and how to put it on a server.

data/docs/understand.md

@@ -0,0 +1,10 @@
+---
+title: Understand the idea
+nav_order: 2
+has_children: true
+permalink: /understand/
+---
+
+# Understand the idea
+
+What Insika is for, what it replaces, and how a turn actually runs end to end.

data/lib/insika/agent_file_store.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+require "time"
+
+module Insika
+  # Per-agent workspace. Holds the
+  # CONTENT of each agent's prompt files (IDENTITY.md/SOUL.md/TOOLS.md
+  # and the like) in the durable Store — not on disk. It is what makes "each one
+  # builds its own BIA with its own identity": the Prompt provider reads from
+  # here, per agent, instead of the fixed `files:` in the wiring (which are now
+  # only the deployment default).
+  #
+  # One record per agent in the ConfigStore (scope "agent_files"):
+  #   { "files" => { "<name>" => { "content" => str,
+  #                                "updated_at" => iso8601,
+  #                                "history" => [ { "content" => str, "at" => iso8601 }, ... ] } } }
+  #
+  # A write versions: the previous content goes into `history` (most recent
+  # first), capped at HISTORY_MAX — restoring becomes a new write
+  # (preserves the linear history, no destructive "time travel").
+  class AgentFileStore
+    SCOPE = "agent_files"
+    HISTORY_MAX = 20
+
+    def initialize(config_store:)
+      @cs = config_store
+    end
+
+    # -> String | nil (current content; nil = nonexistent file/agent).
+    def read(agent_id, filename)
+      entry(agent_id, filename.to_s)&.fetch("content", nil)
+    end
+
+    # -> [String] the agent's file names, lexicographic order.
+    def list(agent_id)
+      files(agent_id).keys.sort
+    end
+
+    # -> [String] every agent that has a workspace here (what `doctor` sweeps).
+    def agents = @cs.keys(SCOPE).sort
+
+    # A prompt file is TEXT. `to_s` on a structured value produces Ruby's `#inspect`
+    # and stores it as if it were the prompt: the pilot lost all 11 files of its
+    # production agent that way (someone wrote an ENTRY back as content, so the
+    # markdown reached the model as `{"content" => "…\n…"}` on a single line, escapes
+    # and all, for three weeks). Nothing legitimate passes a Hash or an Array here, so
+    # this is the one place that has to refuse instead of coerce.
+    def self.text!(content, agent_id, filename)
+      return content if content.is_a?(String)
+      return content.to_s unless content.is_a?(Hash) || content.is_a?(Array)
+
+      raise Insika::ValidationError,
+            "content for '#{filename}' (agent '#{agent_id}') must be text, got #{content.class} — " \
+            "pass the file's markdown, not the store entry or a wrapper object"
+    end
+
+    # Writes (upsert). create_only: refuses to overwrite. Versions the previous
+    # content into history. -> Hash (the stored entry).
+    def write(agent_id, filename, content, create_only: false)
+      name = filename.to_s
+      content = self.class.text!(content, agent_id, name)
+      record = @cs.get(SCOPE, agent_id.to_s) || { "files" => {} }
+      record["files"] ||= {}
+      current = record["files"][name]
+      if create_only && current
+        raise Insika::ValidationError, "file '#{name}' already exists for agent '#{agent_id}'"
+      end
+
+      record["files"][name] = build_entry(content.to_s, current)
+      @cs.put(SCOPE, agent_id.to_s, record)
+      record["files"][name]
+    end
+
+    # -> bool (did it exist?). Removes the file; if the agent ends up with no files,
+    # keeps the empty record (cheap; deleting the agent handles the cleanup).
+    def delete(agent_id, filename)
+      name = filename.to_s
+      record = @cs.get(SCOPE, agent_id.to_s)
+      return false unless record && record.dig("files", name)
+
+      record["files"].delete(name)
+      @cs.put(SCOPE, agent_id.to_s, record)
+      true
+    end
+
+    # -> [ { "content" =>, "at" => } ] older versions, most recent first.
+    def versions(agent_id, filename)
+      entry(agent_id, filename.to_s)&.fetch("history", []) || []
+    end
+
+    # Restores version `index` from history as the current content (a new write:
+    # the current one goes to the top of history). -> Hash (entry) | raises if index
+    # invalid / file nonexistent.
+    def restore(agent_id, filename, index)
+      name = filename.to_s
+      hist = versions(agent_id, name)
+      i = Integer(index)
+      unless entry(agent_id, name)
+        raise Insika::NotFoundError, "file '#{name}' not found for agent '#{agent_id}'"
+      end
+      raise Insika::ValidationError, "version #{index} does not exist" if i.negative? || i >= hist.length
+
+      write(agent_id, name, hist[i]["content"])
+    end
+
+    private
+
+    def files(agent_id)
+      (@cs.get(SCOPE, agent_id.to_s) || {})["files"] || {}
+    end
+
+    def entry(agent_id, filename)
+      files(agent_id)[filename]
+    end
+
+    def build_entry(content, current)
+      history = current ? current.fetch("history", []) : []
+      if current
+        history = [{ "content" => current["content"], "at" => current["updated_at"] }] + history
+        history = history.first(HISTORY_MAX)
+      end
+      { "content" => content, "updated_at" => Time.now.utc.iso8601, "history" => history }
+    end
+  end
+end