rcrewai 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '0024080cd05b151b653bcedb4b29afea563e149f9d68dce1662d636a919020a9'
-  data.tar.gz: 53bb6b1376f88d1379cb0f877eeaa35aed0822f08df5f76d482e28d55101efe6
+  metadata.gz: fe3fce4660f7a0c695a2c2f14a7a391cd8e40eaac319af8b87efd80eb118511c
+  data.tar.gz: 383ad7591cee2299424a1e9c75e7b6795d51d30655fcc7bd81f962845ad83712
 SHA512:
-  metadata.gz: b7923adbf161caa5361568268306513c92e5ffbddc3736cd91cd0fbccedd8cdd4230854f8310cf358964c7ccea09b1e43040e944f4c5472b0f5e58983bf37aaf
-  data.tar.gz: e4c9426c4e948fca50c3708a8a0e57588b4b18c51a5ac8df0bd2dd902aa2440a20d8d1f6a464333bb9c3244d5ff6901070645bb932fd5fb6e77f3d3f82ff9f83
+  metadata.gz: 37b9a760ddbaf054b4a0db7d6c60ffa30f5534feb410b71339c2ed5f374d321722fdac22db6d5731cd363d4e11b328d123242bf2effd6cc284768ede5a383e50
+  data.tar.gz: 6f46d935941d7d82399e178254058de3b9455f699f5e091744add873ff6c244e07fb0492f9ffb802123ebdd836b46a48f8283bc19f3f15e8c1ffa2f60f42422d

data/CHANGELOG.md

@@ -7,6 +7,55 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-09-10
+
+Closes the feature-parity gap against CrewAI's `1.x` line. Adds **LLM message
+interceptors**, an **event hierarchy** with safe fan-out, **task-level
+checkpointing** with resume and lineage, and **four new providers** (AWS
+Bedrock, Snowflake Cortex, any OpenAI-compatible endpoint, and OpenAI's
+Responses API).
+
+Almost entirely additive. One behavior change — `Events.fan_out` now serializes
+delivery, so stream subscribers no longer need their own mutex; sinks that
+already lock remain correct. See **Changed** below.
+
+Also repairs `bin/rcrewai`, which had never worked in any published version.
+
+### Added
+
+#### Interceptors & observability
+- LLM message interceptors: `before_request` / `after_response` hooks on every provider client (`LLMClients::Base`), registered via block, callable, or the `before_request:` / `after_response:` constructor kwargs. A `before_request` hook receives `(payload, context)` and may return a replacement payload; an `after_response` hook receives `(result, context)` and may return a replacement result. Returning `nil` keeps the original, so a pure-observer hook needs no return value. `context` carries `:provider` and `:model`, and `after_response` adds `:duration_ms`. Hooks run in registration order, threading the value through. A hook that raises is reported to stderr and skipped — instrumentation never breaks a call. Wired on both the plain and streaming paths of all five providers.
+- Event hierarchy: every `Events::*` event now carries an auto-assigned `:id`, and `:parent_id` naming the enclosing span. `Events.with_parent(id) { ... }` opens a span for the current thread (nesting, restored on exit, and on raise); `Events.emit(sink, event)` stamps the enclosing parent before delivery. Both runners (`ToolRunner`, `LegacyReactRunner`) open a per-run span, so a subscriber can reassemble the flat stream into a tree — what tracing exporters need.
+
+#### Checkpointing
+- Checkpointing: `crew.execute(checkpoint: store)` records durable per-run state, and `crew.resume(run_id)` replays completed tasks instead of re-executing them. Granularity is task-level — a checkpoint is written after each task settles, so a crash loses at most the task in flight. Failed tasks are recorded as failed rather than omitted, so a resume retries them instead of treating them as never-attempted. Supported on the sequential, hierarchical, and consensual processes.
+- Checkpoint stores follow the existing `Flow::StateStore` shape (`save`/`load`/`list`/`delete`): `Checkpoint::MemoryStore` (volatile) and `Checkpoint::FileStore` (one JSON file per run). `FileStore` rejects run ids containing path separators or traversal segments, since ids arrive both from callers and from stored records.
+- Lineage: a resumed run gets its own run id linked to its parent via `parent_run_id`, leaving the original record intact. `Checkpoint.lineage(store, run_id)` walks the chain back to the root, truncating rather than raising if an ancestor has been pruned.
+- CLI: `rcrewai checkpoint list` / `info RUN_ID` / `delete RUN_ID` inspect saved checkpoints (`--dir`, default `.rcrewai/checkpoints`).
+
+#### Providers
+- New providers: `:openai_compatible` (any endpoint speaking the OpenAI Chat Completions format — Together, Groq, Fireworks, vLLM, LiteLLM, OpenRouter, a self-hosted gateway; requires `base_url`), `:bedrock` (AWS Bedrock via the Converse API, giving every Bedrock model one request shape; requires `aws_region`), and `:snowflake` (Snowflake Cortex inference; requires `snowflake_account`). New configuration attributes `aws_region` and `snowflake_account`, also read from `AWS_REGION`/`AWS_DEFAULT_REGION` and `SNOWFLAKE_ACCOUNT`.
+- `:openai_responses` — OpenAI's Responses API alongside the existing Chat Completions client. Messages go under `input` with the system prompt lifted to `instructions`, `max_tokens` becomes `max_output_tokens`, tools are sent flat rather than nested under `function`, and the `output` array is parsed back into the canonical `content` / `tool_calls` shape. An `incomplete` response capped by `max_output_tokens` is reported as `finish_reason: :length`. Non-streaming only — Responses streams a distinct set of semantic events that this client does not model.
+- `LLMClient::PROVIDERS` — provider resolution is now a table rather than a `case`, so registering a client is a one-line change.
+
+
+### Changed
+- **Behavior change:** subscribers passed to `crew.execute(stream:)` no longer need their own mutex. Existing sinks that lock are unaffected.
+
+### Fixed
+- `Events.fan_out` now serializes delivery: sinks are invoked under a mutex, so a sink shared by concurrently executing agents is never entered from two threads at once. Previously it called sinks inline on the emitting thread with no serialization, which under `async: true` meant every subscriber had to do its own locking or race — the 0.7.1 notes documented this as a caveat, but for any aggregating subscriber it was a live defect. The lock is reentrant, so a sink that emits back through the same fan-out does not deadlock. Sinks that already lock internally remain correct.
+- `LLMClient.for_provider` silently dropped interceptor hooks: it constructed each client with only the config, so `before_request` / `after_response` passed through the normal resolution path never reached the client. It now forwards them.
+- `bin/rcrewai` never worked. `lib/rcrewai/cli.rb` defined `def run`, which Thor reserves, so the class raised `"run" is a Thor reserved word` on load; the file was consequently never required from `lib/rcrewai.rb`, which hid the breakage from the test suite while `bin/rcrewai` — shipped as a gem executable since the initial commit — crashed for every installed user. The command is now defined as `run_crew` and mapped back to `run`, so the user-facing invocation (`rcrewai run --crew NAME`) is unchanged, and the CLI is required and covered by specs.
+
+## [0.7.1] - 2026-08-13
+
+### Fixed
+- `Crew#execute` built its event sink but never delivered it to agents, so no agent-level events (`IterationStart`/`IterationEnd`, tool calls, token usage) reached a sink passed to `crew.execute(stream:)`. `Task` now carries a `stream_sink` that `Crew` populates at execute time and passes to `Agent#execute_task`; the two `Process` call sites that bypass `Task#execute` read `crew.stream_sink` directly. Subscribers now receive the full event stream on the sync, async, hierarchical, and consensual paths.
+- Tasks retained a reference to the caller's sink after `execute` returned, keeping request-scoped subscribers reachable for the lifetime of the task object. The sink is now cleared in an `ensure`.
+
+### Note
+- `Events.fan_out` invokes sinks inline on the emitting thread with no serialization, so under `async: true` a sink may be called concurrently from multiple worker threads. Subscribers must do their own locking. **Superseded in `0.8.0`:** fan-out now serializes delivery.
+
 ## [0.7.0] - 2026-07-07
 
 Turns the `:consensual` crew process from a stub into a real multi-agent
@@ -217,7 +266,8 @@ output, guardrails, planning, and training/testing. See `ROADMAP.md`.
 - CLI usage documentation
 - Real-world use cases and examples
 
-[Unreleased]: https://github.com/gkosmo/rcrewAI/compare/v0.7.0...HEAD
+[Unreleased]: https://github.com/gkosmo/rcrewAI/compare/v0.7.1...HEAD
+[0.7.1]: https://github.com/gkosmo/rcrewAI/compare/v0.7.0...v0.7.1
 [0.7.0]: https://github.com/gkosmo/rcrewAI/compare/v0.6.1...v0.7.0
 [0.6.1]: https://github.com/gkosmo/rcrewAI/compare/v0.6.0...v0.6.1
 [0.6.0]: https://github.com/gkosmo/rcrewAI/compare/v0.5.0...v0.6.0

data/ROADMAP.md

@@ -5,85 +5,154 @@ This roadmap tracks feature parity between **RCrewAI** (Ruby) and the upstream
 
 ## Current status
 
-- **RCrewAI:** `0.3.0` (2026-05-12)
-- **Upstream crewai:** `1.15.x` (mid-2026)
+- **RCrewAI:** `0.7.1` released; `0.8.0`, `0.9.0` and `0.9.x` merged to `main`, unreleased
+- **Upstream crewai:** `1.15.21`
 
 RCrewAI is a faithful port of CrewAI's **"Crews"** mental model (Agents / Tasks /
-Crew, sequential + hierarchical processes, tools, memory, human-in-the-loop). As
-of `0.3.0` the LLM plumbing is modern: native function calling across all five
-providers, a tool-schema DSL, typed streaming events, MCP client, and per-model
-pricing.
+Crew, sequential + hierarchical + consensual processes, tools, memory,
+human-in-the-loop), and it carries CrewAI's second pillar (**Flows**) plus
+**Knowledge (RAG)**, **guardrails**, **structured output**, **planning**, and
+**training/testing**. In one area — cognitive memory (semantic recall, SQLite
+persistence, four memory types) — the gem went past what was originally ported.
+
+**Status: one milestone remaining.** An earlier revision of this file declared
+parity "complete" against a matrix that only covered CrewAI through roughly
+`1.0` (October 2025) while quoting `1.15.x` in its header; everything upstream
+added across `1.1`–`1.15` was unmeasured. That delta was re-derived, and three
+of the four scheduled milestones have since shipped. Only native async (1.0.0)
+is outstanding, and it is blocked on an open decision — see below.
 
-Since CrewAI's `1.0`, the framework grew a second pillar (**Flows**) plus
-**Knowledge (RAG)**, **Guardrails**, **structured output**, **Planning**, and
-**Training/Testing**. RCrewAI now implements all of these — see the matrix below.
+## Parity matrix
 
-**Status: complete.** All milestone issues (#5–#12) shipped in v0.4.0, and all
-backlog items (#15–#20) shipped in v0.5.0. There is no outstanding roadmap work.
+### Shipped
+
+| Concept | crewai | RCrewAI |
+|---|---|---|
+| Agents / Tasks / Crew | ✅ | ✅ |
+| Sequential / hierarchical process | ✅ | ✅ |
+| Consensual process (propose → vote → pick) | ✅ | ✅ (0.7.0) |
+| Native function calling + tool DSL | ✅ | ✅ (0.3.0) |
+| Streaming events | ✅ | ✅ (0.3.0) |
+| MCP client | ✅ | ✅ (0.3.0) |
+| Per-model pricing / cost | ✅ | ✅ (0.3.0) |
+| Per-agent LLM override | ✅ | ✅ (0.4.0) |
+| Structured output (schema) | ✅ | ✅ (0.4.0) |
+| Task guardrails | ✅ | ✅ (0.4.0) |
+| `output_file` / markdown | ✅ | ✅ (0.4.0) |
+| Knowledge / RAG | ✅ | ✅ (0.4.0) |
+| Planning | ✅ | ✅ (0.4.0) |
+| Flows (`start`/`listen`/`router`) | ✅ | ✅ (0.4.0) |
+| Flow state + persistence | ✅ | ✅ (0.4.0) |
+| Training / testing | ✅ | ✅ (0.4.0) |
+| Lifecycle hooks, batch kickoff, rate limiting | ✅ | ✅ (0.5.0) |
+| Reasoning, context window, multimodal | ✅ | ✅ (0.5.0) |
+| Cognitive memory (semantic, persistent, typed) | ✅ | ✅ (0.6.x) |
+| LLM message interceptor hooks | ✅ | ✅ (0.8.0) |
+| Event hierarchy + safe fan-out | ✅ | ✅ (0.8.0) |
+| Checkpointing (save / resume / lineage) | ✅ | ✅ (0.9.0) |
+| Newer providers (Bedrock, Cortex, OpenAI-compatible) | ✅ | ✅ (0.9.x) |
+| OpenAI Responses API | ✅ | ✅ (0.9.x) |
+
+### Gaps
+
+| Concept | crewai | RCrewAI | Plan |
+|---|---|---|---|
+| Native async (LLM + tool level) | ✅ (1.4–1.6) | ⚠️ partial | 1.0.0 |
+| OTel export for the event hierarchy | ✅ (1.10+) | ❌ | 1.0.x |
+| Streaming for Bedrock / Responses | ✅ | ❌ | 1.0.x |
+| Bedrock SigV4 signing | ✅ | ❌ (hook workaround) | 1.0.x |
+| A2A (agent-to-agent) | ✅ (1.7–1.9) | ❌ | deferred |
 
-## Parity matrix
+### Out of scope
+
+Three upstream areas are deliberately **not** targets. They are CrewAI's
+commercial platform surface rather than framework capability, and porting them
+means tracking someone else's product roadmap with no Ruby-side consumer:
+
+- **JSON-first project format** (`agents/*.jsonc`, `crew.jsonc`, declarative and
+  conversational flows in the CLI TUI) — a config format whose shape is set by
+  upstream's tooling.
+- **Skills Repository** — a hosted registry plus authentication.
+- **Policies** — CrewAI enforces these at the infrastructure level via the NVIDIA
+  OpenShell runtime. Reimplementing them in-process would provide the appearance
+  of enforcement without the property that makes enforcement worth having.
+
+If a Ruby-side need for any of these appears, revisit — but not speculatively.
+
+## Milestones
+
+### 0.8.0 — Interceptors & observability ✅ shipped (#39)
+
+`before_request` / `after_response` hooks on `LLMClients::Base`, inherited by
+every provider and wired on both the plain and streaming paths. Events gained
+`:id` / `:parent_id` with `Events.with_parent` spans opened per agent run.
+
+`Events.fan_out` now serializes delivery under a reentrant mutex, fixing a live
+race: it previously called sinks inline on the emitting thread, so under
+`async: true` an aggregating subscriber was entered from several pool workers at
+once. **Behavior change:** subscribers no longer need their own mutex.
+
+### 0.9.0 — Checkpointing ✅ shipped (#42)
+
+Task-level `crew.execute(checkpoint: store)` / `crew.resume(run_id)` across the
+sequential, hierarchical and consensual processes, with `MemoryStore` and
+`FileStore` following the `Flow::StateStore` shape. Resumed runs link to their
+parent via `parent_run_id`; `Checkpoint.lineage` walks the chain to the root.
+CLI: `rcrewai checkpoint list|info|delete`.
+
+Also repaired `bin/rcrewai`, which had never worked: `lib/rcrewai/cli.rb`
+defined `def run`, a Thor reserved word, so the class raised on load and the
+file was never required — hiding the breakage from the suite while the shipped
+gem executable crashed for every installed user.
+
+### 0.9.x — Providers & Responses API ✅ shipped (#41)
+
+`:openai_compatible`, `:bedrock` (Converse v4), `:snowflake` (Cortex) and
+`:openai_responses`. Provider resolution moved to a `LLMClient::PROVIDERS`
+table, which also fixed `for_provider` silently dropping interceptor hooks.
+
+Two deliberate limitations carried forward to 1.0.x: Bedrock does not implement
+SigV4 (a hard `aws-sigv4` dependency for one provider is not worth it; sign via
+a `before_request` hook), and Bedrock/Responses are non-streaming only.
+
+### 1.0.0 — Native async
+
+The largest item, and the one with a genuine architecture decision attached.
+Today `AsyncExecutor` fans tasks out across a `Concurrent::ThreadPoolExecutor` in
+dependency-ordered phases; concurrency stops at the task boundary. CrewAI went
+async *through* the LLM and tool calls (1.4–1.6), covering flows, crews, tasks,
+knowledge, and memory.
+
+Ruby has no direct port of that. Two candidate models:
+
+1. **Fibers** via the `async` gem — closer to upstream's shape, new runtime
+   dependency, and every provider client's HTTP layer has to cooperate.
+2. **Stay on threads** and make the client layer non-blocking — smaller
+   conceptual change, keeps `concurrent-ruby`, less faithful to upstream.
+
+**This decision is open and blocks the milestone.** It touches all five LLM
+clients either way. The 8.2k lines of `lib/` are backed by 4.9k lines of spec,
+which is what makes a refactor at this depth tractable.
+
+### Deferred — A2A
+
+Agent-to-agent task execution utilities and server configuration (upstream
+1.7–1.9). Real framework capability, but it presumes a deployment topology that
+no current RCrewAI user has asked for. Revisit once the items above land.
+
+## Sequencing
 
-| Concept | crewai | RCrewAI 0.3.0 | Target |
+| Milestone | Contents | Risk | Status |
 |---|---|---|---|
-| Agents / Tasks / Crew | ✅ | ✅ | — |
-| Sequential / hierarchical process | ✅ | ✅ | — |
-| Native function calling + tool DSL | ✅ | ✅ (0.3.0) | — |
-| Streaming events | ✅ | ✅ (0.3.0) | — |
-| MCP client | ✅ | ✅ (0.3.0) | — |
-| Per-model pricing / cost | ✅ | ✅ (0.3.0) | — |
-| Per-agent LLM override | ✅ | ✅ (#5) | ✅ done |
-| Structured output (schema) | ✅ | ✅ (#6) | ✅ done |
-| Task guardrails | ✅ | ✅ (#7) | ✅ done |
-| `output_file` / markdown | ✅ | ✅ (#8) | ✅ done |
-| Knowledge / RAG | ✅ | ✅ (#9) | ✅ done |
-| Planning | ✅ | ✅ (#10) | ✅ done |
-| Flows (`start`/`listen`/`router`) | ✅ | ✅ (#11) | ✅ done |
-| Flow state + persistence | ✅ | ✅ (#11) | ✅ done |
-| Training / Testing | ✅ | ✅ (#12) | ✅ done |
-| Reasoning, rate-limiting, batch kickoff, hooks, context window, multimodal | ✅ | ✅ (#15–#20) | ✅ done |
-
-## Milestones (highest leverage first)
-
-### 0.3.1 — Per-agent LLM override
-Let `Agent.new(llm:)` accept a provider/model, instead of only the global
-`RCrewAI.configure`. Unblocks mixed-model crews (cheap model for workers, strong
-model for the manager).
-
-### 0.4.0 — Structured output & guardrails
-Builds directly on the 0.3.0 tool-schema/JSON-schema plumbing.
-- `Task.new(output_schema:)` → validated, coerced structured result.
-- `Task.new(guardrail:)` → proc/object that validates & transforms output, with
-  bounded retries (`guardrail_max_retries`).
-- `output_file:` + `markdown:` output formatting.
-
-### 0.5.0 — Knowledge (RAG) & Planning
-- Knowledge sources: string, `.txt`, PDF (have `pdf-reader`), CSV, JSON, URL
-  (have `nokogiri`). Embeddings client + a pluggable vector store (start with an
-  in-memory / SQLite cosine store; no hard Chroma dependency).
-- Attach at agent **and** crew level.
-- `Crew.new(planning: true)` → a planner pass that drafts a step plan before
-  execution.
-
-### 0.6.0 — Flows
-The flagship. A Ruby DSL mirroring CrewAI Flows:
-- `start`, `listen`, `router` decorators/class-methods.
-- `and_` / `or_` trigger combinators.
-- Structured flow **state** (a plain struct/`Data` or dry-struct) with a UUID.
-- `@persist`-equivalent state persistence across restarts.
-- `human_feedback` pause/resume point.
-
-### 0.7.0 — Training & Testing
-- `crew.train(n_iterations:, filename:)` capturing human feedback.
-- `crew.test(n_iterations:, model:)` scoring runs.
-
-### Backlog — ✅ all complete
-
-Formerly polish items with no set version; all shipped in the `[Unreleased]`
-changes (see CHANGELOG):
-
-- [#15](https://github.com/gkosmo/rcrewAI/issues/15) — `before_kickoff` / `after_kickoff` lifecycle hooks ✅
-- [#16](https://github.com/gkosmo/rcrewAI/issues/16) — `kickoff_for_each` batch execution ✅
-- [#17](https://github.com/gkosmo/rcrewAI/issues/17) — `max_rpm` rate limiting ✅
-- [#18](https://github.com/gkosmo/rcrewAI/issues/18) — per-agent reasoning (`reasoning:`, `max_reasoning_attempts:`) ✅
-- [#19](https://github.com/gkosmo/rcrewAI/issues/19) — `respect_context_window` history trimming ✅
-- [#20](https://github.com/gkosmo/rcrewAI/issues/20) — multimodal agents (image/file inputs) ✅
+| 0.8.0 | Interceptors + observability | Low | ✅ merged (#39) |
+| 0.9.0 | Checkpointing | Moderate | ✅ merged (#42) |
+| 0.9.x | Providers, Responses API | Low | ✅ merged (#41) |
+| 1.0.0 | Native async | High — decision open | ⏳ blocked |
+
+The three shipped milestones are on `main` and unreleased; they want a version
+bump and a release before or alongside 1.0.0 work.
+
+**1.0.0 is blocked on the fibers-vs-threads decision above.** It is the only
+remaining scheduled work, and the largest single change in this roadmap: it
+touches all nine provider clients and the executor. Nothing else should start
+before that call is made.

data/docs/api/agent.md

@@ -28,6 +28,14 @@ Creates a new agent instance.
 - `human_input` (Boolean, optional) - Enable human-in-the-loop interactions (default: false)
 - `require_approval_for_tools` (Boolean, optional) - Require human approval for tool usage
 - `require_approval_for_final_answer` (Boolean, optional) - Require human approval for final results
+- `llm` (Symbol \| Hash \| client, optional) - Per-agent LLM override: a provider symbol (`:anthropic`), an options hash (`{ provider:, model:, api_key:, temperature: }`), or a pre-built client. Defaults to the global configuration. See [Advanced Agent Options]({{ site.baseurl }}/tutorials/agent-options)
+- `reasoning` (Boolean, optional) - Run a reasoning/planning pass before answering (default: false); surfaced on the result as `:reasoning`
+- `max_reasoning_attempts` (Integer, optional) - Reasoning retries on empty output (default: 3)
+- `respect_context_window` (Boolean, optional) - Trim history to fit the model's context window (default: false)
+- `max_rpm` (Integer, optional) - Throttle the agent's LLM calls to this many requests per minute (default: unlimited)
+- `knowledge` (Knowledge::Base, optional) - A knowledge base to ground the agent (see [Knowledge (RAG)]({{ site.baseurl }}/tutorials/knowledge))
+- `knowledge_sources` (Array, optional) - Knowledge sources the agent wraps in a base
+- `memory` (Memory \| Hash, optional) - A pre-built `Memory`, or options (`{ embedder:, store:, scope:, short_term_limit:, entity_extractor: }`); defaults to zero-config in-memory (see [Cognitive Memory]({{ site.baseurl }}/tutorials/memory))
 
 **Returns:** `RCrewAI::Agent` instance
 

data/docs/api/crew.md

@@ -10,18 +10,26 @@ The Crew class is the main orchestrator in RCrewAI. It manages a collection of a
 
 ## Class Methods
 
-### `.new(name)`
+### `.new(name, **options)`
 
 Creates a new crew instance.
 
 **Parameters:**
 - `name` (String) - The name of the crew
+- `process` (Symbol, optional) - `:sequential` (default), `:hierarchical`, or `:consensual`
+- `consensus_agents` (Integer, optional) - For `:consensual`, how many agents propose and vote per task (default: 3). See [Consensual Process]({{ site.baseurl }}/tutorials/consensual-process)
+- `planning` (Boolean, optional) - Run a planner pass that drafts a per-task plan before execution (default: false)
+- `planning_llm` (Symbol \| Hash \| client, optional) - The planner's LLM (defaults to the global provider)
+- `knowledge` (Knowledge::Base, optional) - A knowledge base shared with all agents
+- `knowledge_sources` (Array, optional) - Sources the crew wraps in a shared base (see [Knowledge (RAG)]({{ site.baseurl }}/tutorials/knowledge))
+- `verbose` (Boolean, optional) - Detailed logging (default: false)
+- `max_iterations` (Integer, optional) - Max iterations per agent (default: 10)
 
 **Returns:** `RCrewAI::Crew` instance
 
 **Example:**
 ```ruby
-crew = RCrewAI::Crew.new("research_team")
+crew = RCrewAI::Crew.new("research_team", process: :consensual, consensus_agents: 3)
 ```
 
 ### `.create(name)`
@@ -342,4 +350,46 @@ crew.add_agent(manager)
 # Add other agents...
 
 crew.execute
-```
+```
+
+### Consensual Process
+
+Agents propose competing answers and vote to pick the best (see the
+[Consensual Process]({{ site.baseurl }}/tutorials/consensual-process) tutorial).
+
+```ruby
+crew = RCrewAI::Crew.new("panel", process: :consensual, consensus_agents: 3)
+crew.add_agent(junior)
+crew.add_agent(senior)
+crew.add_task(task)
+
+crew.execute   # each task: propose → vote → pick
+```
+
+## Lifecycle, Batch, Training
+
+### `#before_kickoff { |inputs| ... }` / `#after_kickoff { |result| ... }`
+
+Register callbacks that run before/after execution. A `before_kickoff` hook
+receives the inputs hash (from `execute(inputs:)`) and may transform it; an
+`after_kickoff` hook receives and may transform the result. The resolved inputs
+are exposed on `#last_inputs`.
+
+### `#kickoff_for_each(inputs:)`
+
+Runs the crew once per input set, returning one result per input in order. Runs
+are isolated to their own inputs.
+
+```ruby
+results = crew.kickoff_for_each(inputs: [{ topic: "ruby" }, { topic: "python" }])
+```
+
+### `#train(n_iterations:, filename:, feedback: nil)`
+
+Runs the crew repeatedly, collects feedback after each run (via a `feedback:`
+callable, defaulting to a human prompt), and persists it to JSON.
+
+### `#test(n_iterations:, scorer: nil)`
+
+Runs the crew repeatedly and reports per-run and average scores (via a `scorer:`
+callable, defaulting to the run's success rate).

data/docs/api/index.md

@@ -19,23 +19,24 @@ Individual AI agents with specific roles and capabilities.
 ### [RCrewAI::Task]({{ site.baseurl }}/api/task)
 Tasks that agents execute to achieve goals.
 
-## Tools
+## Capabilities (0.4 – 0.7)
+
+These subsystems are covered in depth by the tutorials (with runnable examples):
 
-### [RCrewAI::Tools::WebSearch]({{ site.baseurl }}/api/tools/web-search)
-Enable agents to search the web for information.
+- **`RCrewAI::Flow`** — event-driven workflows. See [Flows]({{ site.baseurl }}/tutorials/flows)
+- **`RCrewAI::Knowledge::Base`** and sources/embedders — RAG. See [Knowledge]({{ site.baseurl }}/tutorials/knowledge)
+- **`RCrewAI::Memory`** (+ `SqliteStore`, memory types) — cognitive memory. See [Memory]({{ site.baseurl }}/tutorials/memory)
+- **Consensual process** — multi-agent voting. See [Consensual Process]({{ site.baseurl }}/tutorials/consensual-process)
+- **Advanced agent/task options** — per-agent LLM, reasoning, rate limiting, context window, multimodal, structured output, guardrails, hooks. See [Advanced Options]({{ site.baseurl }}/tutorials/agent-options)
 
-### [RCrewAI::Tools::FileSystem]({{ site.baseurl }}/api/tools/file-system)
-Allow agents to read and write files.
+## Tools
 
-### [RCrewAI::Tools::Database]({{ site.baseurl }}/api/tools/database)
-Connect agents to databases for data operations.
+See the [Tools System]({{ site.baseurl }}/api/tools) reference for the full API.
+Built-in tools include `WebSearch`, `FileReader`, `FileWriter`, `SqlDatabase`,
+`EmailSender`, `PdfProcessor`, and `CodeExecutor` — plus native function calling
+and MCP servers.
 
 ## Configuration
 
 ### [RCrewAI::Configuration]({{ site.baseurl }}/api/configuration)
-Configure LLM providers and other settings.
-
-## CLI
-
-### [RCrewAI::CLI]({{ site.baseurl }}/api/cli)
-Command-line interface for managing crews.
+Configure LLM providers, models, and other settings.

data/docs/api/task.md

@@ -32,6 +32,13 @@ Creates a new task instance.
 - `allow_guidance` (Boolean) - Allow human guidance during execution
 - `human_review_points` (Array) - Points where human review is requested ([:completion, :error])
 - `async` (Boolean) - Whether task can be executed asynchronously (default: false)
+- `output_schema` (Hash) - A JSON-schema subset the output is validated/coerced against; parsed result on `#structured_output` (raw string on `#raw_result`). Non-conforming output re-runs the agent with the error fed back
+- `guardrail` (callable) - `->(output) { [ok, value_or_error] }` to validate/transform output before it flows downstream
+- `guardrail_max_retries` (Integer) - Guardrail retries with the reason fed back (default: 3)
+- `output_file` (String) - Path to write the result to after completion
+- `create_directory` (Boolean) - Create the output file's parent dirs (default: true)
+- `markdown` (Boolean) - Prepend a heading when the output isn't already markdown (default: false)
+- `attachments` (Array) - Multimodal image inputs, e.g. `[{ type: :image, path: 'x.png' }]` or `{ type: :image, url: '...' }` (OpenAI/Azure). See [Advanced Agent Options]({{ site.baseurl }}/tutorials/agent-options)
 
 **Returns:** `RCrewAI::Task` instance
 

data/docs/examples/async-execution.md

@@ -30,9 +30,10 @@ require 'benchmark'
 RCrewAI.configure do |config|
   config.llm_provider = :openai
   config.temperature = 0.4
-  config.max_concurrent_tasks = 8  # Allow up to 8 concurrent tasks
-  config.task_timeout = 300        # 5-minute timeout per task
+  config.timeout = 300             # request timeout (seconds)
 end
+# Concurrency is a per-run option, not global config:
+#   crew.execute(async: true, max_concurrency: 8)
 
 # ===== CONCURRENT PROCESSING TOOLS =====
 
@@ -885,9 +886,11 @@ context: []  # Dependency management continues with available results
 Designed for horizontal and vertical scaling:
 
 ```ruby
-config.max_concurrent_tasks = 8    # Configurable concurrency
-config.task_timeout = 300          # Timeout management  
-config.resource_pool_size = 16     # Shared resource scaling
+# Concurrency is controlled per run via the async executor:
+crew.execute(async: true, max_concurrency: 8)   # up to 8 concurrent tasks
+
+# Request timeout is global config:
+RCrewAI.configure { |c| c.timeout = 300 }
 ```
 
 This concurrent processing system provides a complete framework for optimizing performance through intelligent parallel execution while maintaining reliability and quality standards across all processing streams.

data/docs/examples/tool-composition.md

@@ -25,12 +25,12 @@ require 'time'
 
 # Configure RCrewAI
 RCrewAI.configure do |config|
-  config.llm_client = :openai
+  config.llm_provider = :openai
   config.openai_api_key = ENV['OPENAI_API_KEY']
   config.log_level = :info
-  config.max_concurrent_tasks = 6
-  config.task_timeout = 300
+  config.timeout = 300              # request timeout (seconds)
 end
+# Concurrency is set per run: crew.execute(async: true, max_concurrency: 6)
 
 # Base tool for common functionality
 class CompositeToolBase < RCrewAI::Tools::Base

data/docs/index.md

@@ -19,16 +19,31 @@ Build powerful AI agent crews in Ruby that work together to accomplish complex t
 ## Features
 
 - **🤖 Intelligent Agents**: AI agents with reasoning loops, memory, and tool usage capabilities
-- **🔗 Multi-LLM Support**: OpenAI, Anthropic (Claude), Google (Gemini), Azure OpenAI, and Ollama
-- **🛠️ Rich Tool Ecosystem**: Web search, file operations, SQL, email, code execution, PDF processing, and custom tools
-- **🧠 Agent Memory**: Short-term and long-term memory for learning from past executions
+- **🔗 Multi-LLM Support**: OpenAI, Anthropic (Claude), Google (Gemini), Azure OpenAI, and Ollama — configurable per agent
+- **🛠️ Rich Tool Ecosystem**: Native function calling, MCP servers, web search, file operations, SQL, email, code execution, PDF processing, and custom tools
+- **🌊 Flows**: Event-driven workflows (`start`/`listen`/`router`) with branching and persistent state
+- **📚 Knowledge (RAG)**: Ground agents in your own documents (string/file/PDF/CSV/URL) with built-in retrieval
+- **🧠 Cognitive Memory**: Semantic recall (embeddings + cosine) with optional SQLite persistence and short-term/long-term/entity/tool memory types
+- **📤 Structured Output & Guardrails**: Schema-validated task output and validate/transform guardrails
+- **🗳️ Flexible Orchestration**: Sequential, hierarchical, and consensual (propose → vote → pick) processes, plus async execution
 - **🤝 Human-in-the-Loop**: Interactive approval workflows, human guidance, and collaborative decision making
-- **⚡ Advanced Task System**: Dependencies, retries, async/concurrent execution, and context sharing
-- **🏗️ Hierarchical Teams**: Manager agents that coordinate and delegate tasks to specialist agents
-- **🔒 Production Ready**: Security controls, error handling, logging, monitoring, and sandboxing
-- **🎯 Flexible Orchestration**: Sequential, hierarchical, and concurrent execution modes
+- **⚡ Advanced Task System**: Dependencies, retries, async/concurrent execution, planning, and context sharing
+- **🎛️ Production Controls**: Rate limiting (`max_rpm`), context-window management, reasoning passes, multimodal input, and streaming events with cost tracking
 - **💎 Ruby-First Design**: Built specifically for Ruby developers with idiomatic patterns
 
+## What's new (0.4 – 0.7)
+
+RCrewAI has expanded well beyond the classic crew model. Recent releases added:
+
+- **[Flows]({{ site.baseurl }}/tutorials/flows)** — a second orchestration pillar for event-driven, stateful workflows
+- **[Knowledge / RAG]({{ site.baseurl }}/tutorials/knowledge)** — retrieval-augmented grounding from your own documents
+- **[Cognitive Memory]({{ site.baseurl }}/tutorials/memory)** — semantic, persistent, multi-type agent memory
+- **[Consensual process]({{ site.baseurl }}/tutorials/consensual-process)** — multi-agent voting to pick the best answer
+- **[Advanced agent options]({{ site.baseurl }}/tutorials/agent-options)** — per-agent LLM, reasoning, rate limiting, context-window management, multimodal, structured output, guardrails, and lifecycle hooks
+
+See the [CHANGELOG](https://github.com/gkosmo/rcrewAI/blob/main/CHANGELOG.md) and the
+per-release upgrade guides (e.g. [upgrading to 0.7](https://github.com/gkosmo/rcrewAI/blob/main/docs/upgrading-to-0.7.md)) for details.
+
 ## Quick Start
 
 ### Basic Agent Collaboration