rcrewai 0.4.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f742fcf06518c80cbf29191d0eff275830ee618c9b896c3d443258b06aa120ff
-  data.tar.gz: 5cc9c03182d0bd80d73d99edb08f2267252684f64a23da1626d2d61e0abe1683
+  metadata.gz: fb90e26000ddf4a2907b06150fdc87b1707fc701c6c61ba35db33339ed6f38d4
+  data.tar.gz: ba3435eb1918be328ce65faf2c3000553833b09778eb3b06f2a6191cae31aab4
 SHA512:
-  metadata.gz: 0c32e4110a5bbabac9b120262e2ebe0e86b0c58c3b4213a11214efcff647024d9714fcee12934f4b734c9f64184e1f802d82b97ec50e5dc7ceab424c4cec38f1
-  data.tar.gz: 8a678ae53008c7c6a73cf365b460b80989da05f22eb646cf3a9126028a0d6dd220e6af75ef4dee80c02d73e69b536fb3d2b7c23f827e929078b76ed450d8f1ce
+  metadata.gz: 992ac3d7368cb7690f11294b78fe876fcea29d48cffd001a355471a351c22c321563fbd390a0688951be6bcbd0f177322509e59cdd19cf8073e228784d96b503
+  data.tar.gz: 92e7db7609d4503693396f85735fd48633cb5204a34d2431e57ffcb7c758e693bff5b1282ed716899f4242bd9a26b0483f278c991dedb13b21d05894a59b7105

data/CHANGELOG.md

@@ -7,6 +7,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-07-06
+
+Replaces the placeholder agent memory with a **cognitive memory** system —
+semantic recall (embeddings + cosine), optional SQLite persistence, and four
+memory types. This is the one area where CrewAI advanced past what the gem
+originally ported. Backward compatible: the `Memory` public API is unchanged and
+the zero-config default behaves as before.
+
+### Added
+- Cognitive memory: `RCrewAI::Memory` is now a semantic, optionally-persistent, multi-type memory system, replacing the previous word-overlap placeholder. It composes four memory types — `ShortTermMemory` (recent executions, capped), `LongTermMemory` (durable, deduped insights), `EntityMemory` (facts about entities seen in work), and `ToolMemory` (tool-call history) — behind the original `Memory` public API.
+- Semantic recall: pass `Agent.new(memory: { embedder: RCrewAI::Knowledge::Embedder.new })` to recall conceptually related past work via embeddings + cosine similarity. Without an embedder, recall falls back to lexical (word-overlap) similarity — and embedding failures fall back gracefully, so memory never breaks execution.
+- Persistent memory: `RCrewAI::Memory::SqliteStore` persists memories to SQLite (vectors packed as floats, metadata as JSON) so recall survives restarts. Default store is `InMemoryStore` (volatile). The store interface is pluggable.
+- Memory scoping: each agent's memory is scoped to its name by default, so agents sharing a persistent store don't cross-read; override via `memory: { scope: ... }`.
+- `RCrewAI::Similarity` — shared cosine + lexical similarity helpers (extracted from `Knowledge::Store`, now used by both Knowledge and Memory).
+- `sqlite3` added as a runtime dependency (required lazily; only needed for `SqliteStore`).
+
+### Changed
+- `Agent.new(memory:)` accepts a pre-built `Memory`, an options hash (`{ embedder:, store:, scope:, short_term_limit: }`), or nothing (zero-config default, unchanged behavior).
+
+## [0.5.0] - 2026-07-03
+
+Polish release completing the roadmap backlog: crew lifecycle hooks, batch
+execution, rate limiting, per-agent reasoning, context-window management, and
+multimodal image input. All additive — existing code runs unchanged.
+
+### Added
+- Crew lifecycle hooks: `Crew#before_kickoff` and `Crew#after_kickoff` register callbacks that run before/after execution. A `before_kickoff` hook receives the inputs hash (passed via `crew.execute(inputs:)`) and may transform it; an `after_kickoff` hook receives the result and may transform it. Multiple hooks run in registration order. The (possibly transformed) inputs are exposed on `Crew#last_inputs`. (#15)
+- `Crew#kickoff_for_each(inputs:)` runs the crew once per input set and returns one result per input, in order. Runs are isolated — each execution starts from only its own inputs. (#16)
+- Rate limiting: `Agent.new(max_rpm:)` throttles the agent's LLM calls to the given requests-per-minute using a thread-safe rolling-window `RCrewAI::RateLimiter`. The agent's client is transparently wrapped (`RateLimiter::ThrottledClient`) so every `chat` acquires a slot first; `max_rpm` nil/0 means unlimited. (#17)
+- Reasoning: `Agent.new(reasoning: true)` runs a reasoning/planning pass before answering — the LLM drafts a short plan for the task, which is injected into the answer prompt and surfaced on the result hash as `:reasoning` (without polluting `task.result`). Bounded by `max_reasoning_attempts` (default 3), retrying if the model returns empty output; if every attempt is empty, execution proceeds without a plan. Off by default. (#18)
+- Context-window management: `Agent.new(respect_context_window: true)` trims the message history to fit the model's context window before each LLM call, dropping the oldest non-system messages while always keeping system messages and the latest message. The new `RCrewAI::ContextWindow` module provides the token estimate (chars/4 heuristic), a per-model window-size table, and the `fit` trimmer. Off by default. (#19)
+- Multimodal input: `Task.new(attachments:)` accepts image attachments (`{ type: :image, url: '...' }` or `{ type: :image, path: '...' }`). When a task has attachments, the agent builds an OpenAI-style multimodal user message (text + `image_url` parts); local files are base64-encoded into data URLs with a mime type inferred from the extension. Supported on OpenAI/Azure; other providers raise a clear `Multimodal::UnsupportedProviderError`. The new `RCrewAI::Multimodal` module builds the content parts. (#20)
+
 ## [0.4.0] - 2026-07-03
 
 This release closes the feature-parity gap with the modern CrewAI framework,
@@ -165,7 +198,9 @@ output, guardrails, planning, and training/testing. See `ROADMAP.md`.
 - CLI usage documentation
 - Real-world use cases and examples
 
-[Unreleased]: https://github.com/gkosmo/rcrewAI/compare/v0.4.0...HEAD
+[Unreleased]: https://github.com/gkosmo/rcrewAI/compare/v0.6.0...HEAD
+[0.6.0]: https://github.com/gkosmo/rcrewAI/compare/v0.5.0...v0.6.0
+[0.5.0]: https://github.com/gkosmo/rcrewAI/compare/v0.4.0...v0.5.0
 [0.4.0]: https://github.com/gkosmo/rcrewAI/compare/v0.3.0...v0.4.0
 [0.3.0]: https://github.com/gkosmo/rcrewAI/compare/v0.1.0...v0.3.0
 [0.1.0]: https://github.com/gkosmo/rcrewAI/releases/tag/v0.1.0

data/README.md

@@ -13,7 +13,7 @@ RCrewAI is a Ruby implementation of the CrewAI framework, allowing you to create
 - **🤖 Intelligent Agents**: AI agents with reasoning loops, memory, and tool usage capabilities
 - **🔗 Multi-LLM Support**: OpenAI, Anthropic (Claude), Google (Gemini), Azure OpenAI, and Ollama
 - **🛠️ Rich Tool Ecosystem**: Web search, file operations, SQL, email, code execution, PDF processing, and custom tools
-- **🧠 Agent Memory**: Short-term and long-term memory for learning from past executions
+- **🧠 Cognitive Memory**: Semantic recall (embeddings + cosine) with optional SQLite persistence and short-term/long-term/entity/tool memory types
 - **🤝 Human-in-the-Loop**: Interactive approval workflows, human guidance, and collaborative decision making
 - **⚡ Advanced Task System**: Dependencies, retries, async/concurrent execution, and context sharing
 - **🏗️ Hierarchical Teams**: Manager agents that coordinate and delegate tasks to specialist agents
@@ -263,6 +263,88 @@ crew.test(n_iterations: 5)
 # => { iterations: 5, scores: [...], average_score: 92.0 }
 ```
 
+## 🪝 Kickoff Hooks & Batch Runs
+
+Run setup/teardown around a crew, and batch it over many inputs:
+
+```ruby
+crew.before_kickoff { |inputs| inputs.merge(started_at: Time.now) } # may transform inputs
+crew.after_kickoff  { |result| notify(result); result }            # may transform result
+
+crew.execute(inputs: { topic: 'ruby' })
+crew.last_inputs   # => the (possibly transformed) inputs the run used
+
+# Batch: run the crew once per input set, results returned in order.
+results = crew.kickoff_for_each(inputs: [
+  { topic: 'ruby' },
+  { topic: 'python' }
+])
+```
+
+## ⏱️ Rate Limiting
+
+Cap an agent's LLM calls to stay under provider limits. Calls beyond the cap
+block until the rolling 60-second window frees up:
+
+```ruby
+agent = RCrewAI::Agent.new(name: 'a', role: '...', goal: '...', max_rpm: 20)
+```
+
+The limiter (`RCrewAI::RateLimiter`) is thread-safe, so it holds under async
+execution. `max_rpm: nil` (the default) or `0` means unlimited.
+
+## 🧠 Reasoning
+
+Have an agent think through a plan before answering. The reasoning trace is
+surfaced on the result and does not pollute `task.result`:
+
+```ruby
+agent = RCrewAI::Agent.new(name: 'a', role: '...', goal: '...',
+                           reasoning: true, max_reasoning_attempts: 3)
+
+result = agent.execute_task(task)
+result[:reasoning]   # => the plan the agent drafted before answering
+result[:content]     # => the final answer
+```
+
+Off by default. If the reasoning pass keeps returning empty output past
+`max_reasoning_attempts`, the agent proceeds without a plan.
+
+## 🪟 Context Window Management
+
+Keep long tool-use loops or large injected context from overflowing the model's
+context window. When enabled, the oldest non-system messages are dropped to fit
+before each LLM call (system messages and the latest message are always kept):
+
+```ruby
+agent = RCrewAI::Agent.new(name: 'a', role: '...', goal: '...',
+                           respect_context_window: true)
+```
+
+Window sizes come from `RCrewAI::ContextWindow` (with a conservative default for
+unknown models); headroom for the response is reserved from `max_tokens`. Off by
+default.
+
+## 🖼️ Multimodal Input
+
+Pass images to a vision-capable model via task attachments. Local files are
+base64-encoded automatically; URLs pass through:
+
+```ruby
+RCrewAI.configure { |c| c.llm_provider = :openai; c.openai_model = 'gpt-4o' }
+
+task = RCrewAI::Task.new(
+  name: 'describe', description: 'What is in this chart?', agent: agent,
+  attachments: [
+    { type: :image, path: 'chart.png' },
+    { type: :image, url: 'https://example.com/photo.jpg' }
+  ]
+)
+```
+
+Supported on OpenAI and Azure; other providers raise a clear error when
+attachments are present.
+
 ## 📚 Knowledge (RAG)
 
 Ground agents in your own documents. Sources are chunked, embedded, and stored
@@ -292,6 +374,31 @@ Embeddings default to OpenAI's `text-embedding-3-small`; pass a custom
 `embedder:` (anything responding to `embed(texts)`) or vector store to swap the
 backend.
 
+## 🧠 Cognitive Memory
+
+Agents remember what they've done and recall it semantically on future tasks.
+Memory is zero-config by default (in-memory, lexical recall); add an embedder
+for semantic recall and a SQLite store for persistence:
+
+```ruby
+embedder = RCrewAI::Knowledge::Embedder.new
+store    = RCrewAI::Memory::SqliteStore.new(path: '~/.rcrewai/memory.db')
+
+agent = RCrewAI::Agent.new(
+  name: 'engineer', role: '...', goal: '...',
+  memory: { embedder: embedder, store: store }   # both optional
+)
+```
+
+- **Semantic recall** — with an embedder, an agent recalls conceptually related
+  past work even when the wording differs (falls back to word-overlap without one).
+- **Persistence** — a `SqliteStore` makes memory survive restarts.
+- **Memory types** — `agent.memory.short_term` / `long_term` / `entity` / `tool`.
+- **Scoping** — memory is scoped per agent so a shared store doesn't cross-read.
+
+Memory is best-effort: embedding failures fall back to lexical similarity, so it
+never breaks agent execution.
+
 ## 🌊 Flows
 
 Beyond crews, RCrewAI has **Flows** — an event-driven workflow engine for

data/ROADMAP.md

@@ -16,12 +16,10 @@ pricing.
 
 Since CrewAI's `1.0`, the framework grew a second pillar (**Flows**) plus
 **Knowledge (RAG)**, **Guardrails**, **structured output**, **Planning**, and
-**Training/Testing**. As of the `[Unreleased]` changes, RCrewAI now implements
-all of these — see the matrix below. Only backlog polish items remain.
+**Training/Testing**. RCrewAI now implements all of these — see the matrix below.
 
-**Status: all milestone issues (#5–#12) are complete.** The remaining backlog
-covers smaller polish items (reasoning, rate-limiting, batch kickoff, kickoff
-hooks, multimodal).
+**Status: complete.** All milestone issues (#5–#12) shipped in v0.4.0, and all
+backlog items (#15–#20) shipped in v0.5.0. There is no outstanding roadmap work.
 
 ## Parity matrix
 
@@ -42,7 +40,7 @@ hooks, multimodal).
 | Flows (`start`/`listen`/`router`) | ✅ | ✅ (#11) | ✅ done |
 | Flow state + persistence | ✅ | ✅ (#11) | ✅ done |
 | Training / Testing | ✅ | ✅ (#12) | ✅ done |
-| Reasoning, rate-limiting, batch kickoff | ✅ | ❌ | backlog |
+| Reasoning, rate-limiting, batch kickoff, hooks, context window, multimodal | ✅ | ✅ (#15–#20) | ✅ done |
 
 ## Milestones (highest leverage first)
 
@@ -78,7 +76,14 @@ The flagship. A Ruby DSL mirroring CrewAI Flows:
 - `crew.train(n_iterations:, filename:)` capturing human feedback.
 - `crew.test(n_iterations:, model:)` scoring runs.
 
-### Backlog
-Per-agent reasoning (`reasoning:`, `max_reasoning_attempts:`), `max_rpm`
-rate-limiting, `respect_context_window`, `kickoff_for_each` batch execution,
-`before_kickoff` / `after_kickoff` hooks, multimodal agents.
+### Backlog — ✅ all complete
+
+Formerly polish items with no set version; all shipped in the `[Unreleased]`
+changes (see CHANGELOG):
+
+- [#15](https://github.com/gkosmo/rcrewAI/issues/15) — `before_kickoff` / `after_kickoff` lifecycle hooks ✅
+- [#16](https://github.com/gkosmo/rcrewAI/issues/16) — `kickoff_for_each` batch execution ✅
+- [#17](https://github.com/gkosmo/rcrewAI/issues/17) — `max_rpm` rate limiting ✅
+- [#18](https://github.com/gkosmo/rcrewAI/issues/18) — per-agent reasoning (`reasoning:`, `max_reasoning_attempts:`) ✅
+- [#19](https://github.com/gkosmo/rcrewAI/issues/19) — `respect_context_window` history trimming ✅
+- [#20](https://github.com/gkosmo/rcrewAI/issues/20) — multimodal agents (image/file inputs) ✅

data/docs/superpowers/specs/2026-07-06-cognitive-memory-design.md

@@ -0,0 +1,118 @@
+# Cognitive Memory: Semantic, Persistent, Multi-Type Agent Memory
+
+**Date:** 2026-07-06
+**Status:** Approved design (pending implementation)
+**Target version:** `rcrewai` 0.6.0 (current is 0.5.0)
+**Scope:** Replace the placeholder `RCrewAI::Memory` with a cognitive memory system modeled on CrewAI's memory taxonomy — short-term (semantic recent recall), long-term (persistent insights), entity memory, and tool-usage memory — backed by embeddings and a persistent SQLite vector store. No new external dependencies; reuses the in-repo `Knowledge::Embedder` and cosine-search infrastructure.
+
+---
+
+## Motivation
+
+The current `RCrewAI::Memory` is a placeholder. It matches "relevant" past executions with word-overlap similarity (`common_words / total_words`) plus a hardcoded keyword/stopword list and a `classify_task_type` that greps the description for words like "research" or "write". Nothing is embedded; nothing is persisted; everything is lost when the process exits. It is the weakest module in the codebase and it undercuts the framework's core promise — agents that learn from what they've done.
+
+Meanwhile CrewAI has advanced *past* the version this project ported: its 2026 "cognitive memory" work adds semantic recall, distinct memory types (short-term, long-term, entity), and persistence. This is genuine parity work, not gold-plating — and the infrastructure to do it well already exists in-repo (`Knowledge::Embedder`, cosine `Knowledge::Store`), so we compose rather than build from scratch.
+
+## Goals
+
+1. **Semantic recall.** Replace word-overlap with embedding-based similarity so an agent recalls conceptually related past work, not just keyword-overlapping work.
+2. **Persistence.** Memory survives process restarts via a SQLite-backed vector store (no external service).
+3. **Memory types.** Model CrewAI's taxonomy: short-term, long-term, entity, and tool-usage memory, each with clear write/read semantics.
+4. **Backward compatibility.** The existing `Memory` public API (`add_execution`, `add_tool_usage`, `relevant_executions`, `tool_usage_for`, `clear_*!`, `stats`) keeps working, so `Agent` and the runners need no changes to function.
+5. **No new hard dependencies.** SQLite via Ruby's stdlib-adjacent `sqlite3` gem (already common in Ruby projects) — but the store is pluggable, and the default remains in-memory so the gem works with zero setup.
+
+## Non-goals
+
+- External vector databases (Chroma/Qdrant/pgvector). The pluggable store interface leaves room for them later; we ship SQLite + in-memory.
+- Reranking, query rewriting, or summarization of memories (CrewAI has some of this; it's a follow-up).
+- Multi-tenant / shared cross-agent memory servers.
+
+## Decisions captured during brainstorming
+
+| # | Decision | Choice |
+|---|----------|--------|
+| 1 | Similarity mechanism | Embeddings + cosine, reusing `Knowledge::Embedder` and the cosine math already in `Knowledge::Store`. |
+| 2 | Persistence | Pluggable vector store; ship `InMemoryStore` (default) and `SqliteStore`. |
+| 3 | Memory taxonomy | Four types: `ShortTermMemory`, `LongTermMemory`, `EntityMemory`, `ToolMemory`. |
+| 4 | Backward compat | Keep the `Memory` facade and its current method signatures; delegate internally to the new types. |
+| 5 | Default behavior | Zero-config: in-memory store, embeddings only if an embedder is available/keyed; graceful fallback to the current lexical similarity when no embedder. |
+| 6 | Embedding failures | Never fatal. If embedding fails (no key, network), fall back to lexical similarity so agents still run. |
+| 7 | Namespacing | Memories are scoped by agent (role/name) so one agent's memory doesn't leak into another, matching CrewAI's per-role collections. |
+
+## Architecture
+
+```
+RCrewAI::Memory  (facade — preserves today's public API)
+  ├── ShortTermMemory   recent executions, semantic recall, capped, volatile-by-default
+  ├── LongTermMemory    durable insights from successful executions, persistent
+  ├── EntityMemory      facts about entities (people, systems, concepts) extracted from work
+  └── ToolMemory        tool-call history + outcomes (replaces @tool_usage)
+
+Each memory type holds:
+  - a VectorStore  (InMemoryStore | SqliteStore)   ← persistence + search
+  - an Embedder    (Knowledge::Embedder | nil)     ← semantic vectors
+  - a lexical fallback (current word-overlap)       ← when no embedder
+```
+
+### Storage: `Memory::VectorStore`
+
+A small interface (mirrors `Knowledge::Store` so they can converge later):
+
+```ruby
+store.add(id:, text:, vector:, metadata:)   # upsert a record
+store.search(vector, k:, scope:)            # cosine top-k, filtered by scope
+store.all(scope:)                           # enumerate (for stats / lexical fallback)
+store.delete(scope:)                        # clear a scope
+```
+
+- **`InMemoryStore`** — array of records; cosine in Ruby (lift the existing private cosine out of `Knowledge::Store` into a shared `Similarity` helper). Default.
+- **`SqliteStore`** — one row per memory: `id, scope, type, text, vector (blob), metadata (json), created_at`. Vectors stored as packed floats (`Array#pack('e*')`). Cosine computed in Ruby over candidate rows filtered by `scope`/`type` (fine for the thousands-of-memories scale agents produce; a real ANN index is a later optimization). Opened at a configurable path (default `~/.rcrewai/memory.db` or `:memory:`).
+
+### Embeddings
+
+Reuse `Knowledge::Embedder`. The memory system takes an optional `embedder:`; when present, `add_*` embeds the text and stores the vector, and recall embeds the query. When absent (no key, or explicitly disabled), the store keeps `vector: nil` and recall falls back to the current lexical similarity over `store.all`. This keeps the zero-config path working.
+
+### The four memory types
+
+- **ShortTermMemory** — every execution is written here; recall returns the top-k semantically similar recent items. Capped (default 100) and, by default, uses the in-memory store (volatile). This is the direct upgrade of today's `@short_term` + `relevant_executions`.
+- **LongTermMemory** — successful executions promote a distilled "insight" record (task type, what worked, result summary) into a persistent store, deduped by similarity so we don't store near-identical insights repeatedly. Upgrade of today's `@long_term`.
+- **EntityMemory** — optional lightweight entity extraction (heuristic first: capitalized noun phrases / quoted terms; pluggable LLM extractor later) so agents accumulate facts about recurring entities. New capability.
+- **ToolMemory** — replaces `@tool_usage`; same API (`add_tool_usage`, `tool_usage_for`) but persistable and searchable.
+
+### Facade: `RCrewAI::Memory`
+
+Keeps today's signatures, delegating to the types:
+
+```ruby
+Memory.new(embedder: nil, store: nil, scope: nil, short_term_limit: 100)
+
+add_execution(task, result, execution_time)   # -> ShortTerm + (if success) LongTerm + Entity
+add_tool_usage(tool_name, params, result)      # -> ToolMemory
+relevant_executions(task, limit = 3)           # -> semantic recall across Short+Long, formatted string|nil
+tool_usage_for(tool_name, limit = 5)           # -> ToolMemory
+clear_short_term! / clear_all! / stats         # -> delegate
+```
+
+`Agent#initialize` gains optional `memory:` config (embedder/store/persistence) but defaults to the zero-config in-memory, lexical-fallback behavior — so existing code is unchanged.
+
+## Backward compatibility & migration
+
+- The `Memory` public API is preserved exactly; `agent.rb` and the runners are untouched functionally.
+- Default construction (`Memory.new` with no args) behaves like today from the caller's perspective — just with better recall when an embedder is configured, and identical lexical recall when not.
+- `relevant_executions` still returns the same formatted-string-or-nil shape consumed at `agent.rb:241`.
+
+## Testing strategy
+
+- **Unit**: each memory type with a fake deterministic embedder (as `knowledge_spec.rb` already does) — assert semantic recall ranks the conceptually-closest item first, not the keyword-overlapping one.
+- **Persistence**: `SqliteStore` round-trip — write memories, reopen the DB, recall survives.
+- **Fallback**: with no embedder, recall still returns results via lexical similarity (proves the graceful path).
+- **Facade compat**: the existing `memory_spec.rb` expectations continue to pass (or are updated only where the placeholder behavior was clearly a bug).
+- **Agent integration**: an agent with `reasoning`/knowledge off recalls a semantically-related prior execution injected into its context.
+
+## Dependencies
+
+- `sqlite3` gem — added as a runtime dependency, but only required lazily inside `SqliteStore` so the gem loads and the in-memory path works even if it's absent. (If we'd rather avoid the dependency entirely, the fallback is a JSON-file-backed store; SQLite is preferred for query/scale.)
+
+## Rollout
+
+Ship in `0.6.0`. `docs/upgrading-to-0.6.md` documents opt-in persistence and embeddings; the default path is unchanged, so it's a no-action upgrade for existing users.

data/docs/upgrading-to-0.4.md

@@ -0,0 +1,191 @@
+# Upgrading to RCrewAI 0.4
+
+RCrewAI 0.4 closes the feature-parity gap with the modern CrewAI framework. It
+adds CrewAI's second pillar — **Flows** — alongside **Knowledge (RAG)**,
+structured task output, guardrails, planning, and training/testing.
+
+**0.4 is fully backward compatible.** There are no breaking changes: existing
+0.3 code runs unchanged. Everything below is opt-in.
+
+Each new capability has a runnable example under `examples/` — see the links
+throughout.
+
+---
+
+## 1. What you must do
+
+Nothing. Upgrade the gem and your existing code keeps working:
+
+```ruby
+gem 'rcrewai', '~> 0.4'
+```
+
+---
+
+## 2. What you can do (new capabilities)
+
+### 2a. Per-agent LLM
+
+Give each agent its own provider/model instead of only the global default.
+Pass a provider symbol, an options hash, or a pre-built client:
+
+```ruby
+worker  = RCrewAI::Agent.new(name: 'worker', role: '...', goal: '...',
+                             llm: { provider: :openai, model: 'gpt-4o-mini' })
+manager = RCrewAI::Agent.new(name: 'manager', role: '...', goal: '...',
+                             llm: { provider: :anthropic, model: 'claude-3-opus-20240229' })
+```
+
+Omitting `llm:` keeps the global `RCrewAI.configure` behavior. Overrides never
+mutate the global configuration.
+
+### 2b. Structured output, guardrails, and file output
+
+Post-process a task's result after the agent produces it:
+
+```ruby
+task = RCrewAI::Task.new(
+  name: 'extract', description: '...', agent: agent,
+
+  # Validate & coerce against a JSON schema. Non-conforming output re-runs the
+  # agent with the error fed back. Parsed object on task.structured_output.
+  output_schema: { type: 'object', properties: { title: { type: 'string' } },
+                   required: ['title'] },
+
+  # Validate & transform. ->(output) { [ok, value_or_error] }. Retries up to
+  # guardrail_max_retries (default 3) with the reason fed back to the agent.
+  guardrail: ->(out) { [out.length < 5000, 'too long'] },
+
+  # Persist the result (parent dirs created unless create_directory: false).
+  output_file: 'out/report.md', markdown: true
+)
+
+task.execute
+task.structured_output  # => { "title" => "..." }
+task.raw_result         # => the unprocessed string the agent produced
+```
+
+See `examples/structured_output_example.rb`.
+
+### 2c. Knowledge (RAG)
+
+Ground agents in your own documents. Sources are chunked, embedded, and stored
+in an in-memory cosine vector store; relevant chunks are injected into each
+task's prompt automatically.
+
+```ruby
+kb = RCrewAI::Knowledge::Base.new(sources: [
+  RCrewAI::Knowledge::StringSource.new('Refunds within 30 days.'),
+  RCrewAI::Knowledge::FileSource.new('docs/policy.txt'),
+  RCrewAI::Knowledge::PdfSource.new('handbook.pdf'),
+  RCrewAI::Knowledge::UrlSource.new('https://example.com/faq')
+])
+
+# Agent-level (role-specific):
+agent = RCrewAI::Agent.new(name: 'support', role: '...', goal: '...', knowledge: kb)
+
+# Or pass raw sources and let the agent build the base:
+agent = RCrewAI::Agent.new(name: 'support', role: '...', goal: '...',
+                           knowledge_sources: [RCrewAI::Knowledge::StringSource.new('...')])
+
+# Crew-level knowledge is shared with every agent:
+crew = RCrewAI::Crew.new('support', knowledge: kb)
+```
+
+Embeddings default to OpenAI's `text-embedding-3-small`; pass a custom
+`embedder:` (anything responding to `embed(texts)`) to swap the backend.
+See `examples/knowledge_rag_example.rb`.
+
+### 2d. Planning
+
+Have a planner pass draft a per-task plan before execution:
+
+```ruby
+crew = RCrewAI::Crew.new('research_crew', planning: true)
+# Optionally use a dedicated (stronger) planner model:
+crew = RCrewAI::Crew.new('research_crew', planning: true,
+                         planning_llm: { provider: :anthropic, model: 'claude-3-opus-20240229' })
+```
+
+The plan is folded into each task's description. Planning is best-effort: a
+planner error or unparseable output leaves tasks unchanged.
+
+### 2e. Training & testing
+
+Iterate on a crew with feedback, or score repeated runs:
+
+```ruby
+# Run N times, collect feedback after each run, persist to JSON.
+crew.train(n_iterations: 3, filename: 'training.json')
+
+# Provide feedback programmatically instead of prompting a human:
+crew.train(n_iterations: 3, filename: 'training.json',
+           feedback: ->(iteration, result) { "run #{iteration}: #{result[:success_rate]}%" })
+
+# Run N times and score each run (defaults to success_rate).
+crew.test(n_iterations: 5)
+# => { iterations: 5, scores: [...], average_score: 92.0 }
+```
+
+See `examples/planning_and_training_example.rb`.
+
+### 2f. Flows
+
+Flows are an event-driven workflow engine — the biggest addition in 0.4.
+Subclass `RCrewAI::Flow` and wire methods with a class-level DSL:
+
+```ruby
+class ArticleFlow < RCrewAI::Flow
+  start :outline
+  def outline
+    state.sections = %w[intro body conclusion]
+    state.sections.length
+  end
+
+  listen :outline
+  def draft(section_count)
+    state.words = section_count * 100
+    state.words
+  end
+
+  router :draft
+  def review(words)
+    words >= 250 ? :publish : :expand
+  end
+
+  listen :publish
+  def publish = state.status = 'published'
+
+  listen :expand
+  def expand = state.status = 'needs more work'
+end
+
+flow = ArticleFlow.new
+flow.kickoff(inputs: { author: 'me' })
+flow.state.status   # => "published"
+```
+
+- `start` / `listen` / `router` wire methods into a graph; a listener receives
+  its trigger's return value, and a router's return becomes a label listeners
+  fire on.
+- Combine triggers with `and_(:a, :b)` (all) and `or_(:a, :b)` (any).
+- **State** is a schemaless object with a UUID, seedable via `kickoff(inputs:)`.
+- **Persistence**: pass `state_store:`
+  (`RCrewAI::Flow::FileStateStore.new(dir)` or your own `#save`/`#load`) and
+  call `flow.restore(id)` to resume.
+- Invoke a `Crew` inside any step, or pause with `human_feedback('Approve?')`.
+
+See `examples/flow_example.rb`.
+
+---
+
+## When to use Flows vs. Crews
+
+- **Crew** — a team of agents collaborating on a set of tasks (sequential,
+  hierarchical, or async). Reach for a crew when the work is "have these agents
+  produce these outputs."
+- **Flow** — explicit, branching orchestration with state. Reach for a flow
+  when you need conditional paths, joins, persistence/resumption, or you want to
+  coordinate multiple crews and plain Ruby steps.
+
+They compose: a Flow step can kick off a Crew.

data/docs/upgrading-to-0.6.md

@@ -0,0 +1,101 @@
+# Upgrading to RCrewAI 0.6
+
+RCrewAI 0.6 replaces the placeholder memory with a **cognitive memory** system:
+semantic recall (embeddings + cosine), optional SQLite persistence, and four
+memory types (short-term, long-term, entity, tool).
+
+**0.6 is backward compatible.** The `RCrewAI::Memory` public API is unchanged,
+and the default (`Memory.new` with no config) behaves like before from the
+caller's perspective — just with better recall once an embedder is configured.
+Everything below is opt-in.
+
+---
+
+## 1. What you must do
+
+Nothing. Upgrade the gem; existing code keeps working:
+
+```ruby
+gem 'rcrewai', '~> 0.6'
+```
+
+Note: `0.6` adds `sqlite3` as a runtime dependency (used only if you opt into
+persistence). It is required lazily, so the in-memory default works even if the
+native extension isn't present.
+
+---
+
+## 2. What you can do (new capabilities)
+
+### 2a. Semantic recall via embeddings
+
+By default, memory recall uses lexical (word-overlap) similarity — same as
+before. Pass an embedder to get semantic recall, so an agent remembers
+conceptually related work even when the wording differs:
+
+```ruby
+embedder = RCrewAI::Knowledge::Embedder.new   # OpenAI text-embedding-3-small
+agent = RCrewAI::Agent.new(
+  name: 'engineer', role: '...', goal: '...',
+  memory: { embedder: embedder }
+)
+```
+
+If embedding fails (no API key, network error), recall automatically falls back
+to lexical similarity — memory never breaks agent execution.
+
+### 2b. Persistent memory (SQLite)
+
+Give memory a SQLite store and it survives restarts:
+
+```ruby
+store = RCrewAI::Memory::SqliteStore.new(path: '~/.rcrewai/memory.db')
+agent = RCrewAI::Agent.new(
+  name: 'engineer', role: '...', goal: '...',
+  memory: { embedder: embedder, store: store }
+)
+```
+
+The default store is in-memory (volatile). Any object implementing the store
+interface (`add` / `search` / `all` / `delete` / `delete_record`) can be used —
+e.g. a future Chroma/pgvector adapter.
+
+### 2c. Memory scoping
+
+Each agent's memory is scoped to its name by default, so agents sharing a
+persistent store don't read each other's memories. Override with
+`memory: { scope: 'custom' }` if you want deliberate sharing.
+
+### 2d. Memory types (direct access)
+
+The `Memory` facade exposes the underlying types for advanced use:
+
+```ruby
+agent.memory.short_term   # recent executions, semantic recall, capped
+agent.memory.long_term    # durable insights from successful executions, deduped
+agent.memory.entity       # facts about entities (people, systems) seen in work
+agent.memory.tool         # tool-call history + outcomes
+
+agent.memory.entity.entities            # => ["Alice", "AWS", ...]
+agent.memory.long_term.recall('...', limit: 3)
+```
+
+### 2e. Pre-built Memory
+
+You can also construct and pass a `Memory` directly:
+
+```ruby
+memory = RCrewAI::Memory.new(scope: 'shared', embedder: embedder, store: store)
+agent  = RCrewAI::Agent.new(name: 'a', role: '...', goal: '...', memory: memory)
+```
+
+---
+
+## Behavior notes
+
+- `add_execution` writes to short-term always, and (on success) promotes a
+  deduped insight to long-term and extracts entities.
+- `relevant_executions(task, limit)` recalls across short- and long-term and
+  returns the same formatted-string-or-nil shape as before.
+- `clear_short_term!` leaves long-term insights intact; `clear_all!` wipes
+  everything.