rcrewai 0.8.1 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cbd74067513b5593791b282baf20f18c8dd8f6cd9257b02f70423ee4cba3486d
-  data.tar.gz: 215cade06d04b032d254e792a2af9d36d7c00de073580dff31125d46e5de4b48
+  metadata.gz: 1e33f9998ef9f32aa53d4c6df3e8e7b04314f7b5cbb9fe06e5c7d1936b466222
+  data.tar.gz: 294a6a4a5ed7d5f1b5fbb58c3edd0bb69a0cd5d9e61da07b9f8dabf6c4457414
 SHA512:
-  metadata.gz: 735222585596dc578801792d74a6f5435037c78352f10e80670b2e179e0cce11a6257b79624e8a1abd0a3aa6978d86ec852e02cbdd6dc22e00a1364eafbb1058
-  data.tar.gz: 3f5b7145864b603e292b9daced5ef1c39c2d3084847658265d2b67a58ef8e3575ba8383e9494e7a2bcfcd7d6de231daadaac43ae41245afea7ed2c4f817649b9
+  metadata.gz: f64195cd3065e2c26dd65ddea08246bec905754192e8ba0fad1a23f7bcbfc998c2f8a4d3cad91f7491f5d8e0f2fc5c4cd53be60de47e737f9d41b2c898089ba7
+  data.tar.gz: 5107c4953f3ff80f86125fdbb5325bf582b1c624b07e97b739890fd21d0c83c892c7adff4fd982556141dd5a60b186814bcc3b1644056a89d589788810fee860

data/CHANGELOG.md

@@ -7,6 +7,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.10.0] - 2026-10-19
+
+### Added
+- Anthropic live streaming for real-time callers (voice agents):
+  - `RCrewAI::CancelToken`: pass `cancel:` to `chat`; `cancel!` aborts the
+    stream at the next chunk, closes the connection and raises
+    `RCrewAI::StreamCancelled`. Under an Async reactor, stopping the calling
+    task does the same.
+  - Streaming requests reuse one keep-alive connection per client
+    (`RCrewAI::PersistentHTTP`), so back-to-back turns skip TCP/TLS setup.
+  - `chat(model:)` overrides the configured model per call.
+  - Usage now carries `cached_tokens` (Anthropic `cache_read_input_tokens`)
+    in the result hash and on the `Events::Usage` stream event.
+- The Anthropic client honours `config.base_url`.
+
+### Notes
+- Verified fiber-safe on Ruby >= 3.2 / async 2.x: Anthropic streaming inside an
+  Async reactor never blocks other fibers (spec covers it, skipped < 3.2).
+
+## [0.9.0] - 2026-09-10
+
+Concurrency. Three points where independent, IO-bound work ran in sequence now
+fan out across threads, each bounded and each preserving input order.
+
+| | Before | After |
+|---|---|---|
+| Tool calls in one turn (3 x 150ms) | 0.45s | 0.165s |
+| Consensus with 3 agents | 1.22s | 0.22s |
+
+Entirely additive: no API changes, and every fan-out can be disabled or
+re-bounded. Concurrency is deliberately capped rather than unlimited — an
+unbounded fan-out trades a latency problem for a rate-limit one.
+
+### Added
+- Parallel tool execution: when a model requests several tools in one turn, `ToolRunner` now runs them concurrently instead of in sequence, so the turn costs the slowest call rather than their sum. Results are collected by index, so the message history stays aligned with the `tool_call` ids regardless of completion order. Bounded by `max_tool_concurrency` (default 8); a single tool call still runs inline with no thread. Disable per agent with `Agent.new(parallel_tools: false)`, or per runner with `ToolRunner.new(parallel_tools: false)`. The legacy `USE_TOOL[]` runner is unaffected — it parses one directive at a time.
+- Tool events emitted from worker threads carry the enclosing run span, so the 0.8.0 event hierarchy stays intact under concurrency.
+- Concurrent embedding: providers without a batch embeddings endpoint (`:google`, `:ollama`) issued one HTTP request per text, in sequence, so building a knowledge base cost the sum of every chunk's round-trip. Those requests now run concurrently, bounded by `max_concurrency` (default 8, configurable via `Knowledge::Embedder.new(max_concurrency:)`). Vectors are returned in input order, since `Knowledge::Base#build!` zips them back against their chunks positionally. An error in any request propagates rather than yielding a knowledge base with silently missing vectors. `:openai` and `:azure` already batch into a single request and are unchanged.
+- Concurrent consensus: the `:consensual` process ran its proposal and scoring rounds one LLM call at a time — with the default three agents that is 3 proposals plus 9 scoring calls, twelve sequential round-trips per task. Proposals now run concurrently, and scoring fans out across the whole (candidate × voter) grid rather than per candidate (measured: 1.22s → 0.22s for three agents). Bounded by `consensus_max_concurrency` (default 8), since scoring is quadratic in participants — six agents would otherwise put 36 requests in flight at once. Results keep input order, so ties still break toward the task's assigned agent deterministically.
+
 ## [0.8.1] - 2026-09-10
 
 Dependency cleanup. No API or behavior change.

data/ROADMAP.md

@@ -5,7 +5,7 @@ This roadmap tracks feature parity between **RCrewAI** (Ruby) and the upstream
 
 ## Current status
 
-- **RCrewAI:** `0.7.1` released; `0.8.0`, `0.9.0` and `0.9.x` merged to `main`, unreleased
+- **RCrewAI:** `0.8.1` released; 1.0.0 concurrency work merged to `main`, unreleased
 - **Upstream crewai:** `1.15.21`
 
 RCrewAI is a faithful port of CrewAI's **"Crews"** mental model (Agents / Tasks /
@@ -15,12 +15,12 @@ human-in-the-loop), and it carries CrewAI's second pillar (**Flows**) plus
 **training/testing**. In one area — cognitive memory (semantic recall, SQLite
 persistence, four memory types) — the gem went past what was originally ported.
 
-**Status: one milestone remaining.** An earlier revision of this file declared
+**Status: roadmap complete.** An earlier revision of this file declared
 parity "complete" against a matrix that only covered CrewAI through roughly
 `1.0` (October 2025) while quoting `1.15.x` in its header; everything upstream
-added across `1.1`–`1.15` was unmeasured. That delta was re-derived, and three
-of the four scheduled milestones have since shipped. Only native async (1.0.0)
-is outstanding, and it is blocked on an open decision — see below.
+added across `1.1`–`1.15` was unmeasured. That delta was re-derived, and all four
+scheduled milestones have since shipped. What remains are two additive items
+(Bedrock/Responses streaming, SigV4) and one deferred concept (A2A).
 
 ## Parity matrix
 
@@ -57,7 +57,8 @@ is outstanding, and it is blocked on an open decision — see below.
 
 | Concept | crewai | RCrewAI | Plan |
 |---|---|---|---|
-| Native async (LLM + tool level) | ✅ (1.4–1.6) | ⚠️ partial | 1.0.0 |
+| Concurrent tool calls | ✅ (1.4–1.6) | ✅ (#47) | — |
+| Concurrent embedding / consensus | ✅ (1.4–1.6) | ✅ (#49, #50) | — |
 | OTel export for the event hierarchy | ✅ (1.10+) | ❌ | 1.0.x |
 | Streaming for Bedrock / Responses | ✅ | ❌ | 1.0.x |
 | Bedrock SigV4 signing | ✅ | ❌ (hook workaround) | 1.0.x |
@@ -115,24 +116,71 @@ Two deliberate limitations carried forward to 1.0.x: Bedrock does not implement
 SigV4 (a hard `aws-sigv4` dependency for one provider is not worth it; sign via
 a `before_request` hook), and Bedrock/Responses are non-streaming only.
 
-### 1.0.0 — Native async
+### 1.0.0 — Concurrency ✅ shipped
 
-The largest item, and the one with a genuine architecture decision attached.
-Today `AsyncExecutor` fans tasks out across a `Concurrent::ThreadPoolExecutor` in
-dependency-ordered phases; concurrency stops at the task boundary. CrewAI went
-async *through* the LLM and tool calls (1.4–1.6), covering flows, crews, tasks,
-knowledge, and memory.
+**Decision made: threads, not fibers.** The roadmap previously left this open.
+Prototyping settled it, and not the way the fiber option's low apparent cost
+suggested.
 
-Ruby has no direct port of that. Two candidate models:
+The attraction of fibers was that it looked cheap: keep Faraday, wrap calls in
+`Async`, get non-blocking IO for free. That premise is false on this stack. On
+Ruby 3.1.4 with `async` 2.24.0, a Faraday/`net_http` call inside a fiber
+**never returns** — the task is silently abandoned and the process exits `0` as
+though it succeeded. Raw `Net::HTTP` under `Async` raises `NoMethodError`
+rather than yielding. A first benchmark appeared to show a 1.53s → 0.0s win;
+it was measuring five failed requests that never reached the server.
 
-1. **Fibers** via the `async` gem — closer to upstream's shape, new runtime
-   dependency, and every provider client's HTTP layer has to cooperate.
-2. **Stay on threads** and make the client layer non-blocking — smaller
-   conceptual change, keeps `concurrent-ruby`, less faithful to upstream.
+Making fibers work would mean replacing Faraday with `async-http` across all
+nine provider clients, adding a hard runtime dependency on a stack whose
+observed failure mode is *silent abandonment* — the worst possible behavior in
+an agent framework — and likely raising `required_ruby_version` from `3.0`,
+since this fragility lives exactly in 3.0/3.1 scheduler support.
 
-**This decision is open and blocks the milestone.** It touches all five LLM
-clients either way. The 8.2k lines of `lib/` are backed by 4.9k lines of spec,
-which is what makes a refactor at this depth tractable.
+Threads cost none of that: `concurrent-ruby` is already a dependency, the Ruby
+floor is unchanged, and failures are loud.
+
+**Update (2026-10):** the fiber failure is specific to Ruby 3.0/3.1. On Ruby
+3.3 with `async` 2.46, Net::HTTP streaming inside a fiber yields to the
+reactor correctly (`spec/anthropic_live_streaming_spec.rb` asserts a 10 ms
+ticker never stalls, and that stopping the task closes the stream). Threads
+stay the internal concurrency model; fiber callers on 3.2+ are supported.
+
+Upstream's async/await shape is a Python idiom; porting its *form* rather than
+its *effect* would buy nothing here.
+
+#### Shipped
+
+Three fan-out points, all on threads, each bounded and each preserving input
+order (every one of them feeds a positional zip or a deterministic tie-break
+downstream):
+
+- **Parallel tool calls** (#47) — a turn's tool calls run concurrently, so a
+  turn costs the slowest call rather than their sum. Measured 0.45s → 0.165s
+  for three 150ms tools. `Agent.new(parallel_tools: false)` opts out.
+- **Concurrent embedding** (#49) — providers without a batch endpoint
+  (`:google`, `:ollama`) issued one request per text in sequence, so building a
+  knowledge base cost the sum of every chunk's round-trip. Bounded by
+  `Knowledge::Embedder.new(max_concurrency:)`.
+- **Concurrent consensus** (#50) — the `:consensual` process made twelve
+  sequential LLM calls per task with three agents. Proposals and the whole
+  (candidate × voter) scoring grid now fan out. Measured 1.22s → 0.22s.
+  Bounded by `Crew.new(consensus_max_concurrency:)`.
+
+Bounding was not optional: the first consensus implementation was unbounded and
+put 36 concurrent LLM calls in flight with six agents — a thundering herd that
+trips provider rate limits, which is a worse problem than the latency it
+solves. Every fan-out here has a ceiling and a spec asserting it.
+
+#### Remaining
+
+An audit of `lib/` for per-item IO loops now turns up only CPU-bound work
+(lexical similarity), so the task-boundary goal is met. What is left is not
+concurrency work as such:
+
+- Streaming for Bedrock and Responses (both non-streaming today).
+- Bedrock SigV4 signing, currently a `before_request` hook workaround.
+
+Neither blocks a 1.0.0 release; both are tracked in the gaps table.
 
 ### Deferred — A2A
 
@@ -147,12 +195,15 @@ no current RCrewAI user has asked for. Revisit once the items above land.
 | 0.8.0 | Interceptors + observability | Low | ✅ merged (#39) |
 | 0.9.0 | Checkpointing | Moderate | ✅ merged (#42) |
 | 0.9.x | Providers, Responses API | Low | ✅ merged (#41) |
-| 1.0.0 | Native async | High — decision open | ⏳ blocked |
+| 1.0.0 | Concurrency (threads) | Moderate | ✅ merged (#47, #49, #50) |
 
 The three shipped milestones are on `main` and unreleased; they want a version
 bump and a release before or alongside 1.0.0 work.
 
-**1.0.0 is blocked on the fibers-vs-threads decision above.** It is the only
-remaining scheduled work, and the largest single change in this roadmap: it
-touches all nine provider clients and the executor. Nothing else should start
-before that call is made.
+**All scheduled milestones are merged.** The concurrency work landed on threads
+across three fan-out points; see the milestone above for the prototype evidence
+that ruled out fibers.
+
+`main` carries the 1.0.0 concurrency work unreleased on top of `0.8.1`. The
+remaining gaps — Bedrock/Responses streaming and SigV4 — are additive and do
+not block cutting a release.

data/lib/rcrewai/agent.rb

@@ -37,6 +37,7 @@ module RCrewAI
       @logger = Logger.new($stdout)
       @logger.level = verbose ? Logger::DEBUG : Logger::INFO
       @reasoning = options.fetch(:reasoning, false)
+      @parallel_tools = options.fetch(:parallel_tools, true)
       @max_reasoning_attempts = options.fetch(:max_reasoning_attempts, 3)
       @respect_context_window = options.fetch(:respect_context_window, false)
       @memory = build_memory(options[:memory])
@@ -60,11 +61,16 @@ module RCrewAI
         runner_class = pick_runner_class
         @logger.info "[rcrewai] agent=#{name} runner=#{runner_class.name.split('::').last}"
 
-        runner = runner_class.new(
+        runner_opts = {
           agent: self, llm: @llm_client, tools: @tools,
           max_iterations: opts.fetch(:max_iterations, max_iterations),
           event_sink: sink
-        )
+        }
+        # Only the native-tool runner executes tool calls concurrently; the
+        # legacy runner parses one USE_TOOL[] directive at a time.
+        runner_opts[:parallel_tools] = @parallel_tools if runner_class == ToolRunner
+
+        runner = runner_class.new(**runner_opts)
 
         runner_result = runner.run(messages: initial_messages)
         execution_time = Time.now - start_time

data/lib/rcrewai/cancel_token.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module RCrewAI
+  # Raised inside a streaming call when its CancelToken is triggered.
+  class StreamCancelled < Error; end
+
+  # Cooperative cancellation for a streaming LLM call. Pass it as
+  # `cancel:` to a client's #chat; calling #cancel! (from the stream sink,
+  # another thread, or another fiber) aborts the stream at the next chunk,
+  # closes the connection and raises StreamCancelled in the caller.
+  #
+  # Under an Async reactor, stopping the calling task has the same effect.
+  class CancelToken
+    def initialize
+      @cancelled = false
+    end
+
+    def cancel!
+      @cancelled = true
+    end
+
+    def cancelled?
+      @cancelled
+    end
+  end
+end

data/lib/rcrewai/crew.rb

@@ -23,6 +23,7 @@ module RCrewAI
       @planning_llm = options[:planning_llm]
       @planned = false
       @consensus_agents = options.fetch(:consensus_agents, 3)
+      @consensus_max_concurrency = options[:consensus_max_concurrency]
       @knowledge = build_knowledge(options[:knowledge], options[:knowledge_sources])
       @before_kickoff_hooks = []
       @after_kickoff_hooks = []
@@ -35,7 +36,8 @@ module RCrewAI
       validate_process_type!
     end
 
-    attr_reader :knowledge, :stream_sink, :last_inputs, :consensus_agents, :run_id
+    attr_reader :knowledge, :stream_sink, :last_inputs, :consensus_agents, :run_id,
+                :consensus_max_concurrency
 
     def planning?
       @planning

data/lib/rcrewai/events.rb

@@ -17,7 +17,8 @@ module RCrewAI
     ToolCallResult  = Struct.new(*BASE_ATTRS, :tool, :call_id, :result, :duration_ms, keyword_init: true)
     ToolCallError   = Struct.new(*BASE_ATTRS, :tool, :call_id, :error,                keyword_init: true)
     Thinking        = Struct.new(*BASE_ATTRS, :text,                                  keyword_init: true)
-    Usage           = Struct.new(*BASE_ATTRS, :prompt_tokens, :completion_tokens, :total_tokens, :cost_usd, keyword_init: true)
+    Usage           = Struct.new(*BASE_ATTRS, :prompt_tokens, :completion_tokens, :total_tokens, :cost_usd,
+                                 :cached_tokens, keyword_init: true)
     IterationStart  = Struct.new(*BASE_ATTRS, :iteration_index,                       keyword_init: true)
     IterationEnd    = Struct.new(*BASE_ATTRS, :finish_reason,                         keyword_init: true)
     Error           = Struct.new(*BASE_ATTRS, :error,                                 keyword_init: true)

data/lib/rcrewai/knowledge/embedder.rb

@@ -22,9 +22,16 @@ module RCrewAI
       GOOGLE_BASE = 'https://generativelanguage.googleapis.com/v1beta'
       OLLAMA_DEFAULT_URL = 'http://localhost:11434'
 
+      # Providers without a batch embeddings endpoint issue one request per
+      # text. Those are independent and IO-bound, so they run concurrently --
+      # bounded, since a large knowledge base would otherwise open a request
+      # per chunk at once.
+      DEFAULT_MAX_CONCURRENCY = 8
+
       attr_reader :provider, :model
 
-      def initialize(provider: :openai, model: nil, api_key: nil, config: RCrewAI.configuration)
+      def initialize(provider: :openai, model: nil, api_key: nil, config: RCrewAI.configuration,
+                     max_concurrency: DEFAULT_MAX_CONCURRENCY)
         @provider = provider.to_sym
         if @provider == :anthropic
           raise EmbeddingError,
@@ -34,6 +41,7 @@ module RCrewAI
         @config = config
         @model = model || DEFAULT_MODELS[@provider] || DEFAULT_MODELS[:openai]
         @api_key = api_key
+        @max_concurrency = max_concurrency
       end
 
       def embed(texts)
@@ -62,19 +70,29 @@ module RCrewAI
 
       def embed_google(texts)
         key = api_key_for(:google)
-        texts.map do |text|
+        concurrent_map(texts) do |text|
           url = "#{GOOGLE_BASE}/models/#{@model}:embedContent?key=#{key}"
           payload = { model: "models/#{@model}", content: { parts: [{ text: text }] } }
-          body = post_json(url, payload)
-          body.dig('embedding', 'values')
+          post_json(url, payload).dig('embedding', 'values')
         end
       end
 
       def embed_ollama(texts)
         base = @config.base_url || OLLAMA_DEFAULT_URL
-        texts.map do |text|
-          body = post_json("#{base}/api/embeddings", { model: @model, prompt: text })
-          body['embedding']
+        concurrent_map(texts) do |text|
+          post_json("#{base}/api/embeddings", { model: @model, prompt: text })['embedding']
+        end
+      end
+
+      # Maps over texts in bounded parallel, preserving input order: the caller
+      # zips vectors back against their chunks, so completion order must not
+      # leak into the result. An error in any request propagates -- a knowledge
+      # base with silently missing vectors is worse than a failed build.
+      def concurrent_map(texts, &block)
+        return texts.map(&block) if texts.length < 2 || @max_concurrency < 2
+
+        texts.each_slice(@max_concurrency).flat_map do |slice|
+          slice.map { |text| Thread.new { block.call(text) } }.map(&:value)
         end
       end
 

data/lib/rcrewai/llm_clients/anthropic.rb

@@ -7,6 +7,8 @@ require_relative '../events'
 require_relative '../sse_parser'
 require_relative '../provider_schema'
 require_relative '../pricing'
+require_relative '../cancel_token'
+require_relative '../persistent_http'
 
 module RCrewAI
   module LLMClients
@@ -23,7 +25,7 @@ module RCrewAI
 
       def initialize(config = RCrewAI.configuration, **hooks)
         super(config, **hooks)
-        @base_url = BASE_URL
+        @base_url = config.base_url || BASE_URL
       end
 
       def chat(messages:, tools: nil, tool_choice: :auto, stream: nil, **options)
@@ -31,7 +33,7 @@ module RCrewAI
         non_system = messages.reject { |m| m.is_a?(Hash) && m[:role] == 'system' }
 
         payload = {
-          model: config.model,
+          model: options[:model] || config.model,
           messages: format_messages(non_system),
           max_tokens: options[:max_tokens] || config.max_tokens || 1000,
           temperature: options[:temperature] || config.temperature
@@ -57,7 +59,7 @@ module RCrewAI
 
         if stream
           payload[:stream] = true
-          stream_chat(payload, stream)
+          stream_chat(payload, stream, cancel: options[:cancel])
         else
           plain_chat(payload)
         end
@@ -91,7 +93,7 @@ module RCrewAI
         apply_after_response(normalize_non_streaming(body), started_at)
       end
 
-      def stream_chat(payload, sink) # rubocop:disable Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
+      def stream_chat(payload, sink, cancel: nil) # rubocop:disable Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity,Metrics/MethodLength
         url = "#{@base_url}/messages"
         payload = apply_before_request(payload)
         started_at = Time.now
@@ -102,6 +104,7 @@ module RCrewAI
         blocks = {}
         finish_reason = nil
         prompt_tokens = nil
+        cached_tokens = nil
         completion_tokens = nil
 
         parser = SSEParser.new do |sse|
@@ -109,6 +112,7 @@ module RCrewAI
           case data['type']
           when 'message_start'
             prompt_tokens = data.dig('message', 'usage', 'input_tokens')
+            cached_tokens = data.dig('message', 'usage', 'cache_read_input_tokens')
           when 'content_block_start'
             cb = data['content_block'] || {}
             blocks[data['index']] = { id: cb['id'], name: cb['name'], arguments: +'' } if cb['type'] == 'tool_use'
@@ -131,7 +135,7 @@ module RCrewAI
           end
         end
 
-        streaming_post(url, payload) { |chunk| parser.feed(chunk) }
+        streaming_post(payload, cancel: cancel) { |chunk| parser.feed(chunk) }
 
         tool_calls = blocks.values.map do |b|
           {
@@ -144,15 +148,16 @@ module RCrewAI
         usage = {
           prompt_tokens: prompt_tokens,
           completion_tokens: completion_tokens,
-          total_tokens: (prompt_tokens || 0) + (completion_tokens || 0)
+          total_tokens: (prompt_tokens || 0) + (completion_tokens || 0),
+          cached_tokens: cached_tokens
         }
 
         if prompt_tokens || completion_tokens
           sink.call(Events::Usage.new(
                       type: :usage, timestamp: Time.now, agent: nil, iteration: nil,
                       prompt_tokens: prompt_tokens, completion_tokens: completion_tokens,
-                      total_tokens: usage[:total_tokens],
-                      cost_usd: Pricing.cost_for(config.model,
+                      total_tokens: usage[:total_tokens], cached_tokens: cached_tokens,
+                      cost_usd: Pricing.cost_for(payload[:model],
                                                  prompt_tokens: prompt_tokens || 0,
                                                  completion_tokens: completion_tokens || 0)
                     ))
@@ -163,23 +168,25 @@ module RCrewAI
           tool_calls: tool_calls,
           usage: usage,
           finish_reason: finish_reason || :stop,
-          model: config.model,
+          model: payload[:model],
           provider: :anthropic
         }
         apply_after_response(result, started_at)
       end
 
-      def streaming_post(url, payload, &on_chunk)
-        conn = Faraday.new do |f|
-          f.request :json
-          f.options.timeout = config.timeout
-          f.adapter Faraday.default_adapter
-        end
-        conn.post(url) do |req|
-          req.headers = build_headers.merge(auth_header)
-          req.body = payload.to_json
-          req.options.on_data = proc { |chunk, _| on_chunk.call(chunk) }
-        end
+      # Streams over one kept-alive connection per client (see PersistentHTTP).
+      def streaming_post(payload, cancel: nil, &on_chunk)
+        uri = URI("#{@base_url}/messages")
+        response = streaming_connection.post_stream(
+          uri.request_uri, headers: build_headers.merge(auth_header), body: payload.to_json, cancel: cancel, &on_chunk
+        )
+        return response if response.is_a?(Net::HTTPSuccess)
+
+        handle_response(Struct.new(:status, :body).new(response.code.to_i, response.body))
+      end
+
+      def streaming_connection
+        @streaming_connection ||= PersistentHTTP.new(@base_url, timeout: config.timeout)
       end
 
       def normalize_non_streaming(body)

data/lib/rcrewai/persistent_http.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'uri'
+
+module RCrewAI
+  # One keep-alive HTTP connection per client for streaming POSTs, so
+  # back-to-back calls skip TCP/TLS setup. Requests on a connection are
+  # serialised. A stream that does not finish cleanly (cancel, task stop,
+  # error) discards the connection; the next call reconnects. Under Ruby's
+  # fiber scheduler Net::HTTP yields on IO, so this is reactor-friendly.
+  class PersistentHTTP
+    RETRYABLE = [EOFError, IOError, Errno::ECONNRESET, Errno::EPIPE, Errno::ECONNREFUSED].freeze
+
+    def initialize(base_url, timeout:)
+      @uri = URI(base_url)
+      @timeout = timeout
+      @lock = Mutex.new
+      @http = nil
+    end
+
+    # Yields body chunks as they arrive. Returns the Net::HTTPResponse.
+    # Non-2xx responses are returned with their body read (not yielded).
+    def post_stream(path, headers:, body:, cancel: nil, &on_chunk)
+      @lock.synchronize do
+        attempts = 0
+        begin
+          perform(path, headers, body, cancel, &on_chunk)
+        rescue *RETRYABLE
+          # A kept-alive connection the server already closed fails before any
+          # bytes arrive; retry once on a fresh connection.
+          raise if @received || (attempts += 1) > 1
+
+          retry
+        end
+      end
+    end
+
+    def close
+      @http&.finish if @http&.started?
+    rescue IOError
+      nil
+    ensure
+      @http = nil
+    end
+
+    private
+
+    def perform(path, headers, body, cancel, &on_chunk)
+      @received = false
+      completed = false
+      request = Net::HTTP::Post.new(path, headers)
+      request.body = body
+      response = nil
+      connection.request(request) do |res|
+        response = res
+        unless res.is_a?(Net::HTTPSuccess)
+          res.read_body
+          next
+        end
+
+        res.read_body do |chunk|
+          @received = true
+          raise StreamCancelled, 'stream cancelled' if cancel&.cancelled?
+
+          on_chunk.call(chunk)
+          raise StreamCancelled, 'stream cancelled' if cancel&.cancelled?
+        end
+      end
+      completed = true
+      response
+    ensure
+      close unless completed
+    end
+
+    def connection
+      return @http if @http&.started?
+
+      @http = Net::HTTP.new(@uri.host, @uri.port)
+      @http.use_ssl = @uri.scheme == 'https'
+      @http.open_timeout = 10
+      @http.read_timeout = @timeout
+      @http.keep_alive_timeout = 30
+      @http.start
+    end
+  end
+end

data/lib/rcrewai/process.rb

@@ -410,6 +410,11 @@ module RCrewAI
     class Consensual < Base
       DEFAULT_CONSENSUS_AGENTS = 3
 
+      # Scoring is quadratic in participants (agents x candidates), so an
+      # unbounded fan-out would put a request per pair in flight at once --
+      # six agents is already 36 concurrent LLM calls. Cap it.
+      DEFAULT_MAX_CONCURRENCY = 8
+
       def execute
         log_execution_start
         @logger.info 'Consensual execution - agents propose, vote, and pick'
@@ -458,23 +463,47 @@ module RCrewAI
         chosen
       end
 
+      # Proposals are independent LLM calls, so they run concurrently: with the
+      # default three agents this turns three sequential round-trips into one.
+      # filter_map order is preserved so ties still break toward the assigned
+      # agent deterministically.
       def gather_proposals(task, participants)
-        participants.filter_map do |agent|
+        bounded_parallel(participants.map { |a| [a] }) do |agent|
           content = extract_content(agent.execute_task(task, stream: crew.stream_sink))
           { agent: agent, content: content }
         rescue StandardError => e
           @logger.warn "Agent #{agent.name} failed to propose: #{e.message}"
           nil
-        end
+        end.compact
       end
 
+      # Every (candidate, voter) pair is an independent LLM call. Scoring is the
+      # quadratic half of consensus -- three agents means nine calls -- so it
+      # fans out over the whole grid rather than per candidate.
       def score_candidates(task, candidates, participants)
-        candidates.map do |candidate|
-          total = participants.sum { |voter| score(voter, task, candidate[:content]) }
-          candidate.merge(score: total)
+        pairs = candidates.flat_map { |c| participants.map { |v| [c, v] } }
+        scores = bounded_parallel(pairs) { |c, v| score(v, task, c[:content]) }
+
+        candidates.each_with_index.map do |candidate, i|
+          row = scores[(i * participants.length), participants.length]
+          candidate.merge(score: row.sum)
+        end
+      end
+
+      # Runs the block over items in bounded parallel, preserving input order.
+      def bounded_parallel(items)
+        span = Events.current_parent
+        items.each_slice(max_concurrency).flat_map do |slice|
+          slice.map { |item| Thread.new { Events.with_parent(span) { yield(*item) } } }
+               .map(&:value)
         end
       end
 
+      def max_concurrency
+        crew.respond_to?(:consensus_max_concurrency) && crew.consensus_max_concurrency ||
+          DEFAULT_MAX_CONCURRENCY
+      end
+
       def score(voter, task, candidate_content)
         prompt = <<~PROMPT
           Score how well the following answer satisfies the task, from 0 to 10.

data/lib/rcrewai/tool_runner.rb

@@ -7,14 +7,18 @@ require_relative 'provider_schema'
 module RCrewAI
   class ToolRunner
     DEFAULT_MAX_ITERATIONS = 10
+    DEFAULT_TOOL_CONCURRENCY = 8
 
-    def initialize(agent:, llm:, tools:, max_iterations: DEFAULT_MAX_ITERATIONS, event_sink: nil)
+    def initialize(agent:, llm:, tools:, **opts)
       @agent = agent
       @llm = llm
       @tools = tools
       @tools_by_name = tools.each_with_object({}) { |t, h| h[t.name] = t }
-      @max_iterations = max_iterations
-      @sink = event_sink || ->(_) {}
+      @max_iterations = opts.fetch(:max_iterations, DEFAULT_MAX_ITERATIONS)
+      @sink = opts[:event_sink] || ->(_) {}
+      @parallel_tools = opts.fetch(:parallel_tools, true)
+      @max_tool_concurrency = opts.fetch(:max_tool_concurrency, DEFAULT_TOOL_CONCURRENCY)
+      @tool_usage_lock = Mutex.new
     end
 
     def run(messages:)
@@ -23,7 +27,7 @@ module RCrewAI
 
     private
 
-    def run_loop(messages:) # rubocop:disable Metrics/AbcSize
+    def run_loop(messages:)
       msgs = messages.dup
       history = []
       iter = 0
@@ -48,34 +52,9 @@ module RCrewAI
 
         msgs << { role: 'assistant', content: response[:content], tool_calls: response[:tool_calls] }
 
-        response[:tool_calls].each do |tc|
-          tool = @tools_by_name[tc[:name]]
-          emit(Events::ToolCallStart, iteration: iter,
-                                      tool: tc[:name], args: tc[:arguments], call_id: tc[:id])
-
-          if tool.nil?
-            err = "tool not found: #{tc[:name]}"
-            emit(Events::ToolCallError, iteration: iter,
-                                        tool: tc[:name], call_id: tc[:id], error: err)
-            msgs << tool_result_message(tc[:id], "ERROR: #{err}")
-            next
-          end
-
-          started = monotonic_ms
-          begin
-            result = tool.execute_with_validation(tc[:arguments] || {})
-            duration = monotonic_ms - started
-            @agent.memory.add_tool_usage(tc[:name], tc[:arguments], result) if @agent.respond_to?(:memory) && @agent.memory
-            emit(Events::ToolCallResult, iteration: iter,
-                                         tool: tc[:name], call_id: tc[:id], result: result,
-                                         duration_ms: duration)
-            history << { tool: tc[:name], args: tc[:arguments], result: result, duration_ms: duration }
-            msgs << tool_result_message(tc[:id], result.to_s)
-          rescue StandardError => e
-            emit(Events::ToolCallError, iteration: iter,
-                                        tool: tc[:name], call_id: tc[:id], error: e.message)
-            msgs << tool_result_message(tc[:id], "ERROR: #{e.message}")
-          end
+        run_tool_calls(response[:tool_calls], iter).each do |outcome|
+          history << outcome[:history] if outcome[:history]
+          msgs << outcome[:message]
         end
 
         emit(Events::IterationEnd, iteration: iter, finish_reason: :tool_calls)
@@ -95,6 +74,72 @@ module RCrewAI
       { role: 'tool', tool_call_id: call_id, content: content }
     end
 
+    # Executes one turn's tool calls, concurrently when there is more than one.
+    #
+    # Models routinely request several independent tools in a single turn;
+    # running them in sequence makes the turn cost the sum of their latencies
+    # rather than the max. Results are collected by index, so the order the
+    # model asked for is preserved no matter which finishes first -- the
+    # message history must stay aligned with the tool_call ids.
+    def run_tool_calls(tool_calls, iter)
+      return tool_calls.map { |tc| execute_tool_call(tc, iter) } unless parallel?(tool_calls)
+
+      # Events are emitted from worker threads, so carry the run span across
+      # the boundary: Events.with_parent is thread-local by design.
+      span = Events.current_parent
+      tool_calls.each_slice(@max_tool_concurrency).flat_map do |slice|
+        slice.map { |tc| Thread.new { Events.with_parent(span) { execute_tool_call(tc, iter) } } }
+             .map(&:value)
+      end
+    end
+
+    def parallel?(tool_calls)
+      @parallel_tools && tool_calls.length > 1
+    end
+
+    # Runs one tool call and returns what the caller should record. Never
+    # raises: a failing tool becomes an ERROR message fed back to the model,
+    # exactly as it did when this ran inline.
+    def execute_tool_call(call, iter)
+      tool = @tools_by_name[call[:name]]
+      emit(Events::ToolCallStart, iteration: iter,
+                                  tool: call[:name], args: call[:arguments], call_id: call[:id])
+
+      if tool.nil?
+        err = "tool not found: #{call[:name]}"
+        emit(Events::ToolCallError, iteration: iter,
+                                    tool: call[:name], call_id: call[:id], error: err)
+        return { message: tool_result_message(call[:id], "ERROR: #{err}") }
+      end
+
+      started = monotonic_ms
+      begin
+        result = tool.execute_with_validation(call[:arguments] || {})
+        duration = monotonic_ms - started
+        record_tool_usage(call, result)
+        emit(Events::ToolCallResult, iteration: iter,
+                                     tool: call[:name], call_id: call[:id], result: result,
+                                     duration_ms: duration)
+        {
+          history: { tool: call[:name], args: call[:arguments], result: result, duration_ms: duration },
+          message: tool_result_message(call[:id], result.to_s)
+        }
+      rescue StandardError => e
+        emit(Events::ToolCallError, iteration: iter,
+                                    tool: call[:name], call_id: call[:id], error: e.message)
+        { message: tool_result_message(call[:id], "ERROR: #{e.message}") }
+      end
+    end
+
+    # Agent memory is shared across the worker threads of one turn.
+    def record_tool_usage(call, result)
+      return unless @agent.respond_to?(:memory) && @agent.memory
+
+      @tool_usage_lock.synchronize do
+        @agent.memory.add_tool_usage(call[:name], call[:arguments], result)
+      end
+    end
+
     def emit(klass, iteration:, **attrs)
       type_sym = klass.name.split('::').last
                       .gsub(/([A-Z])/) { "_#{Regexp.last_match(1).downcase}" }

data/lib/rcrewai/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module RCrewAI
-  VERSION = '0.8.1'
+  VERSION = '0.10.0'
 end

data/lib/rcrewai.rb

@@ -21,6 +21,8 @@ require_relative 'rcrewai/configuration'
 require_relative 'rcrewai/events'
 require_relative 'rcrewai/checkpoint'
 require_relative 'rcrewai/sse_parser'
+require_relative 'rcrewai/cancel_token'
+require_relative 'rcrewai/persistent_http'
 require_relative 'rcrewai/pricing'
 require_relative 'rcrewai/llm_client'
 require_relative 'rcrewai/similarity'

data/rcrewai.gemspec

@@ -60,6 +60,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'thor', '~> 1.3'
 
   # Development dependencies
+  spec.add_development_dependency 'async', '~> 2.0'
   spec.add_development_dependency 'bundler', '~> 2.0'
   spec.add_development_dependency 'rake', '~> 13.0'
   spec.add_development_dependency 'rspec', '~> 3.12'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rcrewai
 version: !ruby/object:Gem::Version
-  version: 0.8.1
+  version: 0.10.0
 platform: ruby
 authors:
 - gkosmo
@@ -135,6 +135,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: async
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -347,6 +361,7 @@ files:
 - lib/rcrewai/agent.rb
 - lib/rcrewai/agent_augmentations.rb
 - lib/rcrewai/async_executor.rb
+- lib/rcrewai/cancel_token.rb
 - lib/rcrewai/checkpoint.rb
 - lib/rcrewai/checkpoint/cli.rb
 - lib/rcrewai/cli.rb
@@ -392,6 +407,7 @@ files:
 - lib/rcrewai/memory/tool_memory.rb
 - lib/rcrewai/multimodal.rb
 - lib/rcrewai/output_schema.rb
+- lib/rcrewai/persistent_http.rb
 - lib/rcrewai/planning.rb
 - lib/rcrewai/pricing.rb
 - lib/rcrewai/process.rb