async-enumerable 0.1.0

data/docs/reference/concurrency_bounder.md

@@ -0,0 +1,234 @@
+# ConcurrencyBounder Module
+
+The `ConcurrencyBounder` module provides bounded concurrency control for async operations.
+
+## Overview
+
+ConcurrencyBounder provides a helper method for executing async operations with a maximum fiber limit to prevent unbounded concurrency. This module is included in `Async::Enumerable` to provide a consistent way to limit the number of concurrent fibers created during async operations.
+
+## Core Method
+
+### with_bounded_concurrency
+
+Executes a block with bounded concurrency using a semaphore.
+
+This method sets up an `Async::Semaphore` to limit the number of concurrent fibers, creates a barrier under that semaphore, and yields the barrier to the block for spawning async tasks.
+
+#### Parameters
+
+- `early_termination` (Boolean): Whether the operation supports early termination (expects Async::Stop exceptions)
+- `&block`: Block that receives the barrier for spawning tasks
+
+#### Yields
+
+- `barrier` (Async::Barrier): The barrier to use for spawning async tasks
+
+#### Returns
+
+The result of the block execution.
+
+#### Example Usage
+
+```ruby
+def parallel_map(&block)
+  results = []
+  
+  with_bounded_concurrency do |barrier|
+    @items.each_with_index do |item, index|
+      barrier.async do
+        results[index] = block.call(item)
+      end
+    end
+  end
+  
+  results
+end
+```
+
+### max_fibers
+
+Gets the maximum number of fibers for this instance.
+
+#### Returns
+
+`Integer` - The maximum number of concurrent fibers
+
+#### Behavior
+
+1. Returns from instance config if set
+2. Falls back to class config if defined
+3. Falls back to `Async::Enumerable.config.max_fibers` module default
+4. Module default is 1024 if not configured
+
+## Implementation Details
+
+### Semaphore Usage
+
+The module uses `Async::Semaphore` to enforce fiber limits:
+
+```ruby
+semaphore = Async::Semaphore.new(max_fibers, parent:)
+barrier = Async::Barrier.new(parent: semaphore)
+```
+
+This ensures that no more than `max_fibers` tasks run concurrently.
+
+### Early Termination Support
+
+When `early_termination: true`:
+
+```ruby
+with_bounded_concurrency(early_termination: true) do |barrier|
+  # Tasks can call barrier.stop to terminate early
+  # Async::Stop exceptions are caught and handled
+end
+```
+
+This is used by predicate methods like `any?` and `find` to stop processing once the result is determined.
+
+### Normal Execution
+
+When `early_termination: false` (default):
+
+```ruby
+with_bounded_concurrency do |barrier|
+  # All tasks run to completion
+  # barrier.wait blocks until all finish
+end
+```
+
+## Usage in Async::Enumerable
+
+### In Predicate Methods
+
+```ruby
+def any?(&block)
+  found = Concurrent::AtomicBoolean.new(false)
+  
+  with_bounded_concurrency(early_termination: true) do |barrier|
+    @items.each do |item|
+      break if found.true?
+      
+      barrier.async do
+        if block.call(item)
+          found.make_true
+          barrier.stop  # Early termination
+        end
+      end
+    end
+  end
+  
+  found.true?
+end
+```
+
+### In Transform Methods
+
+```ruby
+def each(&block)
+  with_bounded_concurrency do |barrier|
+    @items.each do |item|
+      barrier.async do
+        block.call(item)
+      end
+    end
+  end
+end
+```
+
+## Fiber Limit Configuration
+
+### Global Default
+
+```ruby
+Async::Enumerable.configure { |c| c.max_fibers = 100 }  # Set global default
+```
+
+### Per Instance
+
+```ruby
+class MyEnumerator
+  include Async::Enumerable
+  def_async_enumerable :@items, max_fibers: 50  # Class-level default
+  
+  def initialize(items)
+    @items = items
+  end
+end
+
+# Instance override
+enumerator = MyEnumerator.new(items)
+enumerator.async(max_fibers: 100).map { |item| process(item) }
+```
+
+### Precedence
+
+1. Instance config (passed to `.async` method)
+2. Class config (set via `def_async_enumerable`)
+3. Module config (`Async::Enumerable.configure`)
+4. Default constant (1024)
+
+## Performance Considerations
+
+### Choosing Fiber Limits
+
+- **Too low**: Reduces parallelism, may not fully utilize resources
+- **Too high**: Can cause resource exhaustion, context switching overhead
+- **Recommended**: Start with defaults, tune based on workload
+
+### Typical Values
+
+- CPU-bound tasks: 2-4x CPU cores
+- I/O-bound tasks: 50-200 fibers
+- Mixed workloads: 10-50 fibers
+
+## Thread Safety
+
+The module operates within the async runtime which handles:
+- Fiber scheduling
+- Resource allocation
+- Synchronization via barriers
+
+Combined with atomic variables from concurrent-ruby, this ensures thread-safe operation.
+
+## Error Handling
+
+### Normal Errors
+
+Exceptions in async tasks propagate normally:
+
+```ruby
+with_bounded_concurrency do |barrier|
+  barrier.async do
+    raise "Error!"  # Will propagate after barrier.wait
+  end
+end
+```
+
+### Early Termination
+
+`Async::Stop` exceptions are caught when `early_termination: true`:
+
+```ruby
+with_bounded_concurrency(early_termination: true) do |barrier|
+  barrier.async do
+    barrier.stop  # Raises Async::Stop internally
+  end
+end
+# Async::Stop is caught, execution continues
+```
+
+## Integration with Async Runtime
+
+ConcurrencyBounder requires the async runtime via `Sync` blocks:
+
+```ruby
+def with_bounded_concurrency(...)
+  Sync do |parent|
+    # Async context established
+    # Semaphore and barrier created with parent
+  end
+end
+```
+
+This ensures proper async context even when called from synchronous code.

data/docs/reference/enumerable.md

@@ -0,0 +1,258 @@
+# Async::Enumerable Module
+
+The `Async::Enumerable` module provides asynchronous, parallel execution capabilities for Ruby's Enumerable.
+
+## Overview
+
+This module extends Ruby's Enumerable with an `.async` method that returns an Async::Enumerator wrapper, enabling concurrent execution of enumerable operations using the socketry/async library. This allows for significant performance improvements when dealing with I/O-bound operations or processing large collections.
+
+## Features
+
+- Parallel execution of enumerable methods
+- Thread-safe operation with atomic variables
+- Optimized early-termination implementations for predicates and find operations
+- Full compatibility with standard Enumerable interface
+- Configurable concurrency limits to prevent unbounded fiber creation
+
+## Basic Usage
+
+### Including in Your Class
+
+```ruby
+class MyCollection
+  include Async::Enumerable
+  def_async_enumerable :@items
+  
+  def initialize
+    @items = []
+  end
+  
+  attr_reader :items
+end
+
+collection = MyCollection.new
+collection.items.concat([1, 2, 3])
+collection.async.map { |x| x * 2 }  # => [2, 4, 6]
+```
+
+### Using with Arrays and Standard Collections
+
+```ruby
+# Basic async enumeration
+[1, 2, 3, 4, 5].async.map { |n| n * 2 }
+# => [2, 4, 6, 8, 10] (processed in parallel)
+
+# Async I/O operations
+urls = ["http://api1.com", "http://api2.com", "http://api3.com"]
+results = urls.async.map { |url| fetch_data(url) }
+# All URLs fetched concurrently
+```
+
+## The def_async_enumerable Method
+
+The `def_async_enumerable` class method defines the source of enumeration for async operations.
+
+### Syntax
+
+```ruby
+def_async_enumerable :collection_ref, max_fibers: nil
+```
+
+### Parameters
+
+- `collection_ref` (Symbol): The name of the method or instance variable that returns the enumerable
+- `max_fibers` (Integer, optional): Default max_fibers for this class
+
+### Examples
+
+#### With Method
+
+```ruby
+class DataProcessor
+  include Async::Enumerable
+  def_async_enumerable :dataset
+  
+  def dataset
+    fetch_data_from_source
+  end
+end
+```
+
+#### With Instance Variable
+
+```ruby
+class Queue
+  include Async::Enumerable
+  def_async_enumerable :@items  # Note the @ prefix
+  
+  def initialize
+    @items = []
+  end
+end
+```
+
+#### With Custom Fiber Limit
+
+```ruby
+class LargeDataset
+  include Async::Enumerable
+  def_async_enumerable :@records, max_fibers: 50
+  
+  attr_reader :records
+end
+```
+
+## Idempotent Async Chaining
+
+The `.async` method is idempotent - calling it multiple times returns the same instance:
+
+```ruby
+arr = [1, 2, 3]
+async1 = arr.async
+async2 = async1.async
+async3 = async2.async
+
+async1.equal?(async2)  # => true
+async2.equal?(async3)  # => true
+```
+
+This prevents unnecessary wrapper creation and allows for flexible API design.
+
+## Fiber Limits
+
+### Global Default
+
+Set the global default maximum fibers:
+
+```ruby
+Async::Enumerable.configure { |c| c.max_fibers = 100 }
+```
+
+### Per-Instance
+
+Override for specific calls:
+
+```ruby
+huge_dataset.async(max_fibers: 50).map { |item| process(item) }
+```
+
+### Default Value
+
+The default maximum is 1024 fibers if not explicitly configured.
+
+## Method Categories
+
+When you include `Async::Enumerable`, you get:
+
+### Predicate Methods (Early Termination)
+- `all?`, `any?`, `none?`, `one?`
+- `find`, `find_index`
+- `include?`, `member?`
+
+These methods stop processing as soon as the result is determined.
+
+### Transformer Methods
+- `map`, `select`, `reject`
+- `filter_map`, `flat_map`
+- `compact`, `uniq`
+- `sort`, `sort_by`
+
+These return new `Async::Enumerator` instances for chaining.
+
+### Converter Methods
+- `to_a` - Convert to array
+- `sync` - Alias for `to_a`, semantically ends async chain
+
+## Implementation Details
+
+### Module Inclusion
+
+When included, `Async::Enumerable` automatically:
+1. Extends with `Configurable` for configuration management
+2. Extends with `Configurable::ClassMethods` (provides `def_async_enumerable`)
+3. Includes `Configurable` for instance-level configuration
+4. Includes `Comparable` for comparison operators
+5. Includes all async method implementations
+6. Includes `AsyncMethod` module that provides the `async` method
+
+### Source Resolution
+
+The enumerable source is determined by:
+1. If `def_async_enumerable` was called, uses that source
+2. If source is an instance variable (starts with @), uses `instance_variable_get`
+3. If source is a method name, calls that method
+4. If no source defined, assumes self is enumerable
+
+## Common Patterns
+
+### Processing API Responses
+
+```ruby
+class ApiClient
+  include Async::Enumerable
+  def_async_enumerable :endpoints
+  
+  def endpoints
+    ["users", "posts", "comments"]
+  end
+  
+  def fetch_all
+    async.map { |endpoint| fetch("/api/#{endpoint}") }
+  end
+end
+```
+
+### Batch Processing
+
+```ruby
+class BatchProcessor
+  include Async::Enumerable
+  def_async_enumerable :@items, max_fibers: 10
+  
+  def initialize(items)
+    @items = items
+  end
+  
+  def process_all
+    async.map { |item| expensive_operation(item) }
+  end
+end
+```
+
+### Custom Collections
+
+```ruby
+class ThreadSafeQueue
+  include Async::Enumerable
+  def_async_enumerable :snapshot
+  
+  def initialize
+    @mutex = Mutex.new
+    @items = []
+  end
+  
+  def snapshot
+    @mutex.synchronize { @items.dup }
+  end
+  
+  def add(item)
+    @mutex.synchronize { @items << item }
+  end
+end
+```
+
+## Performance Considerations
+
+- Async processing has overhead - best for I/O-bound or CPU-intensive operations
+- Small collections with simple operations may be slower async
+- Early termination methods provide significant optimization
+- Fiber limits prevent resource exhaustion
+
+## Thread Safety
+
+All async operations use thread-safe atomic variables from concurrent-ruby:
+- `Concurrent::AtomicBoolean` for boolean flags
+- `Concurrent::AtomicFixnum` for counters
+- `Concurrent::AtomicReference` for object references
+
+This ensures correct behavior even with concurrent fiber execution.

data/docs/reference/enumerator.md

@@ -0,0 +1,221 @@
+# Async::Enumerator
+
+The `Async::Enumerator` class is a wrapper that provides asynchronous implementations of Enumerable methods for parallel execution.
+
+## Overview
+
+This class wraps any Enumerable object and provides async versions of standard enumerable methods. It includes the standard Enumerable module for compatibility, as well as specialized async implementations through the Async::Enumerable module.
+
+The Enumerator maintains a reference to the original enumerable and delegates method calls while providing concurrent execution capabilities through the async runtime.
+
+## Creating an Async::Enumerator
+
+### Direct Instantiation
+
+```ruby
+async_enum = Async::Enumerator.new([1, 2, 3, 4, 5])
+async_enum.map { |n| n * 2 }  # Executes in parallel
+```
+
+### Using Enumerable#async (Preferred)
+
+The preferred way to create an Async::Enumerator:
+
+```ruby
+result = [1, 2, 3].async.map { |n| slow_operation(n) }
+```
+
+### With Configuration
+
+```ruby
+# With custom fiber limit
+huge_dataset.async(max_fibers: 100).map { |n| process(n) }
+
+# With config object
+config = Async::Enumerable::Configurable::Config.new(max_fibers: 50)
+enumerator = Async::Enumerator.new(data, config)
+```
+
+## Initialization
+
+### Parameters
+
+- `enumerable` (Enumerable): Any object that includes Enumerable
+- `config` (Config, nil): Configuration object containing settings like max_fibers
+- `**kwargs`: Configuration options passed as keyword arguments (e.g., max_fibers: 100)
+
+### Examples
+
+```ruby
+# Default configuration
+async_array = Async::Enumerator.new([1, 2, 3])
+
+# Custom fiber limit via kwargs
+async_range = Async::Enumerator.new(1..100, max_fibers: 50)
+
+# With config object
+config = Async::Enumerable::Configurable::Config.new(max_fibers: 100)
+async_enum = Async::Enumerator.new(data, config)
+
+# Override config with kwargs
+async_enum = Async::Enumerator.new(data, config, max_fibers: 200)
+```
+
+## Core Methods
+
+### each
+
+Asynchronously iterates over each element in the enumerable, executing the given block in parallel for each item.
+
+This method spawns async tasks for each item in the enumerable, allowing them to execute concurrently. It uses an `Async::Barrier` to coordinate the tasks and waits for all of them to complete before returning.
+
+When called without a block, returns an Enumerator for compatibility with the standard Enumerable interface.
+
+#### Parameters
+- `&block`: Block to execute for each element in parallel
+
+#### Returns
+- With block: Returns self (for chaining)
+- Without block: Returns an Enumerator
+
+#### Examples
+
+```ruby
+# Basic async iteration
+[1, 2, 3].async.each do |n|
+  puts "Processing #{n}"
+  sleep(1)  # All three will complete in ~1 second total
+end
+
+# With I/O operations
+urls.async.each do |url|
+  response = HTTP.get(url)
+  save_to_cache(url, response)
+end
+# All URLs are fetched and cached concurrently
+
+# Chaining
+data.async
+    .each { |item| log(item) }
+    .map { |item| transform(item) }
+```
+
+#### Important Notes
+- The execution order of the block is not guaranteed to match the order of items in the enumerable due to parallel execution
+- All tasks complete before the method returns
+- Returns self to allow chaining, like standard each
+
+### Comparison Methods
+
+#### <=>
+
+Compares this Async::Enumerator with another object. Converts both to arrays for comparison.
+
+```ruby
+async_enum = [1, 2, 3].async
+async_enum <=> [1, 2, 3]  # => 0
+async_enum <=> [1, 2, 4]  # => -1
+```
+
+#### ==
+
+Checks equality with another object. Converts both to arrays for comparison.
+
+```ruby
+result = [1, 2, 3].async.map { |x| x * 2 }
+result == [2, 4, 6]  # => true
+```
+
+Also available as `eql?`.
+
+## Delegated Methods
+
+The following methods are inherently sequential and are delegated back to the wrapped enumerable for efficiency:
+
+- `first` - Returns the first element(s)
+- `take` - Takes the first n elements
+- `take_while` - Takes elements while condition is true
+- `lazy` - Returns a lazy enumerator
+- `size` - Returns the size of the enumerable
+- `length` - Alias for size
+
+These methods bypass async processing since they don't benefit from parallelization.
+
+## Method Chaining
+
+Async::Enumerator supports full method chaining:
+
+```ruby
+result = [1, 2, 3, 4, 5].async
+  .map { |n| fetch_data(n) }      # Parallel fetch
+  .select { |data| data.valid? }   # Filter results
+  .map { |data| process(data) }    # Transform data
+  .sync                            # Materialize results
+```
+
+## Fiber Limits
+
+The `max_fibers` parameter controls concurrency:
+
+```ruby
+# Default limit (from Async::Enumerable.max_fibers)
+data.async.map { |x| process(x) }
+
+# Custom limit for this instance
+data.async(max_fibers: 10).map { |x| process(x) }
+
+# Limit is preserved through chaining
+enum = data.async(max_fibers: 5)
+enum.map { |x| x * 2 }.select { |x| x > 10 }
+# Both map and select respect the 5 fiber limit
+```
+
+## Integration with Async Runtime
+
+Async::Enumerator requires the async runtime to be available. Operations are automatically wrapped in `Sync` blocks when needed:
+
+```ruby
+# Automatically wrapped in Sync block
+result = [1, 2, 3].async.map { |n| n * 2 }
+
+# Explicit async context
+Async do
+  result = urls.async.map { |url| fetch(url) }
+  process_results(result)
+end
+```
+
+## Performance Considerations
+
+- Async processing has overhead - use for I/O-bound or CPU-intensive operations
+- For simple transformations on small collections, synchronous processing may be faster
+- Fiber limits prevent resource exhaustion with large collections
+- Early termination methods (any?, all?, find) stop processing as soon as possible
+
+## Common Patterns
+
+### Parallel API Calls
+```ruby
+user_ids.async.map { |id| fetch_user(id) }
+```
+
+### Concurrent File Processing
+```ruby
+files.async.each { |file| process_file(file) }
+```
+
+### Batch Processing with Limits
+```ruby
+huge_dataset.async(max_fibers: 100).map { |item|
+  expensive_operation(item)
+}
+```
+
+### Pipeline Processing
+```ruby
+raw_data.async
+  .map { |d| parse(d) }
+  .select { |d| validate(d) }
+  .map { |d| transform(d) }
+  .sync
+```