lionagi 0.14.4__tar.gz → 0.14.6__tar.gz

{lionagi-0.14.4 → lionagi-0.14.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lionagi
-Version: 0.14.4
+Version: 0.14.6
 Summary: An Intelligence Operating System.
 Author-email: HaiyangLi <quantocean.li@gmail.com>, Liangbingyan Luo <llby_luo@outlook.com>
 License:            Apache License
@@ -225,7 +225,6 @@ Requires-Dist: anyio>=4.8.0
 Requires-Dist: backoff>=2.2.1
 Requires-Dist: jinja2>=3.1.0
 Requires-Dist: json-repair>=0.40.0
-Requires-Dist: matplotlib>=3.9.0
 Requires-Dist: pillow>=11.0.0
 Requires-Dist: psutil>=7.0.0
 Requires-Dist: pydantic-settings>=2.8.0
@@ -239,6 +238,8 @@ Requires-Dist: claude-code-sdk>=0.0.14; extra == 'all'
 Requires-Dist: datamodel-code-generator>=0.31.2; extra == 'all'
 Requires-Dist: docling>=2.15.1; extra == 'all'
 Requires-Dist: fastmcp>=2.10.5; extra == 'all'
+Requires-Dist: matplotlib>=3.7.0; extra == 'all'
+Requires-Dist: networkx>=3.0.0; extra == 'all'
 Requires-Dist: ollama>=0.4.0; extra == 'all'
 Requires-Dist: pydapter[postgres]; extra == 'all'
 Requires-Dist: rich>=13.0.0; extra == 'all'
@@ -248,6 +249,9 @@ Provides-Extra: docs
 Requires-Dist: furo>=2024.8.6; extra == 'docs'
 Requires-Dist: sphinx-autobuild>=2024.10.3; extra == 'docs'
 Requires-Dist: sphinx>=8.1.3; extra == 'docs'
+Provides-Extra: graph
+Requires-Dist: matplotlib>=3.7.0; extra == 'graph'
+Requires-Dist: networkx>=3.0.0; extra == 'graph'
 Provides-Extra: lint
 Requires-Dist: black[jupyter]>=24.10.0; extra == 'lint'
 Requires-Dist: isort>=5.13.2; extra == 'lint'

lionagi-0.14.6/docs/discussions/anyio-migration.md

@@ -0,0 +1,178 @@
+# Migration to AnyIO: A Technical Discussion
+
+## Overview
+
+In our latest release (v0.14.5), we've made a strategic decision to migrate our asynchronous primitives from raw `asyncio` to `anyio`. This document discusses the rationale, implications, and future benefits of this architectural change.
+
+## Why AnyIO?
+
+### 1. Backend Agnosticism
+
+AnyIO provides a unified interface that works with multiple async backends:
+- **asyncio** (current Python standard)
+- **trio** (structured concurrency pioneer)
+- **curio** (high-performance alternative)
+
+This abstraction layer means our codebase is no longer tightly coupled to asyncio's implementation details. As the Python async ecosystem evolves, we can adapt without major rewrites.
+
+### 2. Structured Concurrency by Default
+
+```python
+# Before (asyncio)
+tasks = []
+for item in items:
+    task = asyncio.create_task(process(item))
+    tasks.append(task)
+results = await asyncio.gather(*tasks)  # What if a task fails?
+
+# After (anyio)
+async with create_task_group() as tg:
+    for item in items:
+        await tg.start_soon(process, item)
+# All tasks are guaranteed to complete or cancel properly
+```
+
+Task groups enforce a fundamental principle: **no task outlives its parent scope**. This eliminates entire classes of bugs related to orphaned tasks and resource leaks.
+
+### 3. Better Cancellation Semantics
+
+AnyIO's cancellation model is more predictable:
+- Cancellation is always delivered at checkpoint
+- Cancel scopes provide fine-grained control
+- No more mysterious "Task was destroyed but it is pending!" errors
+
+```python
+# Timeout with proper cancellation
+with anyio.move_on_after(5.0) as cancel_scope:
+    result = await long_running_operation()
+if cancel_scope.cancelled_caught:
+    print("Operation timed out cleanly")
+```
+
+## Implementation Highlights
+
+### Thread-to-Async Bridge
+
+One subtle but important change is how we handle sync functions in async contexts:
+
+```python
+# Before
+result = await asyncio.to_thread(sync_func, arg)
+
+# After  
+result = await anyio.to_thread.run_sync(sync_func, arg)
+```
+
+AnyIO's approach provides:
+- Better thread pool management
+- Proper cancellation propagation to threads
+- Consistent behavior across different async backends
+
+### Unified Lock Semantics
+
+Our `ConcurrencyLock` now wraps AnyIO's primitives, providing:
+- Same lock works in both sync and async contexts
+- No more deadlocks from mixing lock types
+- Better debugging with lock ownership tracking
+
+### Sleep and Timing
+
+```python
+# Before
+await asyncio.sleep(1.0)
+
+# After
+await anyio.sleep(1.0)
+```
+
+While seemingly trivial, AnyIO's sleep:
+- Respects cancellation properly
+- Works consistently across backends
+- Integrates with structured concurrency
+
+## Performance Implications
+
+### The Good
+1. **Better CPU utilization**: Task groups reduce overhead compared to gather/wait
+2. **Memory efficiency**: Structured concurrency prevents task accumulation
+3. **Predictable performance**: Consistent behavior reduces edge-case slowdowns
+
+### The Trade-offs
+1. **Slight overhead**: Abstraction layer adds minimal overhead (~5-10%)
+2. **Learning curve**: Developers need to understand structured concurrency
+3. **Ecosystem compatibility**: Some asyncio-specific libraries need adapters
+
+## Future Benefits
+
+### 1. Trio Compatibility
+We can now experiment with trio as a backend for specific use cases:
+```python
+# Run lionagi with trio backend
+import trio
+import anyio
+
+async def main():
+    async with anyio.from_thread.start_blocking_portal(
+        backend="trio", backend_options={"trio_token": trio.lowlevel.current_trio_token()}
+    ) as portal:
+        await portal.call(run_lionagi_operation)
+```
+
+### 2. Better Testing
+AnyIO provides excellent testing utilities:
+```python
+async def test_timeout_handling():
+    with anyio.move_on_after(0.1):
+        await anyio.sleep(1.0)  # Will be cancelled
+    # Test continues normally
+```
+
+### 3. WebAssembly Ready
+As Python moves toward WASM support, AnyIO's abstraction will help us adapt to environments where traditional threading doesn't exist.
+
+## Migration Patterns
+
+### For Library Users
+Most changes are transparent, but be aware of:
+- Different exception types (use `anyio.get_cancelled_exc_class()`)
+- Import changes if you were using our internal async utilities
+- Slightly different timeout behavior (more predictable)
+
+### for Library Developers
+Key patterns to adopt:
+```python
+# Always use task groups for concurrent operations
+async with create_task_group() as tg:
+    await tg.start_soon(task1)
+    await tg.start_soon(task2)
+
+# Use cancel scopes for timeouts
+with anyio.move_on_after(timeout):
+    result = await operation()
+
+# Handle backend-specific exceptions
+try:
+    await operation()
+except anyio.get_cancelled_exc_class():
+    # Properly handle cancellation
+    raise
+```
+
+## Philosophical Alignment
+
+This migration aligns with our core principles:
+
+1. **Reliability First**: Structured concurrency eliminates entire bug categories
+2. **Future-Proof Design**: Backend agnosticism protects against ecosystem changes  
+3. **Developer Experience**: Clearer mental models reduce cognitive load
+4. **Performance with Correctness**: We optimize without sacrificing safety
+
+## Conclusion
+
+The migration to AnyIO represents a maturation of our async architecture. While the immediate benefits include better resource management and fewer concurrency bugs, the long-term value lies in our ability to evolve with the Python async ecosystem.
+
+This change positions lionagi to take advantage of future innovations in async Python while providing our users with a more reliable and predictable experience today.
+
+---
+
+*For specific migration examples and API changes, see our [v0.14.6 Release Notes](./RELEASE_NOTES_v0.14.5.md).*

{lionagi-0.14.4 → lionagi-0.14.6}/docs/modules/branch_operations.rst

@@ -64,7 +64,7 @@ of action requests. Optionally retries, logs errors, or merges results.
 This is best used for **explicit function calls** triggered from user 
 instructions or system logic.
 
-.. method:: Branch.act(action_request, *, suppress_errors=True, sanitize_input=False, unique_input=False, num_retries=0, initial_delay=0, retry_delay=0, backoff_factor=1, retry_default=UNDEFINED, retry_timeout=None, retry_timing=False, max_concurrent=None, throttle_period=None, flatten=True, dropna=True, unique_output=False, flatten_tuple_set=False)
+.. method:: Branch.act(action_request, *, suppress_errors=True, sanitize_input=False, unique_input=False, num_retries=0, initial_delay=0, retry_delay=0, backoff_factor=1, retry_default=UNDEFINED, retry_timeout=None, max_concurrent=None, throttle_period=None, flatten=True, dropna=True, unique_output=False, flatten_tuple_set=False)
 
     Public, potentially batched, asynchronous interface to run one or multiple action requests.
 
@@ -90,8 +90,6 @@ instructions or system logic.
         Fallback value if all retries fail (if suppressing errors)
     retry_timeout : float | None, default=None
         Overall timeout for all attempts (None = no limit)
-    retry_timing : bool, default=False
-        If True, track time used for retries
     max_concurrent : int | None, default=None
         Maximum concurrent tasks (if batching)
     throttle_period : float | None, default=None

{lionagi-0.14.4 → lionagi-0.14.6}/lionagi/fields/instruct.py

@@ -35,7 +35,6 @@ class Instruct(HashableModel):
         "reason",
         "actions",
         "action_strategy",
-        "batch_size",
         "request_params",
         "response_params",
     ]
@@ -97,16 +96,10 @@ class Instruct(HashableModel):
             "None: Contextual execution."
         ),
     )
-    action_strategy: Literal["batch", "sequential", "concurrent"] | None = (
-        Field(
-            None,
-            description="Action strategy to use for executing actions. Default "
-            "is 'concurrent'. Only provide for if actions are enabled.",
-        )
-    )
-    batch_size: int | None = Field(
+    action_strategy: Literal["sequential", "concurrent"] | None = Field(
         None,
-        description="Batch size for executing actions. Only provide for 'batch' strategy.",
+        description="Action strategy to use for executing actions. Default "
+        "is 'concurrent'. Only provide for if actions are enabled.",
     )
 
     @field_validator("instruction", "guidance", "context", mode="before")
@@ -123,13 +116,6 @@ class Instruct(HashableModel):
             return "concurrent"
         return v
 
-    @field_validator("batch_size", mode="before")
-    def _validate_batch_size(cls, v):
-        try:
-            return to_num(v, num_type=int)
-        except Exception:
-            return None
-
 
 class InstructResponse(HashableModel):
     instruct: Instruct

{lionagi-0.14.4 → lionagi-0.14.6}/lionagi/libs/concurrency/__init__.py

@@ -1,4 +1,4 @@
-"""Structured concurrency primitives.
+"""Structured concurrency primitives for pynector.
 
 This module provides structured concurrency primitives using AnyIO,
 which allows for consistent behavior across asyncio and trio backends.
@@ -6,7 +6,21 @@ which allows for consistent behavior across asyncio and trio backends.
 
 from .cancel import CancelScope, fail_after, move_on_after
 from .errors import get_cancelled_exc_class, shield
+from .patterns import (
+    ConnectionPool,
+    WorkerPool,
+    parallel_requests,
+    retry_with_timeout,
+)
 from .primitives import CapacityLimiter, Condition, Event, Lock, Semaphore
+from .resource_tracker import (
+    ResourceTracker,
+    cleanup_check,
+    get_global_tracker,
+    resource_leak_detector,
+    track_resource,
+    untrack_resource,
+)
 from .task import TaskGroup, create_task_group
 
 __all__ = [
@@ -15,6 +29,10 @@ __all__ = [
     "CancelScope",
     "move_on_after",
     "fail_after",
+    "ConnectionPool",
+    "WorkerPool",
+    "parallel_requests",
+    "retry_with_timeout",
     "Lock",
     "Semaphore",
     "CapacityLimiter",
@@ -22,4 +40,10 @@ __all__ = [
     "Condition",
     "get_cancelled_exc_class",
     "shield",
+    "ResourceTracker",
+    "resource_leak_detector",
+    "track_resource",
+    "untrack_resource",
+    "cleanup_check",
+    "get_global_tracker",
 ]

{lionagi-0.14.4 → lionagi-0.14.6}/lionagi/libs/concurrency/cancel.py

@@ -4,7 +4,7 @@ import time
 from collections.abc import Iterator
 from contextlib import contextmanager
 from types import TracebackType
-from typing import Optional, TypeVar
+from typing import TypeVar
 
 import anyio
 

lionagi-0.14.6/lionagi/libs/concurrency/patterns.py

@@ -0,0 +1,259 @@
+"""Common concurrency patterns for structured concurrency."""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Awaitable, Callable
+from types import TracebackType
+from typing import Any, TypeVar
+
+import anyio
+
+from .cancel import move_on_after
+from .primitives import CapacityLimiter, Lock
+from .resource_tracker import track_resource, untrack_resource
+from .task import create_task_group
+
+logger = logging.getLogger(__name__)
+
+T = TypeVar("T")
+R = TypeVar("R")
+Response = TypeVar("Response")
+
+
+class ConnectionPool:
+    """A pool of reusable connections."""
+
+    def __init__(
+        self,
+        max_connections: int,
+        connection_factory: Callable[[], Awaitable[T]],
+    ):
+        """Initialize a new connection pool."""
+        if max_connections < 1:
+            raise ValueError("max_connections must be >= 1")
+        if not callable(connection_factory):
+            raise ValueError("connection_factory must be callable")
+
+        self._connection_factory = connection_factory
+        self._limiter = CapacityLimiter(max_connections)
+        self._connections: list[T] = []
+        self._lock = Lock()
+
+        track_resource(self, f"ConnectionPool-{id(self)}", "ConnectionPool")
+
+    def __del__(self):
+        """Clean up resource tracking."""
+        try:
+            untrack_resource(self)
+        except Exception:
+            pass
+
+    async def acquire(self) -> T:
+        """Acquire a connection from the pool."""
+        await self._limiter.acquire()
+
+        try:
+            async with self._lock:
+                if self._connections:
+                    return self._connections.pop()
+
+            # No pooled connection available, create new one
+            return await self._connection_factory()
+        except Exception:
+            self._limiter.release()
+            raise
+
+    async def release(self, connection: T) -> None:
+        """Release a connection back to the pool."""
+        try:
+            async with self._lock:
+                self._connections.append(connection)
+        finally:
+            self._limiter.release()
+
+    async def __aenter__(self) -> ConnectionPool[T]:
+        """Enter the connection pool context."""
+        return self
+
+    async def __aexit__(
+        self,
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> None:
+        """Exit the connection pool context."""
+        # Clean up any remaining connections
+        async with self._lock:
+            self._connections.clear()
+
+
+async def parallel_requests(
+    inputs: list[str],
+    func: Callable[[str], Awaitable[Response]],
+    max_concurrency: int = 10,
+) -> list[Response]:
+    """Execute requests in parallel with controlled concurrency.
+
+    Args:
+        inputs: List of inputs
+        fetch_func: Async function
+        max_concurrency: Maximum number of concurrent requests
+
+    Returns:
+        List of responses in the same order as inputs
+    """
+    if not inputs:
+        return []
+
+    results: list[Response | None] = [None] * len(inputs)
+
+    async def bounded_fetch(
+        semaphore: anyio.Semaphore, idx: int, url: str
+    ) -> None:
+        async with semaphore:
+            results[idx] = await func(url)
+
+    try:
+        async with create_task_group() as tg:
+            semaphore = anyio.Semaphore(max_concurrency)
+
+            for i, inp in enumerate(inputs):
+                await tg.start_soon(bounded_fetch, semaphore, i, inp)
+    except BaseException as e:
+        # Re-raise the first exception directly instead of ExceptionGroup
+        if hasattr(e, "exceptions") and e.exceptions:
+            raise e.exceptions[0]
+        else:
+            raise
+
+    return results  # type: ignore
+
+
+async def retry_with_timeout(
+    func: Callable[[], Awaitable[T]],
+    max_retries: int = 3,
+    timeout: float = 30.0,
+    backoff_factor: float = 1.0,
+) -> T:
+    """Retry an async function with exponential backoff and timeout.
+
+    Args:
+        func: The async function to retry
+        max_retries: Maximum number of retries
+        timeout: Timeout for each attempt
+        backoff_factor: Multiplier for exponential backoff
+
+    Returns:
+        The result of the successful function call
+
+    Raises:
+        Exception: The last exception raised by the function
+    """
+    last_exception = None
+
+    for attempt in range(max_retries):
+        try:
+            with move_on_after(timeout) as cancel_scope:
+                result = await func()
+                if not cancel_scope.cancelled_caught:
+                    return result
+                else:
+                    raise TimeoutError(f"Function timed out after {timeout}s")
+        except Exception as e:
+            last_exception = e
+            if attempt < max_retries - 1:
+                delay = backoff_factor * (2**attempt)
+                await anyio.sleep(delay)
+            continue
+
+    if last_exception:
+        raise last_exception
+    else:
+        raise RuntimeError("Retry failed without capturing exception")
+
+
+class WorkerPool:
+    """A pool of worker tasks that process items from a queue."""
+
+    def __init__(
+        self, num_workers: int, worker_func: Callable[[Any], Awaitable[None]]
+    ):
+        """Initialize a new worker pool."""
+        if num_workers < 1:
+            raise ValueError("num_workers must be >= 1")
+        if not callable(worker_func):
+            raise ValueError("worker_func must be callable")
+
+        self._num_workers = num_workers
+        self._worker_func = worker_func
+        self._queue = anyio.create_memory_object_stream(1000)
+        self._task_group = None
+
+        track_resource(self, f"WorkerPool-{id(self)}", "WorkerPool")
+
+    def __del__(self):
+        """Clean up resource tracking."""
+        try:
+            untrack_resource(self)
+        except Exception:
+            pass
+
+    async def start(self) -> None:
+        """Start the worker pool."""
+        if self._task_group is not None:
+            raise RuntimeError("Worker pool is already started")
+
+        self._task_group = create_task_group()
+        await self._task_group.__aenter__()
+
+        # Start worker tasks
+        for i in range(self._num_workers):
+            await self._task_group.start_soon(self._worker_loop)
+
+    async def stop(self) -> None:
+        """Stop the worker pool."""
+        if self._task_group is None:
+            return
+
+        # Close the queue to signal workers to stop
+        await self._queue[0].aclose()
+
+        # Wait for all workers to finish
+        try:
+            await self._task_group.__aexit__(None, None, None)
+        finally:
+            self._task_group = None
+
+    async def submit(self, item: Any) -> None:
+        """Submit an item for processing."""
+        if self._task_group is None:
+            raise RuntimeError("Worker pool is not started")
+        await self._queue[0].send(item)
+
+    async def _worker_loop(self) -> None:
+        """Main loop for worker tasks."""
+        try:
+            async with self._queue[1]:
+                async for item in self._queue[1]:
+                    try:
+                        await self._worker_func(item)
+                    except Exception as e:
+                        logger.error(f"Worker error processing item: {e}")
+        except anyio.ClosedResourceError:
+            # Queue was closed, worker should exit gracefully
+            pass
+
+    async def __aenter__(self) -> WorkerPool:
+        """Enter the worker pool context."""
+        await self.start()
+        return self
+
+    async def __aexit__(
+        self,
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> None:
+        """Exit the worker pool context."""
+        await self.stop()