kailash 0.1.4__tar.gz → 0.2.0__tar.gz

{kailash-0.1.4/src/kailash.egg-info → kailash-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: kailash
-Version: 0.1.4
+Version: 0.2.0
 Summary: Python SDK for the Kailash container-node architecture
 Home-page: https://github.com/integrum/kailash-python-sdk
 Author: Integrum
@@ -47,6 +47,13 @@ Requires-Dist: pytest-asyncio>=1.0.0
 Requires-Dist: pre-commit>=4.2.0
 Requires-Dist: twine>=6.1.0
 Requires-Dist: ollama>=0.5.1
+Requires-Dist: sqlalchemy>=2.0.0
+Requires-Dist: websockets>=12.0
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: python-jose>=3.5.0
+Requires-Dist: pytest-xdist>=3.6.0
+Requires-Dist: pytest-timeout>=2.3.0
+Requires-Dist: pytest-split>=0.9.0
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0; extra == "dev"
 Requires-Dist: pytest-cov>=3.0; extra == "dev"
@@ -66,7 +73,7 @@ Dynamic: requires-python
   <a href="https://pepy.tech/project/kailash"><img src="https://static.pepy.tech/badge/kailash" alt="Downloads"></a>
   <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT License">
   <img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Code style: black">
-  <img src="https://img.shields.io/badge/tests-753%20passing-brightgreen.svg" alt="Tests: 753 passing">
+  <img src="https://img.shields.io/badge/tests-734%20passing-brightgreen.svg" alt="Tests: 734 passing">
   <img src="https://img.shields.io/badge/coverage-100%25-brightgreen.svg" alt="Coverage: 100%">
 </p>
 
@@ -92,6 +99,13 @@ Dynamic: requires-python
 - 🧠 **Retrieval-Augmented Generation**: Full RAG pipeline with intelligent document processing
 - 🌐 **REST API Wrapper**: Expose any workflow as a production-ready API in 3 lines
 - 🚪 **Multi-Workflow Gateway**: Manage multiple workflows through unified API with MCP integration
+- 🤖 **Self-Organizing Agents**: Autonomous agent pools with intelligent team formation and convergence detection
+- 🧠 **Agent-to-Agent Communication**: Shared memory pools and intelligent caching for coordinated multi-agent systems
+- 🔒 **Production Security**: Comprehensive security framework with path traversal prevention, code sandboxing, and audit logging
+- 🎨 **Visual Workflow Builder**: Kailash Workflow Studio - drag-and-drop interface for creating and managing workflows (coming soon)
+- 🔁 **Cyclic Workflows (v0.2.0)**: Universal Hybrid Cyclic Graph Architecture with 30,000+ iterations/second performance
+- 🛠️ **Developer Tools**: CycleAnalyzer, CycleDebugger, CycleProfiler for production-ready cyclic workflows
+- 📈 **High Performance**: Optimized execution engine supporting 100,000+ iteration workflows
 
 ## 🎯 Who Is This For?
 
@@ -276,6 +290,141 @@ results, run_id = runtime.execute(workflow)
 print("RAG Response:", results["llm_agent"]["response"])
 ```
 
+### Cyclic Workflows - Iterative Processing with Convergence
+
+Build workflows that iterate until a condition is met, perfect for optimization, retries, and ML training:
+
+```python
+from kailash.workflow import Workflow
+from kailash.nodes.base_cycle_aware import CycleAwareNode
+from kailash.nodes.base import NodeParameter
+from kailash.runtime.local import LocalRuntime
+from typing import Any, Dict
+
+# Create a custom cycle-aware node for data quality improvement
+class DataQualityImproverNode(CycleAwareNode):
+    def get_parameters(self) -> Dict[str, NodeParameter]:
+        return {
+            "data": NodeParameter(name="data", type=list, required=True),
+            "target_quality": NodeParameter(name="target_quality", type=float, required=False, default=0.95)
+        }
+
+    def run(self, context: Dict[str, Any], **kwargs) -> Dict[str, Any]:
+        """Iteratively improve data quality."""
+        data = kwargs["data"]
+        target_quality = kwargs.get("target_quality", 0.95)
+
+        # Get current iteration and previous state
+        iteration = self.get_iteration(context)
+        prev_state = self.get_previous_state(context)
+
+        # Calculate current quality
+        quality = prev_state.get("quality", 0.5) + 0.1  # Improve by 10% each iteration
+        quality = min(quality, 1.0)
+
+        # Process data (simplified)
+        processed_data = [item for item in data if item is not None]
+
+        # Track quality history
+        quality_history = self.accumulate_values(context, "quality_history", quality, max_history=10)
+
+        # Detect convergence trend
+        trend = self.detect_convergence_trend(context, "quality_history", window_size=5)
+        converged = quality >= target_quality or (trend and trend["slope"] < 0.01)
+
+        # Log progress
+        self.log_cycle_info(context, f"Iteration {iteration}: Quality={quality:.2%}")
+
+        # Save state for next iteration
+        self.set_cycle_state({"quality": quality, "processed_count": len(processed_data)})
+
+        return {
+            "data": processed_data,
+            "quality": quality,
+            "converged": converged,
+            "iteration": iteration,
+            "history": quality_history
+        }
+
+# Build cyclic workflow
+workflow = Workflow("quality_improvement", "Iterative Data Quality")
+workflow.add_node("improver", DataQualityImproverNode())
+
+# Create a cycle - node connects to itself!
+workflow.connect("improver", "improver",
+                 mapping={"data": "data"},  # Pass data to next iteration
+                 cycle=True,  # This is a cycle
+                 max_iterations=20,  # Safety limit
+                 convergence_check="converged == True")  # Stop condition
+
+# Execute with automatic iteration management
+runtime = LocalRuntime()
+results, run_id = runtime.execute(workflow, parameters={
+    "improver": {
+        "data": [1, None, 3, None, 5, 6, None, 8, 9, 10],
+        "target_quality": 0.9
+    }
+})
+
+print(f"Converged after {results['improver']['iteration']} iterations")
+print(f"Final quality: {results['improver']['quality']:.2%}")
+print(f"Quality history: {results['improver']['history']}")
+```
+
+### NEW in v0.2.0: CycleBuilder API
+
+The new CycleBuilder API provides a fluent interface for creating cyclic workflows:
+
+```python
+# Modern approach with CycleBuilder
+workflow.create_cycle("optimization_loop")
+    .connect("gradient", "optimizer")
+    .connect("optimizer", "evaluator")
+    .connect("evaluator", "gradient")
+    .max_iterations(100)
+    .converge_when("loss < 0.01")
+    .early_stop_when("gradient_norm < 1e-6")
+    .checkpoint_every(10)
+    .build()
+
+# Developer tools for production workflows
+from kailash.workflow import CycleAnalyzer, CycleDebugger, CycleProfiler
+
+# Analyze cycle patterns
+analyzer = CycleAnalyzer(workflow)
+report = analyzer.analyze()
+print(f"Found {len(report.cycles)} cycles")
+print(f"Max depth: {report.max_cycle_depth}")
+
+# Debug with breakpoints
+debugger = CycleDebugger(workflow)
+debugger.set_breakpoint("optimizer", iteration=50)
+debugger.set_trace("gradient_norm", lambda x: x < 0.001)
+
+# Profile performance
+profiler = CycleProfiler(workflow)
+profile_data = profiler.profile(runtime, parameters)
+print(f"Bottleneck: {profile_data.bottleneck_node}")
+print(f"Iterations/sec: {profile_data.iterations_per_second}")
+```
+
+#### Cyclic Workflow Features
+
+- **Built-in Iteration Management**: No manual loops or recursion needed
+- **State Persistence**: Maintain state across iterations with `get_previous_state()` and `set_cycle_state()`
+- **Convergence Detection**: Automatic trend analysis with `detect_convergence_trend()`
+- **Value Accumulation**: Track metrics over time with `accumulate_values()`
+- **Safety Limits**: Max iterations prevent infinite loops
+- **Performance**: Optimized execution with ~30,000 iterations/second
+- **Developer Tools**: CycleAnalyzer, CycleDebugger, CycleProfiler for production workflows
+
+Common cyclic patterns include:
+- **Retry with Backoff**: ETL pipelines with automatic retry
+- **Optimization Loops**: Iterative parameter tuning
+- **ML Training**: Training until accuracy threshold
+- **Polling**: API polling with rate limiting
+- **Stream Processing**: Windowed data processing
+
 ### Workflow API Wrapper - Expose Workflows as REST APIs
 
 Transform any Kailash workflow into a production-ready REST API in just 3 lines of code:
@@ -410,6 +559,173 @@ gateway.run(port=8000)
 
 See the [Gateway examples](examples/integration_examples/gateway_comprehensive_demo.py) for complete implementation patterns.
 
+### Self-Organizing Agent Pools - Autonomous Multi-Agent Systems
+
+Build intelligent agent systems that can autonomously form teams, share information, and solve complex problems collaboratively:
+
+```python
+from kailash import Workflow
+from kailash.runtime import LocalRuntime
+from kailash.nodes.ai.intelligent_agent_orchestrator import (
+    OrchestrationManagerNode,
+    IntelligentCacheNode,
+    ConvergenceDetectorNode
+)
+from kailash.nodes.ai.self_organizing import (
+    AgentPoolManagerNode,
+    TeamFormationNode,
+    ProblemAnalyzerNode
+)
+from kailash.nodes.ai.a2a import SharedMemoryPoolNode, A2AAgentNode
+
+# Create self-organizing agent workflow
+workflow = Workflow("self_organizing_research")
+
+# Shared infrastructure
+memory_pool = SharedMemoryPoolNode(
+    memory_size_limit=1000,
+    attention_window=50
+)
+workflow.add_node("memory", memory_pool)
+
+# Intelligent caching to prevent redundant operations
+cache = IntelligentCacheNode(
+    ttl=3600,  # 1 hour cache
+    similarity_threshold=0.8,
+    max_entries=1000
+)
+workflow.add_node("cache", cache)
+
+# Problem analysis and team formation
+problem_analyzer = ProblemAnalyzerNode()
+team_former = TeamFormationNode(
+    formation_strategy="capability_matching",
+    optimization_rounds=3
+)
+workflow.add_node("analyzer", problem_analyzer)
+workflow.add_node("team_former", team_former)
+
+# Self-organizing agent pool
+pool_manager = AgentPoolManagerNode(
+    max_active_agents=20,
+    agent_timeout=120
+)
+workflow.add_node("pool", pool_manager)
+
+# Convergence detection for stopping criteria
+convergence = ConvergenceDetectorNode(
+    quality_threshold=0.85,
+    improvement_threshold=0.02,
+    max_iterations=10
+)
+workflow.add_node("convergence", convergence)
+
+# Orchestration manager coordinates the entire system
+orchestrator = OrchestrationManagerNode(
+    max_iterations=10,
+    quality_threshold=0.85,
+    parallel_execution=True
+)
+workflow.add_node("orchestrator", orchestrator)
+
+# Execute with complex business problem
+runtime = LocalRuntime()
+result, _ = runtime.execute(workflow, parameters={
+    "orchestrator": {
+        "query": "Analyze market trends and develop growth strategy for fintech",
+        "agent_pool_size": 12,
+        "mcp_servers": [
+            {"name": "market_data", "command": "python", "args": ["-m", "market_mcp"]},
+            {"name": "financial", "command": "python", "args": ["-m", "finance_mcp"]},
+            {"name": "research", "command": "python", "args": ["-m", "research_mcp"]}
+        ],
+        "context": {
+            "domain": "fintech",
+            "depth": "comprehensive",
+            "output_format": "strategic_report"
+        }
+    }
+})
+
+print(f"Solution Quality: {result['orchestrator']['quality_score']:.2%}")
+print(f"Agents Used: {result['orchestrator']['agents_deployed']}")
+print(f"Iterations: {result['orchestrator']['iterations_completed']}")
+print(f"Final Strategy: {result['orchestrator']['final_solution']['strategy']}")
+```
+
+#### Key Self-Organizing Features
+
+- **Autonomous Team Formation**: Agents automatically form optimal teams based on:
+  - Capability matching for skill-specific tasks
+  - Swarm-based formation for exploration
+  - Market-based allocation for resource constraints
+  - Hierarchical organization for complex problems
+
+- **Intelligent Information Sharing**:
+  - **SharedMemoryPoolNode**: Selective attention mechanisms for relevant information
+  - **IntelligentCacheNode**: Semantic similarity detection prevents redundant operations
+  - **A2AAgentNode**: Direct agent-to-agent communication with context awareness
+
+- **Convergence Detection**: Automatic termination when:
+  - Solution quality exceeds threshold (e.g., 85% confidence)
+  - Improvement rate drops below minimum (e.g., <2% per iteration)
+  - Maximum iterations reached
+  - Time limits exceeded
+
+- **MCP Integration**: Agents can access external tools and data sources:
+  - File systems, databases, APIs
+  - Web scraping and research tools
+  - Specialized domain knowledge bases
+  - Real-time data streams
+
+- **Performance Optimization**:
+  - Multi-level caching strategies
+  - Parallel agent execution
+  - Resource management and monitoring
+  - Cost tracking for API usage
+
+See the [Self-Organizing Agents examples](examples/integration_examples/) for complete implementation patterns and the [Agent Systems Guide](docs/guides/self_organizing_agents.rst) for detailed documentation.
+
+### Zero-Code MCP Ecosystem - Visual Workflow Builder
+
+Build and deploy workflows through an interactive web interface without writing any code:
+
+```python
+from kailash.api.gateway import WorkflowAPIGateway
+from kailash.api.mcp_integration import MCPServerRegistry
+
+# Run the MCP ecosystem demo
+# cd examples/integration_examples
+# ./run_ecosystem.sh
+
+# Or run programmatically:
+python examples/integration_examples/mcp_ecosystem_demo.py
+```
+
+#### Features
+
+- **Drag-and-Drop Builder**: Visual interface for creating workflows
+  - Drag nodes from palette (CSV Reader, Python Code, JSON Writer, etc.)
+  - Drop onto canvas to build workflows
+  - Deploy with one click
+
+- **Live Dashboard**: Real-time monitoring and statistics
+  - Connected MCP server status
+  - Running workflow count
+  - Execution logs with timestamps
+
+- **Pre-built Templates**: One-click deployment
+  - GitHub → Slack Notifier
+  - Data Processing Pipeline (CSV → Transform → JSON)
+  - AI Research Assistant
+
+- **Technology Stack**: Lightweight and fast
+  - Backend: FastAPI + Kailash SDK
+  - Frontend: Vanilla HTML/CSS/JavaScript (no frameworks)
+  - Zero build process required
+
+See the [MCP Ecosystem example](examples/integration_examples/) for the complete zero-code workflow deployment platform.
+
 ## 📚 Documentation
 
 | Resource | Description |
@@ -471,6 +787,21 @@ The SDK includes a rich set of pre-built nodes for common operations:
 - `SentimentAnalyzer` - Sentiment analysis
 - `NamedEntityRecognizer` - NER extraction
 
+**Self-Organizing Agent Nodes**
+- `SharedMemoryPoolNode` - Agent memory sharing
+- `A2AAgentNode` - Agent-to-agent communication
+- `A2ACoordinatorNode` - Multi-agent coordination
+- `IntelligentCacheNode` - Semantic caching system
+- `MCPAgentNode` - MCP-enabled agents
+- `QueryAnalysisNode` - Query complexity analysis
+- `OrchestrationManagerNode` - System orchestration
+- `ConvergenceDetectorNode` - Solution convergence
+- `AgentPoolManagerNode` - Agent pool management
+- `ProblemAnalyzerNode` - Problem decomposition
+- `TeamFormationNode` - Optimal team creation
+- `SolutionEvaluatorNode` - Multi-criteria evaluation
+- `SelfOrganizingAgentNode` - Adaptive individual agents
+
 </td>
 <td width="50%">
 
@@ -918,6 +1249,95 @@ This example demonstrates:
 - **Context formatting** for LLM input
 - **Answer generation** using Ollama's llama3.2 model
 
+### 🔒 Access Control and Security
+
+Kailash SDK provides comprehensive access control and security features for enterprise deployments:
+
+#### Role-Based Access Control (RBAC)
+```python
+from kailash.access_control import UserContext, PermissionRule, NodePermission
+from kailash.runtime.access_controlled import AccessControlledRuntime
+
+# Define user with roles
+user = UserContext(
+    user_id="analyst_001",
+    tenant_id="company_abc",
+    email="analyst@company.com",
+    roles=["analyst", "viewer"]
+)
+
+# Create secure runtime
+runtime = AccessControlledRuntime(user_context=user)
+
+# Execute workflow with automatic permission checks
+results, run_id = runtime.execute(workflow, parameters={})
+```
+
+#### Multi-Tenant Isolation
+```python
+from kailash.access_control import get_access_control_manager, PermissionEffect, WorkflowPermission
+
+# Configure tenant-based access rules
+acm = get_access_control_manager()
+acm.enabled = True
+
+# Tenant isolation rule
+acm.add_rule(PermissionRule(
+    id="tenant_isolation",
+    resource_type="workflow",
+    resource_id="customer_analytics",
+    permission=WorkflowPermission.EXECUTE,
+    effect=PermissionEffect.ALLOW,
+    tenant_id="company_abc"  # Only this tenant can access
+))
+```
+
+#### Data Masking and Field Protection
+```python
+from kailash.nodes.base_with_acl import add_access_control
+
+# Add access control to sensitive data nodes
+secure_reader = add_access_control(
+    CSVReaderNode(file_path="customers.csv"),
+    enable_access_control=True,
+    required_permission=NodePermission.READ_OUTPUT,
+    mask_output_fields=["ssn", "phone"]  # Mask for non-admin users
+)
+
+workflow.add_node("secure_data", secure_reader)
+```
+
+#### Permission-Based Routing
+```python
+# Different execution paths based on user permissions
+from kailash.access_control import NodePermission
+
+# Admin users get full processing
+admin_processor = PythonCodeNode.from_function(
+    lambda data: {"result": process_all_data(data)},
+    name="admin_processor"
+)
+
+# Analyst users get limited processing
+analyst_processor = PythonCodeNode.from_function(
+    lambda data: {"result": process_limited_data(data)},
+    name="analyst_processor"
+)
+
+# Runtime automatically routes based on user permissions
+workflow.add_node("admin_path", admin_processor)
+workflow.add_node("analyst_path", analyst_processor)
+```
+
+**Security Features:**
+- 🔐 **JWT Authentication**: Token-based authentication with refresh support
+- 👥 **Multi-Tenant Isolation**: Complete data separation between tenants
+- 🛡️ **Field-Level Security**: Mask sensitive data based on user roles
+- 📊 **Audit Logging**: Complete access attempt logging for compliance
+- 🚫 **Path Traversal Prevention**: Built-in protection against directory attacks
+- 🏗️ **Backward Compatibility**: Existing workflows work unchanged
+- ⚡ **Performance Optimized**: Minimal overhead with caching
+
 ## 💻 CLI Commands
 
 The SDK includes a comprehensive CLI for workflow management:
@@ -1104,7 +1524,7 @@ pre-commit run pytest-check
 <td width="40%">
 
 ### ✅ Completed
-- Core node system with 15+ node types
+- Core node system with 66+ node types
 - Workflow builder with DAG validation
 - Local & async execution engines
 - Task tracking with metrics
@@ -1114,21 +1534,33 @@ pre-commit run pytest-check
 - Immutable state management
 - API integration with rate limiting
 - OAuth 2.0 authentication
+- Production security framework
+- Path traversal prevention
+- Code execution sandboxing
+- Comprehensive security testing
 - SharePoint Graph API integration
+- **Self-organizing agent pools with 13 specialized nodes**
+- **Agent-to-agent communication and shared memory**
+- **Intelligent caching and convergence detection**
+- **MCP integration for external tool access**
+- **Multi-strategy team formation algorithms**
 - **Real-time performance metrics collection**
 - **Performance visualization dashboards**
 - **Real-time monitoring dashboard with WebSocket streaming**
 - **Comprehensive performance reports (HTML, Markdown, JSON)**
-- **89% test coverage (571 tests)**
-- **15 test categories all passing**
-- 37 working examples
+- **100% test coverage (591 tests)**
+- **All test categories passing**
+- 68 working examples
 
 </td>
 <td width="30%">
 
 ### 🚧 In Progress
-- Comprehensive API documentation
-- Security audit & hardening
+- **Kailash Workflow Studio** - Visual workflow builder UI
+  - React-based drag-and-drop interface
+  - Multi-tenant architecture with Docker
+  - WorkflowStudioAPI backend
+  - Real-time execution monitoring
 - Performance optimizations
 - Docker runtime finalization
 
@@ -1146,11 +1578,12 @@ pre-commit run pytest-check
 </table>
 
 ### 🎯 Test Suite Status
-- **Total Tests**: 571 passing (89%)
-- **Test Categories**: 15/15 at 100%
-- **Integration Tests**: 65 passing
-- **Examples**: 37/37 working
-- **Code Coverage**: 89%
+- **Total Tests**: 591 passing (100%)
+- **Test Categories**: All passing
+- **Integration Tests**: All passing
+- **Security Tests**: 10 consolidated comprehensive tests
+- **Examples**: 68/68 working
+- **Code Coverage**: 100%
 
 ## ⚠️ Known Issues