flock-core 0.5.9__tar.gz → 0.5.11__tar.gz

flock_core-0.5.11/.github/issue_draft_non_blocking_serve.md

@@ -0,0 +1,205 @@
+# 🚀 [FEATURE] Non-Blocking `serve()` for Dashboard Examples
+
+### Is your feature request related to a problem?
+
+Dashboard examples currently face a limitation: once `flock.serve(dashboard=True)` is called, execution blocks indefinitely. This makes it impossible to publish messages or perform any setup logic after starting the dashboard server, unlike CLI examples which use `run_until_idle()` for controlled execution.
+
+**Current workaround:**
+Users must publish all messages *before* calling `serve()`, which is awkward for demos and prevents realistic workflows where the server runs continuously while processing incoming messages.
+
+**Example of the problem:**
+```python
+# This doesn't work - can't execute after serve()
+async def main():
+    await flock.serve(dashboard=True)  # blocks forever
+
+    # This code never runs! ❌
+    pizza = MyDreamPizza(pizza_idea="pizza with pineapple")
+    await flock.publish(pizza)
+    await flock.run_until_idle()
+```
+
+### Describe the solution you want to see
+
+Add a `blocking` parameter to `serve()` that allows starting the dashboard server as a background task:
+
+```python
+async def serve(
+    self,
+    *,
+    dashboard: bool = False,
+    dashboard_v2: bool = False,
+    host: str = "127.0.0.1",
+    port: int = 8344,
+    blocking: bool = True,  # NEW: default True for backwards compatibility
+) -> Task[None] | None:
+```
+
+**Usage example:**
+```python
+async def main():
+    # Start dashboard in background
+    await flock.serve(dashboard=True, blocking=False)
+
+    # Now we can publish messages!
+    pizza = MyDreamPizza(pizza_idea="pizza with pineapple")
+    await flock.publish(pizza)
+    await flock.run_until_idle()
+
+    # Dashboard stays alive for inspection
+    print("Check dashboard at http://localhost:8344")
+    await asyncio.Event().wait()  # Keep running
+```
+
+**Key requirements:**
+- Backwards compatible (`blocking=True` by default)
+- Return `Task[None]` handle when `blocking=False`
+- Proper cleanup of dashboard launcher and server on shutdown or task cancellation
+- Works seamlessly with `run_until_idle()`
+
+### Describe alternatives you have considered
+
+**Option 1: Startup hook pattern**
+```python
+async def setup():
+    await flock.publish(...)
+
+await flock.serve(dashboard=True, on_startup=setup)
+```
+❌ Less flexible, doesn't allow continuous message publishing
+
+**Option 2: Separate `serve_background()` method**
+```python
+await flock.serve_background(dashboard=True)
+```
+❌ API duplication, less discoverable
+
+**Option 3: Always non-blocking**
+```python
+await flock.serve(dashboard=True)  # always returns immediately
+```
+❌ Breaking change for existing users
+
+### Additional context
+
+**Complexity: Medium** (6-9 hours estimated)
+- Core async pattern is straightforward (codebase already uses `asyncio.create_task` extensively)
+- Cleanup coordination requires careful handling (dashboard launcher + server task lifecycle)
+- Backwards compatibility is preserved with default parameter
+
+---
+
+## 🛠️ Implementation Plan
+
+### 1. Core Changes
+
+**File: `src/flock/orchestrator.py`**
+
+- Add `blocking: bool = True` parameter to `serve()` method (line 753)
+- Add `_server_task: Task[None] | None = None` field to `__init__` (around line 165)
+- Update return type: `-> Task[None] | None`
+- When `blocking=False`:
+  - Wrap server execution in `asyncio.create_task()`
+  - Add cleanup callback for dashboard launcher on task completion
+  - Track task in `_server_task` field
+  - Return task handle
+
+**Estimated effort:** 2-3 hours
+
+### 2. Cleanup Logic
+
+**File: `src/flock/orchestrator.py`**
+
+- Add server cleanup to `shutdown()` method (line 718)
+  - Cancel `_server_task` if running
+  - Stop `_dashboard_launcher` if present
+  - Handle `asyncio.CancelledError` gracefully
+- Ensure proper resource cleanup on both normal completion and cancellation
+
+**Estimated effort:** 1-2 hours
+
+### 3. Testing
+
+**File: `tests/integration/test_orchestrator_dashboard.py` (or new test file)**
+
+Test scenarios:
+- ✅ Non-blocking serve starts successfully and returns task
+- ✅ Can publish messages after non-blocking serve
+- ✅ Server processes messages during `run_until_idle()`
+- ✅ Dashboard launcher cleanup on task completion
+- ✅ Cleanup on explicit `shutdown()` call
+- ✅ Backwards compatibility: `blocking=True` works as before
+- ✅ Task cancellation cleans up resources properly
+
+**Estimated effort:** 2-3 hours
+
+### 4. Documentation
+
+**Files to update:**
+- `src/flock/orchestrator.py` - Update `serve()` docstring with:
+  - New `blocking` parameter description
+  - Example showing non-blocking usage
+  - Return type documentation
+
+**Estimated effort:** 30 minutes
+
+### 5. Example
+
+**File: `examples/02-dashboard/02_non_blocking_serve.py` (new)**
+
+Create example demonstrating:
+- Starting dashboard in non-blocking mode
+- Publishing messages after server starts
+- Using `run_until_idle()` to process messages
+- Keeping server alive for inspection
+
+**Estimated effort:** 1 hour
+
+---
+
+## ✅ Definition of Done
+
+- [ ] `blocking` parameter implemented with default `True`
+- [ ] Non-blocking mode returns `Task[None]` handle
+- [ ] Server cleanup logic handles both normal and cancelled completion
+- [ ] All tests passing (new + existing)
+- [ ] Docstring updated with parameter and examples
+- [ ] New example created: `examples/02-dashboard/02_non_blocking_serve.py`
+- [ ] Backwards compatibility verified (existing examples still work)
+- [ ] Manual validation: example runs and dashboard stays accessible
+
+---
+
+## 🧪 Validation Strategy
+
+**Automated:**
+- Unit/integration tests for non-blocking behavior
+- Cleanup verification tests
+- Backwards compatibility regression tests
+
+**Manual:**
+1. Run new example: `python examples/02-dashboard/02_non_blocking_serve.py`
+2. Verify dashboard accessible at http://localhost:8344
+3. Verify messages appear in dashboard
+4. Verify server stays alive after `run_until_idle()`
+5. Test graceful shutdown (Ctrl+C)
+
+---
+
+## 📅 Timeline Estimate
+
+**Total effort:** 6-9 hours
+
+Breakdown:
+- Implementation: 3-5 hours
+- Testing: 2-3 hours
+- Documentation + Example: 1.5 hours
+
+---
+
+## 🔗 Technical References
+
+- `orchestrator.py:753` - Current `serve()` implementation
+- `orchestrator.py:580, 588, 1417` - Existing `asyncio.create_task` usage patterns
+- `orchestrator.py:718` - Existing `shutdown()` method
+- `service.py:279` - `BlackboardHTTPService.run_async()` (blocks on `uvicorn.Server.serve()`)

{flock_core-0.5.9 → flock_core-0.5.11}/AGENTS.md

@@ -53,7 +53,7 @@ A blackboard architecture framework where specialized AI agents collaborate thro
 
 ### Prerequisites
 
-- **Python 3.10+** (we use modern async features)
+- **Python 3.12+** (we use modern async features)
 - **UV package manager** (faster than pip, handles virtual envs)
 - **Node.js 18+** (22+ recommended) for dashboard frontend
 - **OpenAI API key** (for running examples)
@@ -107,6 +107,7 @@ For deep dives into specific topics, see:
 **Core Architecture:**
 - **[Architecture & Blackboard](docs/guides/blackboard.md)** - Core pattern, structure, and behavior
 - **[Agent Guide](docs/guides/agents.md)** - Complete agent development reference
+- **[Context Providers](docs/guides/context-providers.md)** - Smart filtering & security boundaries for agent context ⭐ **NEW in 0.5**
 
 **Components & Extensibility:**
 - **[Agent Components](docs/guides/components.md)** - Extend agent behavior with lifecycle hooks
@@ -296,6 +297,288 @@ Kick the tyres with `examples/02-the-blackboard/01_persistent_pizza.py`, then la
 
 ---
 
+### 🔐 Context Providers - Smart Filtering & Security Boundaries
+
+**Context Providers are the intelligent filter layer between agents and the blackboard—controlling what each agent sees, reducing token costs by 90%+, and automatically redacting sensitive data.**
+
+#### Why Context Providers Matter
+
+**The Problem:**
+- Agents seeing ALL artifacts wastes tokens (costs)
+- No way to filter sensitive data before agents see it
+- Performance degrades as blackboard grows
+- Security vulnerabilities if agents bypass visibility
+
+**The Solution:**
+Context Providers enforce a security boundary that:
+- ✅ **Reduces token costs** - Agents see only relevant artifacts (90%+ savings)
+- ✅ **Protects sensitive data** - Auto-redact passwords, API keys, credit cards
+- ✅ **Improves performance** - Less context = faster agent execution
+- ✅ **Enforces security** - Agents cannot bypass provider filtering
+
+**Understanding the Three-Layer Model:**
+
+Flock uses three complementary filtering layers:
+
+1. **Visibility** (security boundary) - Controls BOTH which agents trigger AND which artifacts they see in context
+2. **Subscription Filters** (routing logic) - Controls WHEN agents trigger (e.g., `.consumes(Task, tags={"urgent"})`)
+3. **Context Providers** (context shaping) - Controls WHAT agents see in their historical context
+
+These layers work together to provide fine-grained control over agent execution and data access. Context Providers are the third layer—they filter what agents see in context but do NOT control triggering.
+
+#### Quick Start: Global Filtering
+
+**Filter all agents to see only urgent items:**
+
+```python
+from flock import Flock
+from flock.context_provider import FilteredContextProvider
+from flock.store import FilterConfig
+
+# Create provider that filters by tags
+urgent_only = FilteredContextProvider(
+    FilterConfig(tags={"urgent", "high"}),
+    limit=50  # Max 50 artifacts
+)
+
+# Apply to ALL agents
+flock = Flock("openai/gpt-4.1", context_provider=urgent_only)
+
+# Publish with tags
+await flock.publish(Task(...), tags={"urgent"})  # ✅ Agents see this
+await flock.publish(Task(...), tags={"low"})     # ❌ Agents don't see this
+```
+
+#### Per-Agent Filtering (Role-Based)
+
+**Different agents see different context:**
+
+```python
+# Global: Errors only
+error_provider = FilteredContextProvider(FilterConfig(tags={"ERROR"}))
+flock = Flock("openai/gpt-4.1", context_provider=error_provider)
+
+# Junior engineer: Errors only (uses global)
+junior = flock.agent("junior").consumes(LogEntry).publishes(Report).agent
+
+# Senior engineer: Errors + Warnings (override global)
+warn_provider = FilteredContextProvider(FilterConfig(tags={"ERROR", "WARN"}))
+senior = flock.agent("senior").consumes(LogEntry).publishes(Analysis).agent
+senior.context_provider = warn_provider  # 🎯 Per-agent override!
+
+# Platform team: Everything (override global)
+all_provider = FilteredContextProvider(FilterConfig(tags={"DEBUG", "INFO", "WARN", "ERROR"}))
+platform = flock.agent("platform").consumes(LogEntry).publishes(Report).agent
+platform.context_provider = all_provider
+```
+
+**Provider Priority:**
+```
+Per-Agent Provider  >  Global Provider  >  DefaultContextProvider
+     (highest)            (medium)              (fallback)
+```
+
+#### Production-Ready: Password Redaction
+
+**Automatically redact sensitive data from agent context:**
+
+```python
+from examples.context_provider import PasswordRedactorProvider
+
+# Create provider with built-in patterns
+provider = PasswordRedactorProvider(
+    redaction_text="[REDACTED]",
+    redact_emails=True,
+    log_redactions=True  # Audit trail
+)
+
+flock = Flock("openai/gpt-4.1", context_provider=provider)
+
+# Publish data with secrets
+await flock.publish(UserData(
+    password="MySecret123",           # → [REDACTED]
+    api_key="sk-1234567890abcdef",    # → [REDACTED]
+    email="user@example.com"          # → [REDACTED]
+))
+
+# Agents see redacted version automatically!
+```
+
+**What gets redacted:**
+- 🔒 Passwords and secrets
+- 🔑 API keys (OpenAI, AWS, GitHub, GitLab, Stripe, Google)
+- 🎫 Bearer tokens and JWT
+- 💳 Credit card numbers (Visa, MC, Amex, Discover)
+- 🆔 Social Security Numbers
+- 🔐 Private keys (RSA, EC, OpenSSH)
+- 📧 Email addresses (optional)
+
+#### FilterConfig API Reference
+
+**Declarative filtering with multiple criteria:**
+
+```python
+from flock.store import FilterConfig
+
+# Combine multiple filters (AND logic)
+config = FilterConfig(
+    tags={"ERROR", "CRITICAL"},           # OR logic: any of these tags
+    type_names={"SystemEvent", "Alert"},  # OR logic: any of these types
+    produced_by=["monitor", "watchdog"],  # OR logic: from any of these agents
+    correlation_id="incident-123"         # Exact match
+)
+
+provider = FilteredContextProvider(config, limit=100)
+```
+
+**Criteria are combined with AND logic:**
+- Must have (ERROR OR CRITICAL) tag
+- AND must be (SystemEvent OR Alert) type
+- AND must be from (monitor OR watchdog) agent
+- AND must have correlation_id "incident-123"
+
+#### Custom Context Providers
+
+**Build your own filtering logic:**
+
+```python
+from flock.context_provider import BaseContextProvider, ContextRequest
+from flock.artifacts import Artifact
+
+class TimeBoundProvider(BaseContextProvider):
+    """Only show artifacts from last 1 hour.
+
+    Security is enforced automatically by BaseContextProvider!
+    """
+
+    async def get_artifacts(self, request: ContextRequest) -> list[Artifact]:
+        cutoff = datetime.now() - timedelta(hours=1)
+
+        # Query all artifacts
+        artifacts, _ = await request.store.query_artifacts(limit=1000)
+
+        # Apply time-based filtering
+        recent = [a for a in artifacts if a.created_at >= cutoff]
+
+        return recent
+
+        # ✨ BaseContextProvider automatically:
+        # - Filters by visibility (MANDATORY, cannot bypass!)
+        # - Excludes requested IDs
+        # - Serializes to standard format
+
+# Usage
+provider = TimeBoundProvider()
+flock = Flock("openai/gpt-4.1", context_provider=provider)
+```
+
+**Why this is better:**
+- ✅ **75% less code** - 5 lines vs 20+ lines
+- ✅ **Impossible to bypass security** - visibility enforced by base class
+- ✅ **Consistent serialization** - automatic standard format
+- ✅ **Focus on business logic** - not infrastructure
+
+#### Real-World Use Cases
+
+**Healthcare (HIPAA Compliance):**
+```python
+# Redact PHI (Protected Health Information)
+from examples.context_provider import PasswordRedactorProvider
+
+phi_provider = PasswordRedactorProvider(
+    custom_patterns={
+        "ssn": r"\b\d{3}-\d{2}-\d{4}\b",
+        "mrn": r"MRN[:\s]*\d{6,10}",
+        "dob": r"\b\d{2}/\d{2}/\d{4}\b",
+    }
+)
+
+flock = Flock("openai/gpt-4.1", context_provider=phi_provider)
+```
+
+**DevOps (Log Filtering):**
+```python
+# Error engineers see only errors
+error_provider = FilteredContextProvider(FilterConfig(tags={"ERROR", "CRITICAL"}))
+error_agent.context_provider = error_provider
+
+# Platform team sees everything
+all_provider = FilteredContextProvider(FilterConfig(tags={"DEBUG", "INFO", "WARN", "ERROR"}))
+platform_agent.context_provider = all_provider
+```
+
+**Multi-Tenant SaaS (Customer Isolation):**
+```python
+# Each tenant gets isolated context
+tenant_a_provider = FilteredContextProvider(FilterConfig(correlation_id=f"tenant_{customer_a.id}"))
+tenant_a_agent.context_provider = tenant_a_provider
+
+tenant_b_provider = FilteredContextProvider(FilterConfig(correlation_id=f"tenant_{customer_b.id}"))
+tenant_b_agent.context_provider = tenant_b_provider
+```
+
+#### Performance Impact
+
+**Token Cost Reduction:**
+
+```python
+# ❌ Before (no filtering): Agent sees 1000 artifacts = 200,000 tokens
+# Cost: $0.006 per agent call
+
+# ✅ After (smart filtering): Agent sees 10 relevant artifacts = 2,000 tokens
+# Cost: $0.00006 per agent call
+
+# 💰 99% cost reduction!
+
+provider = FilteredContextProvider(FilterConfig(tags={"urgent"}), limit=10)
+```
+
+#### Security Best Practices
+
+**✅ DO:**
+- Always enforce visibility (NEVER skip)
+- Use `FilteredContextProvider` for common patterns
+- Test redaction patterns with real data
+- Log security events for audit trails
+- Limit by default (start restrictive)
+
+**❌ DON'T:**
+- Don't skip visibility filtering (security boundary!)
+- Don't leak sensitive data in nested objects
+- Don't cache redacted data without re-applying redaction
+- Don't trust agent input (agents are untrusted)
+- Don't expose infrastructure details
+
+#### Learn More
+
+**📚 Complete Guide:** [docs/guides/context-providers.md](docs/guides/context-providers.md) - Architecture, security model, advanced patterns
+
+**💡 Examples:** [examples/08-context-provider/](examples/08-context-provider/) - 5 progressive examples from beginner to expert
+
+**🎁 Production Code:** [examples/08-context-provider/05_password_redactor.py](examples/08-context-provider/05_password_redactor.py) - Copy-paste ready password filtering
+
+#### Quick Reference
+
+```python
+# Global filtering
+flock = Flock("openai/gpt-4.1", context_provider=FilteredContextProvider(FilterConfig(tags={"urgent"})))
+
+# Per-agent override
+agent.context_provider = FilteredContextProvider(FilterConfig(tags={"ERROR", "WARN"}))
+
+# Password redaction
+from examples.context_provider import PasswordRedactorProvider
+flock = Flock("openai/gpt-4.1", context_provider=PasswordRedactorProvider())
+
+# Custom provider
+class MyProvider(ContextProvider):
+    async def __call__(self, request: ContextRequest) -> list[dict[str, Any]]:
+        # Query + filter by visibility (MANDATORY) + your logic
+        pass
+```
+
+---
+
 ### 🔒 Test Isolation and Mock Cleanup - The Contamination Trap
 
 **We fixed 32 failing tests caused by test contamination - here's what we learned:**
@@ -1335,7 +1618,7 @@ Add to existing test file if relevant, or create new file following naming conve
 
 ### Q: What Python version features can I use?
 
-Python 3.10+, so you can use:
+Python 3.12+, so you can use:
 - `match`/`case` statements
 - `TaskGroup` for parallel execution
 - Improved type hints (`list[str]` not `List[str]`)