flock-core 0.5.10__tar.gz → 0.5.11__tar.gz

flock_core-0.5.11/.github/issue_draft_non_blocking_serve.md

@@ -0,0 +1,205 @@
+# 🚀 [FEATURE] Non-Blocking `serve()` for Dashboard Examples
+
+### Is your feature request related to a problem?
+
+Dashboard examples currently face a limitation: once `flock.serve(dashboard=True)` is called, execution blocks indefinitely. This makes it impossible to publish messages or perform any setup logic after starting the dashboard server, unlike CLI examples which use `run_until_idle()` for controlled execution.
+
+**Current workaround:**
+Users must publish all messages *before* calling `serve()`, which is awkward for demos and prevents realistic workflows where the server runs continuously while processing incoming messages.
+
+**Example of the problem:**
+```python
+# This doesn't work - can't execute after serve()
+async def main():
+    await flock.serve(dashboard=True)  # blocks forever
+
+    # This code never runs! ❌
+    pizza = MyDreamPizza(pizza_idea="pizza with pineapple")
+    await flock.publish(pizza)
+    await flock.run_until_idle()
+```
+
+### Describe the solution you want to see
+
+Add a `blocking` parameter to `serve()` that allows starting the dashboard server as a background task:
+
+```python
+async def serve(
+    self,
+    *,
+    dashboard: bool = False,
+    dashboard_v2: bool = False,
+    host: str = "127.0.0.1",
+    port: int = 8344,
+    blocking: bool = True,  # NEW: default True for backwards compatibility
+) -> Task[None] | None:
+```
+
+**Usage example:**
+```python
+async def main():
+    # Start dashboard in background
+    await flock.serve(dashboard=True, blocking=False)
+
+    # Now we can publish messages!
+    pizza = MyDreamPizza(pizza_idea="pizza with pineapple")
+    await flock.publish(pizza)
+    await flock.run_until_idle()
+
+    # Dashboard stays alive for inspection
+    print("Check dashboard at http://localhost:8344")
+    await asyncio.Event().wait()  # Keep running
+```
+
+**Key requirements:**
+- Backwards compatible (`blocking=True` by default)
+- Return `Task[None]` handle when `blocking=False`
+- Proper cleanup of dashboard launcher and server on shutdown or task cancellation
+- Works seamlessly with `run_until_idle()`
+
+### Describe alternatives you have considered
+
+**Option 1: Startup hook pattern**
+```python
+async def setup():
+    await flock.publish(...)
+
+await flock.serve(dashboard=True, on_startup=setup)
+```
+❌ Less flexible, doesn't allow continuous message publishing
+
+**Option 2: Separate `serve_background()` method**
+```python
+await flock.serve_background(dashboard=True)
+```
+❌ API duplication, less discoverable
+
+**Option 3: Always non-blocking**
+```python
+await flock.serve(dashboard=True)  # always returns immediately
+```
+❌ Breaking change for existing users
+
+### Additional context
+
+**Complexity: Medium** (6-9 hours estimated)
+- Core async pattern is straightforward (codebase already uses `asyncio.create_task` extensively)
+- Cleanup coordination requires careful handling (dashboard launcher + server task lifecycle)
+- Backwards compatibility is preserved with default parameter
+
+---
+
+## 🛠️ Implementation Plan
+
+### 1. Core Changes
+
+**File: `src/flock/orchestrator.py`**
+
+- Add `blocking: bool = True` parameter to `serve()` method (line 753)
+- Add `_server_task: Task[None] | None = None` field to `__init__` (around line 165)
+- Update return type: `-> Task[None] | None`
+- When `blocking=False`:
+  - Wrap server execution in `asyncio.create_task()`
+  - Add cleanup callback for dashboard launcher on task completion
+  - Track task in `_server_task` field
+  - Return task handle
+
+**Estimated effort:** 2-3 hours
+
+### 2. Cleanup Logic
+
+**File: `src/flock/orchestrator.py`**
+
+- Add server cleanup to `shutdown()` method (line 718)
+  - Cancel `_server_task` if running
+  - Stop `_dashboard_launcher` if present
+  - Handle `asyncio.CancelledError` gracefully
+- Ensure proper resource cleanup on both normal completion and cancellation
+
+**Estimated effort:** 1-2 hours
+
+### 3. Testing
+
+**File: `tests/integration/test_orchestrator_dashboard.py` (or new test file)**
+
+Test scenarios:
+- ✅ Non-blocking serve starts successfully and returns task
+- ✅ Can publish messages after non-blocking serve
+- ✅ Server processes messages during `run_until_idle()`
+- ✅ Dashboard launcher cleanup on task completion
+- ✅ Cleanup on explicit `shutdown()` call
+- ✅ Backwards compatibility: `blocking=True` works as before
+- ✅ Task cancellation cleans up resources properly
+
+**Estimated effort:** 2-3 hours
+
+### 4. Documentation
+
+**Files to update:**
+- `src/flock/orchestrator.py` - Update `serve()` docstring with:
+  - New `blocking` parameter description
+  - Example showing non-blocking usage
+  - Return type documentation
+
+**Estimated effort:** 30 minutes
+
+### 5. Example
+
+**File: `examples/02-dashboard/02_non_blocking_serve.py` (new)**
+
+Create example demonstrating:
+- Starting dashboard in non-blocking mode
+- Publishing messages after server starts
+- Using `run_until_idle()` to process messages
+- Keeping server alive for inspection
+
+**Estimated effort:** 1 hour
+
+---
+
+## ✅ Definition of Done
+
+- [ ] `blocking` parameter implemented with default `True`
+- [ ] Non-blocking mode returns `Task[None]` handle
+- [ ] Server cleanup logic handles both normal and cancelled completion
+- [ ] All tests passing (new + existing)
+- [ ] Docstring updated with parameter and examples
+- [ ] New example created: `examples/02-dashboard/02_non_blocking_serve.py`
+- [ ] Backwards compatibility verified (existing examples still work)
+- [ ] Manual validation: example runs and dashboard stays accessible
+
+---
+
+## 🧪 Validation Strategy
+
+**Automated:**
+- Unit/integration tests for non-blocking behavior
+- Cleanup verification tests
+- Backwards compatibility regression tests
+
+**Manual:**
+1. Run new example: `python examples/02-dashboard/02_non_blocking_serve.py`
+2. Verify dashboard accessible at http://localhost:8344
+3. Verify messages appear in dashboard
+4. Verify server stays alive after `run_until_idle()`
+5. Test graceful shutdown (Ctrl+C)
+
+---
+
+## 📅 Timeline Estimate
+
+**Total effort:** 6-9 hours
+
+Breakdown:
+- Implementation: 3-5 hours
+- Testing: 2-3 hours
+- Documentation + Example: 1.5 hours
+
+---
+
+## 🔗 Technical References
+
+- `orchestrator.py:753` - Current `serve()` implementation
+- `orchestrator.py:580, 588, 1417` - Existing `asyncio.create_task` usage patterns
+- `orchestrator.py:718` - Existing `shutdown()` method
+- `service.py:279` - `BlackboardHTTPService.run_async()` (blocks on `uvicorn.Server.serve()`)

{flock_core-0.5.10 → flock_core-0.5.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flock-core
-Version: 0.5.10
+Version: 0.5.11
 Summary: Flock: A declrative framework for building and orchestrating AI agents.
 Author-email: Andre Ratzenberger <andre.ratzenberger@whiteduck.de>
 License: MIT
@@ -287,63 +287,6 @@ asyncio.run(main())
 
 ---
 
-## Context Provider Primer (Security + Cost)
-
-Context Providers are the security and efficiency layer between agents and the blackboard. They decide what each agent sees in its historical context and always enforce visibility rules.
-
-Why use them
-- Enforce access control (visibility is applied before engines see data)
-- Cut context size (often 90%+) by filtering to what matters
-- Specialize per agent or set a global default
-
-Copy/paste examples
-```python
-from flock import Flock
-from flock.context_provider import FilteredContextProvider
-from flock.store import FilterConfig
-
-# Global: show only urgent items (limit context size)
-flock = Flock(
-    "openai/gpt-4.1",
-    context_provider=FilteredContextProvider(FilterConfig(tags={"urgent"}), limit=50)
-)
-
-# Per-agent: override global for a specific role
-senior = flock.agent("senior").consumes(LogEntry).publishes(Analysis)
-senior.context_provider = FilteredContextProvider(
-    FilterConfig(tags={"ERROR", "WARN"}),
-    limit=200
-)
-```
-
-Learn more: docs/guides/context-providers.md
-
----
-
-## Persistent Blackboard History
-
-The in-memory store is still great for local tinkering, but production teams now have a durable option. Plugging in `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
-
-- **Long-lived artifacts** — every field (payload, tags, partition keys, visibility) is stored for replay, audits, and postmortems
-- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
-- **Dashboard module** — the new **Historical Blackboard** experience preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
-- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
-
-Quick start:
-
-```python
-from flock import Flock
-from flock.store import SQLiteBlackboardStore
-
-store = SQLiteBlackboardStore(".flock/blackboard.db")
-await store.ensure_schema()
-flock = Flock("openai/gpt-4.1", store=store)
-```
-
-Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` and explore previous runs, consumption trails, and retention banners inside the dashboard.
-
----
-
 ## Core Concepts
 
 ### Typed Artifacts (The Vocabulary)
@@ -673,6 +616,28 @@ agent.context_provider = FilteredContextProvider(
 
 **📖 [Learn more: Context Providers Guide](https://whiteducksoftware.github.io/flock/guides/context-providers/) | [Steal production code →](examples/08-context-provider/)**
 
+### Persistent Blackboard History
+
+The in-memory store is great for local development, but production teams need durability. The `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
+
+**What you get:**
+- **Long-lived artifacts** — Every field (payload, tags, partition keys, visibility) stored for replay, audits, and postmortems
+- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
+- **Dashboard integration** — The **Historical Blackboard** view preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
+- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
+
+**Quick start:**
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db")
+await store.ensure_schema()
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+**Try it:** Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` to explore previous runs, consumption trails, and retention banners.
+
 ### Batching Pattern: Parallel Execution Control
 
 **A key differentiator:** The separation of `publish()` and `run_until_idle()` enables parallel execution.
@@ -800,6 +765,35 @@ agent.best_of(150, ...)  # ⚠️ Warns: "best_of(150) is very high - high LLM c
 
 ## Production-Ready Observability
 
+### Sophisticated REST API
+
+**Production-ready HTTP endpoints with comprehensive OpenAPI documentation:**
+
+Flock includes a fully-featured REST API for programmatic access to the blackboard, agents, and workflow orchestration. Perfect for integration with external systems, building custom UIs, or monitoring production deployments.
+
+**Key endpoints:**
+- `POST /api/v1/artifacts` - Publish artifacts to the blackboard
+- `GET /api/v1/artifacts` - Query artifacts with filtering, pagination, and consumption metadata
+- `POST /api/v1/agents/{name}/run` - Direct agent invocation
+- `GET /api/v1/correlations/{correlation_id}/status` - Workflow completion tracking
+- `GET /api/v1/agents` - List all registered agents with subscriptions
+- `GET /health` and `GET /metrics` - Production monitoring
+
+**Start the API server:**
+```python
+await flock.serve(dashboard=True)  # API + Dashboard on port 8344
+# API docs: http://localhost:8344/docs
+```
+
+**Features:**
+- ✅ **OpenAPI 3.0** - Interactive documentation at `/docs`
+- ✅ **Pydantic validation** - Type-safe request/response models
+- ✅ **Correlation tracking** - Monitor workflow completion with polling
+- ✅ **Consumption metadata** - Full artifact lineage and agent execution trails
+- ✅ **Production monitoring** - Health checks and Prometheus-compatible metrics
+
+**📖 [Explore the API →](http://localhost:8344/docs)** (start the server first!)
+
 ### Real-Time Dashboard
 
 **Start the dashboard with one line:**

{flock_core-0.5.10 → flock_core-0.5.11}/README.md

@@ -254,63 +254,6 @@ asyncio.run(main())
 
 ---
 
-## Context Provider Primer (Security + Cost)
-
-Context Providers are the security and efficiency layer between agents and the blackboard. They decide what each agent sees in its historical context and always enforce visibility rules.
-
-Why use them
-- Enforce access control (visibility is applied before engines see data)
-- Cut context size (often 90%+) by filtering to what matters
-- Specialize per agent or set a global default
-
-Copy/paste examples
-```python
-from flock import Flock
-from flock.context_provider import FilteredContextProvider
-from flock.store import FilterConfig
-
-# Global: show only urgent items (limit context size)
-flock = Flock(
-    "openai/gpt-4.1",
-    context_provider=FilteredContextProvider(FilterConfig(tags={"urgent"}), limit=50)
-)
-
-# Per-agent: override global for a specific role
-senior = flock.agent("senior").consumes(LogEntry).publishes(Analysis)
-senior.context_provider = FilteredContextProvider(
-    FilterConfig(tags={"ERROR", "WARN"}),
-    limit=200
-)
-```
-
-Learn more: docs/guides/context-providers.md
-
----
-
-## Persistent Blackboard History
-
-The in-memory store is still great for local tinkering, but production teams now have a durable option. Plugging in `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
-
-- **Long-lived artifacts** — every field (payload, tags, partition keys, visibility) is stored for replay, audits, and postmortems
-- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
-- **Dashboard module** — the new **Historical Blackboard** experience preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
-- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
-
-Quick start:
-
-```python
-from flock import Flock
-from flock.store import SQLiteBlackboardStore
-
-store = SQLiteBlackboardStore(".flock/blackboard.db")
-await store.ensure_schema()
-flock = Flock("openai/gpt-4.1", store=store)
-```
-
-Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` and explore previous runs, consumption trails, and retention banners inside the dashboard.
-
----
-
 ## Core Concepts
 
 ### Typed Artifacts (The Vocabulary)
@@ -640,6 +583,28 @@ agent.context_provider = FilteredContextProvider(
 
 **📖 [Learn more: Context Providers Guide](https://whiteducksoftware.github.io/flock/guides/context-providers/) | [Steal production code →](examples/08-context-provider/)**
 
+### Persistent Blackboard History
+
+The in-memory store is great for local development, but production teams need durability. The `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
+
+**What you get:**
+- **Long-lived artifacts** — Every field (payload, tags, partition keys, visibility) stored for replay, audits, and postmortems
+- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
+- **Dashboard integration** — The **Historical Blackboard** view preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
+- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
+
+**Quick start:**
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db")
+await store.ensure_schema()
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+**Try it:** Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` to explore previous runs, consumption trails, and retention banners.
+
 ### Batching Pattern: Parallel Execution Control
 
 **A key differentiator:** The separation of `publish()` and `run_until_idle()` enables parallel execution.
@@ -767,6 +732,35 @@ agent.best_of(150, ...)  # ⚠️ Warns: "best_of(150) is very high - high LLM c
 
 ## Production-Ready Observability
 
+### Sophisticated REST API
+
+**Production-ready HTTP endpoints with comprehensive OpenAPI documentation:**
+
+Flock includes a fully-featured REST API for programmatic access to the blackboard, agents, and workflow orchestration. Perfect for integration with external systems, building custom UIs, or monitoring production deployments.
+
+**Key endpoints:**
+- `POST /api/v1/artifacts` - Publish artifacts to the blackboard
+- `GET /api/v1/artifacts` - Query artifacts with filtering, pagination, and consumption metadata
+- `POST /api/v1/agents/{name}/run` - Direct agent invocation
+- `GET /api/v1/correlations/{correlation_id}/status` - Workflow completion tracking
+- `GET /api/v1/agents` - List all registered agents with subscriptions
+- `GET /health` and `GET /metrics` - Production monitoring
+
+**Start the API server:**
+```python
+await flock.serve(dashboard=True)  # API + Dashboard on port 8344
+# API docs: http://localhost:8344/docs
+```
+
+**Features:**
+- ✅ **OpenAPI 3.0** - Interactive documentation at `/docs`
+- ✅ **Pydantic validation** - Type-safe request/response models
+- ✅ **Correlation tracking** - Monitor workflow completion with polling
+- ✅ **Consumption metadata** - Full artifact lineage and agent execution trails
+- ✅ **Production monitoring** - Health checks and Prometheus-compatible metrics
+
+**📖 [Explore the API →](http://localhost:8344/docs)** (start the server first!)
+
 ### Real-Time Dashboard
 
 **Start the dashboard with one line:**

{flock_core-0.5.10 → flock_core-0.5.11}/docs/guides/dashboard.md

@@ -45,6 +45,34 @@ INFO: Uvicorn running on http://127.0.0.1:8344
 [Dashboard] Browser launched successfully
 ```
 
+### Non-Blocking Mode (New!)
+
+Need to publish messages **after** starting the dashboard? Use non-blocking mode:
+
+```python
+# Start dashboard in background
+await flock.serve(dashboard=True, blocking=False)
+
+# Now you can publish messages and run logic!
+await flock.publish(my_message)
+await flock.run_until_idle()
+
+# Dashboard stays alive for inspection
+```
+
+**Perfect for:**
+- 🎯 Demo scripts that populate data after server starts
+- 🧪 Testing scenarios where you control execution flow
+- 📊 Scripts that need to perform setup after dashboard is running
+
+**Example:** See [`examples/02-dashboard/02_non_blocking_serve.py`](https://github.com/whiteducksoftware/flock/blob/main/examples/02-dashboard/02_non_blocking_serve.py) for a complete demo that publishes multiple pizza orders after starting the dashboard.
+
+**How it works:**
+- `blocking=True` (default): Server runs forever, blocks execution
+- `blocking=False`: Returns immediately, server runs in background
+- Automatic cleanup when task completes or is cancelled
+- Works with both dashboard and non-dashboard modes
+
 <p align="center">
   <img alt="Flock Blackboard View" src="../img/flock_ui_blackboard_view.png" width="1000">
   <i>Blackboard View: Track data lineage and transformations across the system</i>

{flock_core-0.5.10 → flock_core-0.5.11}/docs/guides/index.md

@@ -40,6 +40,14 @@ Comprehensive guides for building production-ready multi-agent systems with Floc
 
     [:octicons-arrow-right-24: Dashboard Guide](dashboard.md)
 
+-   **🌐 REST API**
+
+    ---
+
+    Production-ready HTTP endpoints with OpenAPI documentation for integration.
+
+    [:octicons-arrow-right-24: REST API Guide](rest-api.md)
+
 </div>
 
 ---
@@ -124,6 +132,7 @@ Comprehensive observability with OpenTelemetry and DuckDB.
 - **[Time-Based Access](visibility.md#5-aftervisibility-time-delayed)** - Temporal constraints
 
 ### Observability
+- **[REST API](rest-api.md)** - HTTP endpoints for integration and monitoring
 - **[Trace Capture](tracing/auto-tracing.md)** - Automatic trace collection
 - **[Trace Analysis](tracing/how_to_use_tracing_effectively.md)** - Debugging workflows
 - **[Dashboard Viewer](tracing/trace-module.md)** - Visual trace exploration
@@ -139,6 +148,8 @@ Looking for specific tasks? Here are quick links:
 - **Enable parallel execution** → [Batching Pattern](blackboard.md)
 - **Debug agent execution** → [Tracing Effectively](tracing/how_to_use_tracing_effectively.md)
 - **Add real-time monitoring** → [Dashboard Guide](dashboard.md)
+- **Integrate with external systems** → [REST API Guide](rest-api.md)
+- **Track workflow completion** → [Correlation Status](rest-api.md#correlation-status-workflow-tracking)
 - **Implement multi-tenancy** → [Tenant Visibility](visibility.md#3-tenantvisibility-multi-tenancy)
 - **Query artifacts** → [Blackboard Guide](blackboard.md)