flock-core 0.5.0b63__tar.gz → 0.5.0b70__tar.gz

{flock_core-0.5.0b63 → flock_core-0.5.0b70}/.gitignore

@@ -7,6 +7,7 @@ __pycache__/
 *.so
 .claude-flow/
 .swarm/
+.codex/
 # Distribution / packaging
 .Python
 build/

{flock_core-0.5.0b63 → flock_core-0.5.0b70}/AGENTS.md

@@ -234,6 +234,35 @@ The separation of `publish()` and `run_until_idle()` gives you **control over ex
 
 ---
 
+### 🗄️ Persistent Blackboard History (SQLite Store)
+
+**Production teams asked for auditability—now the blackboard can keep a full historical trail.**
+
+#### Why this matters
+- **Durable artifacts**: Everything on the blackboard (payloads, tags, visibility, correlation IDs) is stored on disk for replay, compliance, and debugging.
+- **Faster postmortems**: Query `/api/v1/artifacts` with filters (`type`, `produced_by`, `tags`, `visibility`, time windows) to reconstruct agent cascades after the fact.
+- **Operational dashboards**: The new **Historical Blackboard** module preloads persisted pages, exposes consumption metadata, and shows retention banners so operators know how far back they can scroll.
+- **Retention policies**: CLI helpers make maintenance routine—`flock sqlite-maintenance my.db --delete-before 2025-01-01T00:00:00Z --vacuum` prunes old artifacts and compacts the store.
+
+#### Quick start
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/history.db")
+await store.ensure_schema()
+
+flock = Flock("openai/gpt-4.1", store=store)
+await flock.publish(MyDreamPizza(pizza_idea="fermented garlic delight"))
+await flock.run_until_idle()
+```
+
+Kick the tyres with `examples/02-the-blackboard/01_persistent_pizza.py`, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` to inspect the retained history alongside live WebSocket updates.
+
+> **Heads-up:** The interface now returns `ArtifactEnvelope` objects when `embed_meta=True`. Future backends (Postgres, BigQuery, etc.) can implement the same contract to plug straight into the runtime and dashboard.
+
+---
+
 ### 🔒 Test Isolation and Mock Cleanup - The Contamination Trap
 
 **We fixed 32 failing tests caused by test contamination - here's what we learned:**

flock_core-0.5.0b63/README.md → flock_core-0.5.0b70/PKG-INFO

@@ -1,3 +1,35 @@
+Metadata-Version: 2.4
+Name: flock-core
+Version: 0.5.0b70
+Summary: Add your description here
+Author-email: Andre Ratzenberger <andre.ratzenberger@whiteduck.de>
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: aiosqlite>=0.20.0
+Requires-Dist: devtools>=0.12.2
+Requires-Dist: dspy==3.0.0
+Requires-Dist: duckdb>=1.1.0
+Requires-Dist: fastapi>=0.117.1
+Requires-Dist: hanging-threads>=2.0.7
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: litellm==1.75.3
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: mcp>=1.7.1
+Requires-Dist: opentelemetry-api>=1.30.0
+Requires-Dist: opentelemetry-exporter-jaeger-proto-grpc>=1.21.0
+Requires-Dist: opentelemetry-exporter-jaeger>=1.21.0
+Requires-Dist: opentelemetry-exporter-otlp>=1.30.0
+Requires-Dist: opentelemetry-instrumentation-logging>=0.51b0
+Requires-Dist: opentelemetry-sdk>=1.30.0
+Requires-Dist: poethepoet>=0.30.0
+Requires-Dist: pydantic[email]>=2.11.9
+Requires-Dist: rich>=14.1.0
+Requires-Dist: toml>=0.10.2
+Requires-Dist: typer>=0.19.2
+Requires-Dist: uvicorn>=0.37.0
+Requires-Dist: websockets>=15.0.1
+Description-Content-Type: text/markdown
+
 <p align="center">
   <img alt="Flock Banner" src="https://raw.githubusercontent.com/whiteducksoftware/flock/master/docs/assets/images/flock.png" width="800">
 </p>
@@ -252,6 +284,30 @@ asyncio.run(main())
 
 ---
 
+## Persistent Blackboard History
+
+The in-memory store is still great for local tinkering, but production teams now have a durable option. Plugging in `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
+
+- **Long-lived artifacts** — every field (payload, tags, partition keys, visibility) is stored for replay, audits, and postmortems
+- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
+- **Dashboard module** — the new **Historical Blackboard** experience preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
+- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
+
+Quick start:
+
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db")
+await store.ensure_schema()
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` and explore previous runs, consumption trails, and retention banners inside the dashboard.
+
+---
+
 ## Core Concepts
 
 ### Typed Artifacts (The Vocabulary)

flock_core-0.5.0b63/PKG-INFO → flock_core-0.5.0b70/README.md

@@ -1,33 +1,3 @@
-Metadata-Version: 2.4
-Name: flock-core
-Version: 0.5.0b63
-Summary: Add your description here
-Author-email: Andre Ratzenberger <andre.ratzenberger@whiteduck.de>
-License-File: LICENSE
-Requires-Python: >=3.10
-Requires-Dist: devtools>=0.12.2
-Requires-Dist: dspy==3.0.0
-Requires-Dist: duckdb>=1.1.0
-Requires-Dist: fastapi>=0.117.1
-Requires-Dist: httpx>=0.28.1
-Requires-Dist: litellm==1.75.3
-Requires-Dist: loguru>=0.7.3
-Requires-Dist: mcp>=1.7.1
-Requires-Dist: opentelemetry-api>=1.30.0
-Requires-Dist: opentelemetry-exporter-jaeger-proto-grpc>=1.21.0
-Requires-Dist: opentelemetry-exporter-jaeger>=1.21.0
-Requires-Dist: opentelemetry-exporter-otlp>=1.30.0
-Requires-Dist: opentelemetry-instrumentation-logging>=0.51b0
-Requires-Dist: opentelemetry-sdk>=1.30.0
-Requires-Dist: poethepoet>=0.30.0
-Requires-Dist: pydantic[email]>=2.11.9
-Requires-Dist: rich>=14.1.0
-Requires-Dist: toml>=0.10.2
-Requires-Dist: typer>=0.19.2
-Requires-Dist: uvicorn>=0.37.0
-Requires-Dist: websockets>=15.0.1
-Description-Content-Type: text/markdown
-
 <p align="center">
   <img alt="Flock Banner" src="https://raw.githubusercontent.com/whiteducksoftware/flock/master/docs/assets/images/flock.png" width="800">
 </p>
@@ -282,6 +252,30 @@ asyncio.run(main())
 
 ---
 
+## Persistent Blackboard History
+
+The in-memory store is still great for local tinkering, but production teams now have a durable option. Plugging in `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
+
+- **Long-lived artifacts** — every field (payload, tags, partition keys, visibility) is stored for replay, audits, and postmortems
+- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
+- **Dashboard module** — the new **Historical Blackboard** experience preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
+- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
+
+Quick start:
+
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db")
+await store.ensure_schema()
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` and explore previous runs, consumption trails, and retention banners inside the dashboard.
+
+---
+
 ## Core Concepts
 
 ### Typed Artifacts (The Vocabulary)

flock_core-0.5.0b70/docs/bugfixes/websocket-streaming-deadlock-fix.md

@@ -0,0 +1,203 @@
+# WebSocket Streaming Deadlock Fix
+
+## Issue Description
+
+**Symptom**: When using the dashboard UI, agent execution would freeze (deadlock) when the agent called MCP tools during streaming, particularly with filesystem operations. The issue did NOT occur when running via CLI without the dashboard.
+
+**Affected Component**: `src/flock/engines/dspy_engine.py` - streaming execution with WebSocket broadcasting
+
+## Root Cause Analysis
+
+### The Deadlock Scenario
+
+The deadlock occurred due to a **circular wait condition** in the async event loop:
+
+1. **Agent starts streaming** tokens via WebSocket: `await ws_manager.broadcast(event)`
+2. **Agent calls an MCP tool** (e.g., filesystem read) - this is an `await` operation
+3. **WebSocket send buffer fills up** because the client can't process messages fast enough
+4. **`broadcast()` blocks** waiting for `client.send_text(message)` to complete (500ms timeout)
+5. **Client is blocked** waiting for the agent to finish
+6. **Agent can't continue** because it's waiting for `broadcast()` to return
+
+This creates a **circular dependency**:
+```
+Streaming loop → await broadcast() → await client.send_text() → 
+  → client waiting for response → agent waiting to send → 
+    → agent blocked on MCP tool → deadlock!
+```
+
+### Why It Only Happened with Dashboard
+
+- **CLI mode**: No WebSocket, streaming goes directly to console (no blocking I/O to client)
+- **Dashboard mode**: WebSocket adds network I/O and client buffering, creating blocking points
+
+### Key Code Pattern (Before Fix)
+
+```python
+async for value in stream_generator:
+    if isinstance(value, StatusMessage):
+        # ... process token ...
+        
+        # BLOCKING: Waits for WebSocket to complete send
+        await ws_manager.broadcast(event)  # ← DEADLOCK HERE
+        
+    # ... later ...
+    
+    # MCP tool call happens here
+    result = await server.call_tool(...)  # ← Can't reach if blocked above
+```
+
+## The Fix
+
+### Solution: Non-Blocking WebSocket Broadcasts
+
+Changed all `await ws_manager.broadcast(event)` calls inside the streaming loop to **fire-and-forget** using `asyncio.create_task()`:
+
+```python
+# Before (blocking)
+await ws_manager.broadcast(event)
+
+# After (non-blocking)
+task = asyncio.create_task(ws_manager.broadcast(event))
+ws_broadcast_tasks.add(task)
+task.add_done_callback(ws_broadcast_tasks.discard)
+```
+
+### Changes Made
+
+**File**: `src/flock/engines/dspy_engine.py`
+
+1. **Added asyncio import**:
+   ```python
+   import asyncio
+   ```
+
+2. **Track background tasks** to prevent garbage collection:
+   ```python
+   # Track background WebSocket broadcast tasks to prevent garbage collection
+   ws_broadcast_tasks: set[asyncio.Task] = set()
+   ```
+
+3. **Updated all broadcast calls** (4 locations in streaming loop):
+   - `StatusMessage` handling (line ~625)
+   - `StreamResponse` handling (line ~670)
+   - `ModelResponseStream` handling (line ~715)
+   - Final `Prediction` handling (lines ~745, ~760)
+
+### Why This Works
+
+1. **Breaks the circular wait**: Streaming loop doesn't wait for WebSocket sends to complete
+2. **Maintains order**: Tasks are created in sequence, AsyncIO scheduler maintains order
+3. **Prevents garbage collection**: Tasks stored in set until completion
+4. **Graceful error handling**: Errors in broadcast don't crash the streaming loop
+5. **Timeout still applies**: WebSocket still has 500ms timeout per client (in broadcast method)
+
+## Testing
+
+### Before Fix
+```bash
+# This would deadlock when agent calls filesystem tool
+uv run python examples/03-the-dashboard/test_streaming_fix.py
+# Result: Hangs indefinitely when tool is called
+```
+
+### After Fix
+```bash
+# Should complete successfully
+uv run python examples/03-the-dashboard/test_streaming_fix.py
+# Result: Streams tokens, calls tool, completes normally
+```
+
+### Verification Steps
+
+1. **Start dashboard example** with MCP filesystem tools
+2. **Publish artifact** that triggers agent with tool calls
+3. **Observe streaming**: Should see tokens streaming in real-time
+4. **Tool execution**: Agent should call filesystem tools without freezing
+5. **Completion**: Agent should finish and display final output
+
+## Related Code
+
+### WebSocket Timeout Protection
+
+The WebSocket broadcast already had timeout protection (500ms per client):
+
+```python
+# In src/flock/dashboard/websocket.py
+send_tasks = [
+    asyncio.wait_for(client.send_text(message), timeout=0.5)  # 500ms timeout
+    for client in clients_list
+]
+results = await asyncio.gather(*send_tasks, return_exceptions=True)
+```
+
+This timeout prevented infinite hangs on individual client sends, but didn't prevent the streaming loop from blocking.
+
+### MCP Tool Execution
+
+MCP tools are async operations that can take time:
+
+```python
+# In src/flock/mcp/tool.py
+async def func(*args, **kwargs):
+    result = await server.call_tool(
+        agent_id=self.agent_id,
+        run_id=self.run_id,
+        name=self.name,
+        arguments=kwargs,
+    )
+    return self._convert_mcp_tool_result(result)
+```
+
+When streaming loop blocks on `await broadcast()`, it can't reach tool execution.
+
+## Lessons Learned
+
+### Async Programming Gotchas
+
+1. **Never block in a loop that generates async events**: Use `create_task()` for background operations
+2. **Always track background tasks**: Prevents garbage collection and allows cleanup
+3. **WebSocket I/O is a blocking point**: Even with timeouts, can create circular waits
+4. **Test with real network conditions**: CLI vs Dashboard revealed the issue
+
+### Best Practices Applied
+
+1. ✅ **Fire-and-forget for broadcasts**: Don't wait for all clients to receive
+2. ✅ **Task tracking**: Prevent garbage collection with set + done_callback
+3. ✅ **Timeout protection**: Already in place at WebSocket level (500ms)
+4. ✅ **Error isolation**: try/except around each broadcast creation
+5. ✅ **Sequence preservation**: Tasks maintain order in AsyncIO scheduler
+
+## Impact
+
+### Fixed
+- ✅ Dashboard streaming works with MCP tool calls
+- ✅ No more deadlocks during agent execution
+- ✅ Real-time token streaming continues during tool execution
+
+### No Regressions
+- ✅ CLI mode still works (no WebSocket = no change)
+- ✅ Streaming order preserved (AsyncIO guarantees)
+- ✅ Error handling unchanged (try/except still catches failures)
+- ✅ Client timeout protection still active (500ms per client)
+
+## Future Considerations
+
+### Potential Improvements
+
+1. **Backpressure handling**: Could monitor `ws_broadcast_tasks` size and pause streaming if queue gets too large
+2. **Broadcast batching**: Could batch multiple tokens into single broadcast for efficiency
+3. **Client-side buffering**: Could implement client-side buffer to handle burst traffic
+4. **Metrics**: Could track broadcast task queue size and completion time
+
+### Architecture Decision
+
+**Decision**: Use fire-and-forget for WebSocket broadcasts during streaming
+**Rationale**: Streaming must not block on network I/O; client displays are eventually consistent
+**Trade-off**: Slight delay in UI updates vs. preventing deadlocks (clear win for preventing deadlocks)
+
+## Version
+
+- **Fixed in**: v0.5.0b64 (or current version)
+- **Issue discovered**: October 9, 2025
+- **Fix applied**: October 9, 2025