flock-core 0.5.0b65__tar.gz → 0.5.0b71__tar.gz

{flock_core-0.5.0b65 → flock_core-0.5.0b71}/.gitignore

@@ -7,6 +7,7 @@ __pycache__/
 *.so
 .claude-flow/
 .swarm/
+.codex/
 # Distribution / packaging
 .Python
 build/

{flock_core-0.5.0b65 → flock_core-0.5.0b71}/AGENTS.md

@@ -234,6 +234,35 @@ The separation of `publish()` and `run_until_idle()` gives you **control over ex
 
 ---
 
+### 🗄️ Persistent Blackboard History (SQLite Store)
+
+**Production teams asked for auditability—now the blackboard can keep a full historical trail.**
+
+#### Why this matters
+- **Durable artifacts**: Everything on the blackboard (payloads, tags, visibility, correlation IDs) is stored on disk for replay, compliance, and debugging.
+- **Faster postmortems**: Query `/api/v1/artifacts` with filters (`type`, `produced_by`, `tags`, `visibility`, time windows) to reconstruct agent cascades after the fact.
+- **Operational dashboards**: The new **Historical Blackboard** module preloads persisted pages, exposes consumption metadata, and shows retention banners so operators know how far back they can scroll.
+- **Retention policies**: CLI helpers make maintenance routine—`flock sqlite-maintenance my.db --delete-before 2025-01-01T00:00:00Z --vacuum` prunes old artifacts and compacts the store.
+
+#### Quick start
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/history.db")
+await store.ensure_schema()
+
+flock = Flock("openai/gpt-4.1", store=store)
+await flock.publish(MyDreamPizza(pizza_idea="fermented garlic delight"))
+await flock.run_until_idle()
+```
+
+Kick the tyres with `examples/02-the-blackboard/01_persistent_pizza.py`, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` to inspect the retained history alongside live WebSocket updates.
+
+> **Heads-up:** The interface now returns `ArtifactEnvelope` objects when `embed_meta=True`. Future backends (Postgres, BigQuery, etc.) can implement the same contract to plug straight into the runtime and dashboard.
+
+---
+
 ### 🔒 Test Isolation and Mock Cleanup - The Contamination Trap
 
 **We fixed 32 failing tests caused by test contamination - here's what we learned:**

flock_core-0.5.0b65/README.md → flock_core-0.5.0b71/PKG-INFO

@@ -1,3 +1,35 @@
+Metadata-Version: 2.4
+Name: flock-core
+Version: 0.5.0b71
+Summary: Add your description here
+Author-email: Andre Ratzenberger <andre.ratzenberger@whiteduck.de>
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: aiosqlite>=0.20.0
+Requires-Dist: devtools>=0.12.2
+Requires-Dist: dspy==3.0.0
+Requires-Dist: duckdb>=1.1.0
+Requires-Dist: fastapi>=0.117.1
+Requires-Dist: hanging-threads>=2.0.7
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: litellm==1.75.3
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: mcp>=1.7.1
+Requires-Dist: opentelemetry-api>=1.30.0
+Requires-Dist: opentelemetry-exporter-jaeger-proto-grpc>=1.21.0
+Requires-Dist: opentelemetry-exporter-jaeger>=1.21.0
+Requires-Dist: opentelemetry-exporter-otlp>=1.30.0
+Requires-Dist: opentelemetry-instrumentation-logging>=0.51b0
+Requires-Dist: opentelemetry-sdk>=1.30.0
+Requires-Dist: poethepoet>=0.30.0
+Requires-Dist: pydantic[email]>=2.11.9
+Requires-Dist: rich>=14.1.0
+Requires-Dist: toml>=0.10.2
+Requires-Dist: typer>=0.19.2
+Requires-Dist: uvicorn>=0.37.0
+Requires-Dist: websockets>=15.0.1
+Description-Content-Type: text/markdown
+
 <p align="center">
   <img alt="Flock Banner" src="https://raw.githubusercontent.com/whiteducksoftware/flock/master/docs/assets/images/flock.png" width="800">
 </p>
@@ -252,6 +284,30 @@ asyncio.run(main())
 
 ---
 
+## Persistent Blackboard History
+
+The in-memory store is still great for local tinkering, but production teams now have a durable option. Plugging in `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
+
+- **Long-lived artifacts** — every field (payload, tags, partition keys, visibility) is stored for replay, audits, and postmortems
+- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
+- **Dashboard module** — the new **Historical Blackboard** experience preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
+- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
+
+Quick start:
+
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db")
+await store.ensure_schema()
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` and explore previous runs, consumption trails, and retention banners inside the dashboard.
+
+---
+
 ## Core Concepts
 
 ### Typed Artifacts (The Vocabulary)

flock_core-0.5.0b65/PKG-INFO → flock_core-0.5.0b71/README.md

@@ -1,34 +1,3 @@
-Metadata-Version: 2.4
-Name: flock-core
-Version: 0.5.0b65
-Summary: Add your description here
-Author-email: Andre Ratzenberger <andre.ratzenberger@whiteduck.de>
-License-File: LICENSE
-Requires-Python: >=3.10
-Requires-Dist: devtools>=0.12.2
-Requires-Dist: dspy==3.0.0
-Requires-Dist: duckdb>=1.1.0
-Requires-Dist: fastapi>=0.117.1
-Requires-Dist: hanging-threads>=2.0.7
-Requires-Dist: httpx>=0.28.1
-Requires-Dist: litellm==1.75.3
-Requires-Dist: loguru>=0.7.3
-Requires-Dist: mcp>=1.7.1
-Requires-Dist: opentelemetry-api>=1.30.0
-Requires-Dist: opentelemetry-exporter-jaeger-proto-grpc>=1.21.0
-Requires-Dist: opentelemetry-exporter-jaeger>=1.21.0
-Requires-Dist: opentelemetry-exporter-otlp>=1.30.0
-Requires-Dist: opentelemetry-instrumentation-logging>=0.51b0
-Requires-Dist: opentelemetry-sdk>=1.30.0
-Requires-Dist: poethepoet>=0.30.0
-Requires-Dist: pydantic[email]>=2.11.9
-Requires-Dist: rich>=14.1.0
-Requires-Dist: toml>=0.10.2
-Requires-Dist: typer>=0.19.2
-Requires-Dist: uvicorn>=0.37.0
-Requires-Dist: websockets>=15.0.1
-Description-Content-Type: text/markdown
-
 <p align="center">
   <img alt="Flock Banner" src="https://raw.githubusercontent.com/whiteducksoftware/flock/master/docs/assets/images/flock.png" width="800">
 </p>
@@ -283,6 +252,30 @@ asyncio.run(main())
 
 ---
 
+## Persistent Blackboard History
+
+The in-memory store is still great for local tinkering, but production teams now have a durable option. Plugging in `SQLiteBlackboardStore` turns the blackboard into a persistent event log with first-class ergonomics:
+
+- **Long-lived artifacts** — every field (payload, tags, partition keys, visibility) is stored for replay, audits, and postmortems
+- **Historical APIs** — `/api/v1/artifacts`, `/summary`, and `/agents/{agent_id}/history-summary` expose pagination, filtering, and consumption counts
+- **Dashboard module** — the new **Historical Blackboard** experience preloads persisted history, enriches the graph with consumer metadata, and highlights retention windows
+- **Operational tooling** — CLI helpers (`init-sqlite-store`, `sqlite-maintenance --delete-before ... --vacuum`) make schema setup and retention policies scriptable
+
+Quick start:
+
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db")
+await store.ensure_schema()
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+Run `examples/02-the-blackboard/01_persistent_pizza.py` to generate history, then launch `examples/03-the-dashboard/04_persistent_pizza_dashboard.py` and explore previous runs, consumption trails, and retention banners inside the dashboard.
+
+---
+
 ## Core Concepts
 
 ### Typed Artifacts (The Vocabulary)

flock_core-0.5.0b71/docs/guides/persistent-blackboard.md

@@ -0,0 +1,92 @@
+---
+title: Persistent Blackboard History
+description: Store artifacts beyond process lifetime with the SQLite-backed blackboard store and dashboard history module.
+tags:
+  - blackboard
+  - persistence
+  - sqlite
+  - dashboard
+search:
+  boost: 1.4
+---
+
+# Persistent Blackboard History
+
+The in-memory blackboard is perfect for local prototyping, but production teams need durable history for audits, debugging, and operator experience. The new **SQLiteBlackboardStore** turns every artifact into long-lived record while the dashboard's **Historical Blackboard** module visualises that history.
+
+---
+
+## Why Persist the Blackboard?
+
+- **Replay & Audits** — Inspect any published artifact (including payloads, tags, visibility, version, correlation IDs) even after the orchestrator restarts.
+- **Lifecycle Analytics** — Summaries and agent history endpoints expose production/consumption counts, visibility breakdowns, and tag trends over arbitrary windows.
+- **Operator Experience** — Dashboard users can scroll back through previous runs, inspect payload details, and view retention banners that explain how far history goes.
+- **Future Backends** — The updated store interface (`FilterConfig`, `ArtifactEnvelope`) paves the way for Postgres, DuckDB, or cloud warehouses with identical semantics.
+
+---
+
+## Swapping in the SQLite Store
+
+```python
+from flock import Flock
+from flock.store import SQLiteBlackboardStore
+
+store = SQLiteBlackboardStore(".flock/blackboard.db", timeout=5.0)
+await store.ensure_schema()  # idempotent; safe to call on startup
+
+flock = Flock("openai/gpt-4.1", store=store)
+```
+
+Key capabilities:
+
+- **Upserts by ID** — Duplicate artifact IDs overwrite previous payloads to remain consistent with the in-memory store.
+- **Full-field persistence** — All artifact fields (payload, tags, partition key, visibility) are saved for later hydration.
+- **Consumption tracking** — `record_consumptions()` writes `artifact_consumptions` rows so downstream dashboards know who consumed what.
+- **Operational helpers** — `sqlite-maintenance` CLI can prune (`--delete-before`) and vacuum the database on a schedule.
+
+> Tip: Run `examples/02-the-blackboard/01_persistent_pizza.py` to seed `.flock/examples/pizza_history.db`, then reuse that file across sessions.
+
+---
+
+## Historical APIs
+
+Once the SQLite store is active, the HTTP control plane exposes richer endpoints:
+
+| Endpoint | Purpose |
+| --- | --- |
+| `GET /api/v1/artifacts` | Paginated artifacts with filter support (type, producer, tags, visibility, correlation ID, time window). |
+| `GET /api/v1/artifacts/summary` | Aggregates by type, producer, visibility, tag counts, earliest/latest timestamps. |
+| `GET /api/v1/agents/{agent_id}/history-summary` | Produced/consumed totals per agent with optional filters. |
+
+Pass `embed_meta=true` to `GET /api/v1/artifacts` to receive `ArtifactEnvelope` responses that include consumption records (consumer IDs, run IDs, timestamps).
+
+---
+
+## Dashboard: Historical Blackboard Module
+
+Enable the dashboard with `await orchestrator.serve(dashboard=True)` or run `examples/03-the-dashboard/04_persistent_pizza_dashboard.py`. Then open **Add Module → Historical Blackboard** to:
+
+- Load persisted artifacts before WebSocket replay.
+- Apply the same multi-select filters as the REST API (types, producers, tags, visibility, correlation ID, time range).
+- Inspect payloads with the JSON viewer alongside consumer history (who read the artifact, when, and in which run).
+- See retention banners that highlight the earliest/latest artifacts currently stored and whether older pages can be fetched.
+- Stream new artifacts as they arrive—persisted history and live events share the same view.
+
+---
+
+## Retention & Maintenance
+
+- **Retention policy** — Use `sqlite-maintenance db.sqlite --delete-before <ISO-8601 timestamp>` to truncate old artifacts. Pair with `--vacuum` to reclaim disk space.
+- **Migrations** — `ensure_schema()` is idempotent and applies schema upgrades automatically. Call it during startup before processing any artifacts.
+- **Backups** — SQLite files can be copied or snapshotted; do this while the orchestrator is idle or after calling `await store.close()`.
+
+Future releases will add configurable retention windows and additional storage backends, but the contract exposed by `BlackboardStore` already supports them.
+
+---
+
+## Next Steps
+
+1. Switch local examples to `SQLiteBlackboardStore` and explore the persisted history via the dashboard module.
+2. Integrate the historical REST endpoints into operational tooling (e.g., CI smoke tests, analytics dashboards).
+3. Plan retention and maintenance jobs using the provided CLI commands.
+4. Contribute alternative store implementations by targeting the same `BlackboardStore` contract.

flock_core-0.5.0b71/docs/internal/sqlite-blackboard-store/README.md

@@ -0,0 +1,24 @@
+# SQLiteBlackboardStore Research Packet
+
+Welcome! This folder collects the working notes for implementing a persistent `SQLiteBlackboardStore`. Start here to understand what already exists and how to extend Flock Flow’s storage layer safely.
+
+## Quickstart
+
+1. **Read the domain rules** (`domain.md`) to learn the behavioural guarantees we must keep (duplicate handling, type resolution, visibility data, retention expectations).
+2. **Review the implementation pattern** (`patterns.md`) for schema design, transaction strategy, and migration guidance.
+3. **Check the integration notes** (`interfaces.md`) to see which APIs and dashboard affordances need to evolve once historical storage is available.
+4. **Plan next steps** using the “Action Items” section below when you’re ready to convert research into issues or pull requests.
+
+## Folder Contents
+
+| File | Purpose |
+|------|---------|
+| `domain.md` | Business and lifecycle rules the SQLite store must preserve. |
+| `patterns.md` | Technical blueprint for schema, read/write flows, concurrency, and observability. |
+| `interfaces.md` | Backend API extensions and dashboard UX recommendations for historical data access. |
+
+## Action Items
+
+- Validate schema design and indexing strategy against projected artifact volumes.
+- Scope required FastAPI endpoints and dashboard filter updates into implementation tickets.
+- Prototype concurrency benchmarks (publish throughput, WAL performance) to confirm operational limits.

flock_core-0.5.0b71/docs/internal/sqlite-blackboard-store/dashboard-playwright-mcp.md

@@ -0,0 +1,44 @@
+# Manual Dashboard Verification (playwright-mcp)
+
+This checklist replaces the original automated Playwright suite. It uses the
+`playwright-mcp` tool to exercise the historical dashboard manually.
+
+## Prerequisites
+
+- SQLite backend configuration available (see project README).
+- `uv` CLI installed.
+- `playwright-mcp` MCP extension enabled in the Codex CLI environment.
+
+## Launch the Dashboard Edge-Case Scenario
+
+```bash
+uv run python examples/03-the-dashboard/02-dashboard-edge-cases.py
+```
+
+Wait for the following log lines:
+
+- `[Dashboard] Production build completed`
+- `Uvicorn running on http://127.0.0.1:8000`
+- `[Dashboard] Browser launched successfully`
+
+## Manual Verification Steps (playwright-mcp)
+
+1. `mcp__playwright__browser_navigate(url="http://127.0.0.1:8000")`
+   - Confirm Agent View renders all agents with idle status.
+2. Close the Publish panel (`mcp__playwright__browser_click` with `Close publish panel`) and switch to **Blackboard View**.
+3. Toggle the **Filters** panel and verify:
+   - Correlation search, time range presets, multi-select filters, and saved preset controls appear.
+4. Enter a sample correlation ID (e.g., `"abc123"`) to ensure filter pills update.
+5. Re-open the Filters panel and ensure layout remains stable (no overlap between time-range buttons and fields).
+6. Interact with the **Historical Blackboard** module (if enabled):
+   - Trigger “Load Older” and capture behavior.
+7. Capture a screenshot with `mcp__playwright__browser_take_screenshot(fullPage=True, ...)`.
+
+## Expected Outcomes
+
+- No React errors or ErrorBoundary screens.
+- Filter controls render without overlapping.
+- Manual interactions update filter pills and historical listings.
+- WebSocket reconnect warnings are acceptable when the example shuts down.
+
+Record findings in the Phase 3 validation notes before closing tasks.

flock_core-0.5.0b71/docs/internal/sqlite-blackboard-store/domain.md

@@ -0,0 +1,23 @@
+# SQLiteBlackboardStore Domain Considerations
+
+## Artifact Lifecycle Rules
+
+- A SQLite-backed store must preserve the baseline contract defined by `BlackboardStore.publish`, `get`, `list`, and `list_by_type` so the orchestrator can persist an artifact, bump `artifacts_published`, and immediately schedule consumers without additional coordination (`src/flock/orchestrator.py:807`).
+- Artifacts carry canonical business data—including correlation IDs, partition keys, tags, visibility, timestamps, and version counters—because the `Artifact` model already exposes them (`src/flock/artifacts.py:15`). Every field must be recorded to guarantee downstream features like visibility filtering and replay work unchanged.
+- Duplicate publishes with the same UUID intentionally overwrite prior values: the tests assert “latest wins” semantics (`tests/test_store.py:152`). A persistent store therefore needs an UPSERT strategy keyed on `artifact_id` so callers never see stale payloads.
+
+## Retrieval & Type Normalisation
+
+- `list()` is expected to return artifacts in insertion order (`tests/test_store.py:126`); a SQLite implementation should sort by `created_at` (or a monotonic sequence) to mirror in-memory behaviour.
+- Type-aware queries must keep working: `list_by_type("Document")` resolves the canonical name through `type_registry.resolve_name` before filtering (`src/flock/store.py:73`). That contract is validated by the storage contract tests which mix canonical and simple names (`tests/contract/test_artifact_storage_contract.py:56`). A SQLite store therefore needs a column that stores the canonical type string to make these lookups efficient.
+- `get_by_type(ModelClass)` returns fully materialised Pydantic models (`src/flock/store.py:78`). Persisted payloads must remain JSON-serialised dictionaries so they can be rehydrated into model instances without losing validation logic.
+
+## Visibility & Scheduling Guarantees
+
+- Visibility rules (`PublicVisibility`, label/tenant controls, future embargoes) are enforced when the orchestrator schedules agents, so the store has to persist the original `visibility` payload intact for enforcement and later audits (`src/flock/orchestrator.py:826`).
+- Because correlation IDs and partition keys participate in routing, they must remain queryable fields. The dashboard’s IndexedDB schema already assumes correlation and producer filters (`src/flock/frontend/src/services/indexeddb.ts:55`), so the canonical store must expose the same columns to keep downstream consumers consistent.
+
+## Historical Retention & Audit
+
+- Moving from volatile memory to SQLite adds expectations around retention windows, GDPR/PII erasure, and audit trails. The store should document default retention, compaction, and purging policies so teams know when old artifacts are deleted or archived.
+- Persistent history enables richer diagnostic tools (e.g., the DuckDB trace statistics endpoint at `src/flock/dashboard/service.py:683`). A SQLite store should outline how historical artifact queries and export workflows stay performant without overwhelming the orchestrator.

flock_core-0.5.0b71/docs/internal/sqlite-blackboard-store/interfaces.md

@@ -0,0 +1,25 @@
+# SQLite Blackboard Store Interfaces & Dashboard UX
+
+## Orchestrator Integration
+
+- Expose the store via `Flock(store=SQLiteBlackboardStore(db_path="..."))` so existing constructors keep working (`src/flock/orchestrator.py:807`).
+- Provide async `startup()` / `shutdown()` hooks to initialise connections and close them gracefully when the dashboard or CLI stops, matching the lifecycle of `DashboardHTTPService`.
+
+## Backend APIs for Historical Data
+
+- Extend the FastAPI layer with a paginated `GET /api/v1/artifacts` endpoint that supports query parameters: `type`, `produced_by`, `correlation_id`, `tag`, `from`, `to`, `limit`, `offset`. Results should mirror the artifact schema returned by `GET /api/v1/artifacts/{artifact_id}` (`src/flock/service.py:86`).
+- Add an optional `GET /api/v1/artifacts/summary` endpoint for aggregations (counts per type, per producer) to let the dashboard populate filters without scanning the entire table.
+- Keep WebSocket streaming history for live sessions, but backfill the IndexedDB cache on load by calling the new history endpoint before subscribing to real-time events. That honours the existing IndexedDB schema which already tracks correlation IDs, producers, and timestamps (`src/flock/frontend/src/services/indexeddb.ts:55`).
+
+## Dashboard Filter & View Enhancements
+
+- The current filter store only allows correlation ID and time-range filters (`src/flock/frontend/src/store/filterStore.ts:11`). Persisted history will overwhelm this UX; add multi-select filters for artifact type, producing agent, tags, and visibility class so operators can slice larger datasets.
+- Introduce a “Historical Blackboard” view that renders a virtualised table or timeline of artifacts, pulling pages from SQLite via the new endpoints and letting users jump to a specific time window.
+- Provide saved filter presets (the IndexedDB schema already reserves a `filters` store, `src/flock/frontend/src/services/indexeddb.ts:127`) so analysts can recall frequent queries quickly.
+- For streaming output, keep the existing `/api/streaming-history/{agent_name}` endpoint (`src/flock/dashboard/service.py:703`) but document how far back history can be fetched and surface UI affordances (e.g., “Load older output”) that rely on SQLite retention policies.
+
+## Usability Guardrails
+
+- Indicate the active dataset (live vs archived) directly in the dashboard to avoid confusion when viewing historical records.
+- Add server-side limits (max rows per page, time-range caps) with clear UI feedback to prevent slow queries and keep SQLite responsive.
+- Provide export actions (CSV/JSON) for selected artifacts so teams can move historical data into external analytics tools without direct database access.

flock_core-0.5.0b71/docs/internal/sqlite-blackboard-store/patterns.md

@@ -0,0 +1,44 @@
+# SQLiteBlackboardStore Implementation Pattern
+
+## Storage Strategy
+
+- Use `aiosqlite` (or another async driver) so `publish`, `get`, and `list*` stay awaitable and integrate with the orchestrator’s async scheduling pipeline (`src/flock/orchestrator.py:807`).
+- Open one shared connection per process with `row_factory=aiosqlite.Row`, wrap statements in `asyncio.Lock`/connection-level transactions, and enable WAL mode plus `journal_mode=wal` + `synchronous=normal` to reduce writer contention.
+
+## Schema Design
+
+| Table | Purpose | Key Columns |
+|-------|---------|-------------|
+| `artifacts` | Primary artifact store | `artifact_id` (PK TEXT UUID), `type`, `canonical_type`, `produced_by`, `payload` (JSON), `version`, `visibility` (JSON), `tags` (TEXT[] as JSON), `correlation_id`, `partition_key`, `created_at` (TIMESTAMP) |
+| `artifact_consumers` (optional) | Future-proof actual consumer tracking | `artifact_id`, `agent_name`, `consumed_at` |
+
+- Indexes: `(canonical_type, created_at)`, `(produced_by, created_at)`, `(correlation_id)`, `(tags_json -> '$[*]')` using `json_each` for tag filtering, plus `(partition_key)`. These support list-by-type, producer history, correlation filters, and eventual dashboard queries (`src/flock/frontend/src/services/indexeddb.ts:55`).
+
+## Write Path (`publish`)
+
+1. Resolve canonical type via `type_registry.resolve_name` before persistence so simple names remain queryable (`src/flock/store.py:73`).
+2. Convert sets (`tags`) and visibility models to JSON. Persist `created_at` using the artifact’s timestamp rather than `CURRENT_TIMESTAMP` to preserve ingest order (`src/flock/artifacts.py:26`).
+3. Execute an UPSERT on `artifact_id`. On conflict update payload, metadata, version, and `created_at` so duplicate publishes follow the “latest wins” rule (`tests/test_store.py:152`).
+
+## Read Path (`get`, `list`, `list_by_type`, `get_by_type`)
+
+- `get` selects by `artifact_id` and rehydrates the Pydantic model by feeding the stored payload into the registered class (`src/flock/store.py:79`).
+- `list` orders by `created_at ASC`, returning rows mapped back to `Artifact`.
+- `list_by_type` filters on `canonical_type` so both canonical and simple names pass the contract tests (`tests/contract/test_artifact_storage_contract.py:56`).
+- `get_by_type` builds the target model directly by deserialising JSON payloads (no extra round-trip through `Artifact`), ensuring users still receive typed lists.
+
+## Concurrency & Backpressure
+
+- SQLite serialises writes; wrap publishes in short transactions and batch inserts when ingesting cascades. For heavy workloads, queue writes through an `asyncio.TaskGroup` that coalesces `publish` calls to amortise commit cost.
+- Explicitly document write throughput limits and point teams toward sharding or an alternative store for high-frequency publish workloads.
+
+## Schema Migration & Maintenance
+
+- Ship an `ensure_schema()` coroutine that runs during store initialisation to create tables, indexes, and PRAGMA settings idempotently.
+- Provide a migration table (`schema_version INTEGER`) so future upgrades (e.g., adding artifact lineages) can run without manual SQL.
+- Offer housekeeping helpers: vacuum/pragma toggles, “delete before timestamp” utilities for retention, and export routines for snapshotting.
+
+## Observability & Testing
+
+- Emit OpenTelemetry spans around SQL operations so they appear in the existing DuckDB trace summaries (`src/flock/dashboard/service.py:683`).
+- Reuse `tests/test_store.py` and `tests/contract/test_artifact_storage_contract.py` via parametrised fixtures to validate behaviour parity, then add integration tests for persistence, WAL mode, and concurrent writers to cover new failure modes.

flock_core-0.5.0b71/docs/specs/001-sqlite-blackboard-store/PLAN.md

@@ -0,0 +1,169 @@
+# Implementation Plan
+
+## Validation Checklist
+- [ ] Context Ingestion section complete with all required specs
+- [ ] Implementation phases logically organized
+- [ ] Each phase starts with test definition (TDD approach)
+- [ ] Dependencies between phases identified
+- [ ] Parallel execution marked where applicable
+- [ ] Multi-component coordination identified (if applicable)
+- [ ] Final validation phase included
+- [ ] No placeholder content remains
+
+## Specification Compliance Guidelines
+
+### How to Ensure Specification Adherence
+
+1. **Before Each Phase**: Complete the Pre-Implementation Specification Gate
+2. **During Implementation**: Reference specific SDD sections in each task
+3. **After Each Task**: Run Specification Compliance checks
+4. **Phase Completion**: Verify all specification requirements are met
+
+### Deviation Protocol
+
+If implementation cannot follow specification exactly:
+1. Document the deviation and reason
+2. Get approval before proceeding
+3. Update SDD if the deviation is an improvement
+4. Never deviate without documentation
+
+## Metadata Reference
+
+- `[parallel: true]` - Tasks that can run concurrently
+- `[component: component-name]` - For multi-component features
+- `[ref: document/section; lines: 1, 2-3]` - Links to specifications, patterns, or interfaces and (if applicable) line(s)
+- `[activity: type]` - Activity hint for specialist agent selection
+
+---
+
+## Context Priming
+
+*GATE: You MUST fully read all files mentioned in this section before starting any implementation.*
+
+**Specification**:
+
+- `docs/internal/sqlite-blackboard-store/domain.md` - Domain rules for persistent blackboard storage
+- `docs/internal/sqlite-blackboard-store/patterns.md` - Implementation blueprint for SQLite store
+- `docs/internal/sqlite-blackboard-store/interfaces.md` - API and dashboard interface expectations
+
+**Key Design Decisions**:
+
+- SQLite must faithfully reproduce current in-memory semantics (canonical type resolution, ordered reads, duplicate overwrite) while persisting every `Artifact` field for visibility and audit parity.
+- Historical access requires new paginated artifact APIs and richer dashboard filters so operators can query large datasets without overwhelming the live graph.
+
+**Implementation Context**:
+
+- Commands to run: `uv run pytest tests/test_store.py tests/contract/test_artifact_storage_contract.py`, `uv run pytest tests/dashboard -k artifact` (after adding cases), `uv run python examples/03-the-dashboard/02-dashboard-edge-cases.py` followed by manual validation with `playwright-mcp`.
+- Patterns to follow: `docs/internal/sqlite-blackboard-store/patterns.md` (schema, write/read paths, concurrency, migrations).
+- Interfaces to implement: `docs/internal/sqlite-blackboard-store/interfaces.md` (REST endpoints for historical artifacts, dashboard UX enhancements).
+
+---
+
+## Implementation Phases
+
+- [x] **Phase 1**: Foundation — SQLite schema & store implementation
+
+    - [x] **Prime Context**: `docs/internal/sqlite-blackboard-store/domain.md`, `docs/internal/sqlite-blackboard-store/patterns.md`
+        - [x] Confirm semantic parity requirements (canonical type usage, overwrite behaviour) `[ref: docs/internal/sqlite-blackboard-store/domain.md]`
+        - [x] Review schema blueprint, migration strategy, and concurrency guidance `[ref: docs/internal/sqlite-blackboard-store/patterns.md]`
+    - [x] **Write Tests**: Extend store contract coverage for SQLite backend `[activity: design-tests]`
+        - [x] Parameterise `tests/test_store.py` to run against both in-memory and SQLite implementations `[ref: tests/test_store.py; activity: update-tests]`
+        - [x] Mirror `tests/contract/test_artifact_storage_contract.py` against SQLite store to enforce canonical name resolution `[ref: tests/contract/test_artifact_storage_contract.py; activity: update-tests]`
+        - [x] Add migration/initialisation regression test (e.g., ensuring `ensure_schema()` idempotency) `[activity: add-tests]`
+    - [x] **Implement**: Introduce `SQLiteBlackboardStore` following async pattern `[activity: implement-backend]`
+        - [x] Create schema management (`ensure_schema`, migrations, PRAGMA setup) and connection lifecycle helpers `[ref: docs/internal/sqlite-blackboard-store/patterns.md]`
+        - [x] Implement `publish`, `get`, `list`, `list_by_type`, `get_by_type`, honouring duplicate overwrite, ordered reads, and JSON payload hydration `[activity: implement-backend]`
+    - [x] **Implement**: Observability & maintenance tooling `[activity: implement-backend]`
+        - [x] Add OpenTelemetry spans around SQL operations, vacuum/retention helpers as defined in research `[ref: docs/internal/sqlite-blackboard-store/patterns.md]`
+    - [x] **Validate**: Ensure reliability & parity
+        - [x] Run updated pytest suites (`tests/test_store.py`, `tests/contract/test_artifact_storage_contract.py`) `[activity: run-tests]`
+        - [x] Execute concurrency stress test (async gather publish) against SQLite backend `[activity: run-tests]`
+        - [x] Review schema artefacts for adherence to domain rules and document retention defaults `[activity: review-docs]`
+
+- [x] **Phase 2**: Orchestrator & service integration
+
+    - [x] **Prime Context**: `docs/internal/sqlite-blackboard-store/interfaces.md`, orchestrator/service modules (`src/flock/orchestrator.py`, `src/flock/service.py`)
+        - [x] Map new store into orchestrator lifecycle & dependency injection `[ref: docs/internal/sqlite-blackboard-store/interfaces.md]`
+        - [x] Review API expansion requirements for artifact history `[ref: docs/internal/sqlite-blackboard-store/interfaces.md]`
+    - [x] **Write Tests**: Define acceptance tests for new endpoints & configuration `[activity: design-tests]`
+        - [x] Add API contract tests for paginated artifact listing (`/api/v1/artifacts`) and summaries `[activity: add-tests]`
+        - [x] Create integration test ensuring `Flock(store=SQLiteBlackboardStore(...))` publishes/reads successfully `[activity: add-tests]`
+    - [x] **Implement**: Wiring the store into runtime `[activity: implement-backend]`
+        - [x] Allow `Flock` to accept SQLite store via configuration/CLI, update CLI helpers and documentation `[activity: implement-backend]`
+        - [x] Extend FastAPI service with historical artifact endpoints (pagination, filtering, summaries) `[activity: implement-backend]`
+    - [x] **Implement**: Migration & configuration tooling `[activity: implement-backend]`
+        - [x] Provide CLI commands or scripts to initialise database, run migrations, and configure retention `[activity: implement-backend]`
+    - [x] **Validate**: Cross-check orchestration guarantees
+        - [x] Run FastAPI endpoint tests and contract suites `[activity: run-tests]`
+        - [x] Smoke test CLI/orchestrator flows using SQLite backend (publish/list) `[activity: run-tests]`
+        - [x] Verify metrics/tracing capture database operations as required `[activity: review-logs]`
+
+- [~] **Phase 3**: Dashboard & UX enhancements for historical data
+
+    - [x] **Prime Context**: `docs/internal/sqlite-blackboard-store/interfaces.md`, dashboard filter implementation (`src/flock/frontend/src/store/filterStore.ts`, `src/flock/frontend/src/services/indexeddb.ts`)
+        - [x] Review required filter additions (artifact type, producer, tags, visibility) `[ref: docs/internal/sqlite-blackboard-store/interfaces.md]`
+        - [x] Inspect IndexedDB schema alignment for saved filters/pagination `[ref: src/flock/frontend/src/services/indexeddb.ts]`
+    - [x] **Write Tests**: Outline UI regression coverage `[activity: design-tests]`
+        - [x] Add unit tests for new filter store state, filters UI, and saved preset persistence `[activity: add-tests]`
+        - [x] Document manual playwright-mcp flow for historical blackboard view and pagination `[ref: docs/internal/sqlite-blackboard-store/dashboard-playwright-mcp.md; activity: add-tests]`
+    - [ ] **Implement**: Dashboard data ingestion `[activity: implement-frontend]`
+        - [x] Fetch historical artifact pages on load before WebSocket subscription, hydrate IndexedDB caches `[activity: implement-frontend]`
+        - [x] Add multi-select filters (type, producer, tags, visibility) and saved presets UI `[activity: implement-frontend]`
+        - [x] Provide persistent data seeding example (`examples/02-the-blackboard/01_persistent_pizza.py`) `[activity: implement-frontend]`
+        - [x] Backend follow-up: expose embedded consumption metadata via store API and REST (see discussion note)
+    - [ ] **Implement**: New historical view & affordances `[activity: implement-frontend]`
+        - [x] Create paginated “Historical Blackboard” table/timeline with virtualization `[activity: implement-frontend]`
+        - [x] Surface retention messaging and “Load older output” interactions tied to SQLite limits `[activity: implement-frontend]`
+        - [x] Add payload detail viewer using shared JSON component
+        - [x] Remove legacy Event Log module once payload viewer shipped
+    - [x] **Implement**: Historical backend API enhancements `[activity: implement-backend]`
+    - [x] Add consumption history persistence (`artifact_consumptions` or equivalent)
+    - [x] Update store interface (`FilterConfig`, `embed_meta`) and document requirements for future stores
+    - [x] Extend REST endpoints (`/api/v1/artifacts`, `/summary`, `/agents/{id}/history-summary`) to surface metadata
+    - [x] Ensure in-memory store honours new interface semantics for parity
+    - [ ] Document design rationale (embedded metadata, consumption joins, normalization)
+
+- [ ] **Validate**: Store/REST integration `[activity: run-tests]`
+    - [x] Add unit tests for store consumption queries and filters
+    - [x] Add integration tests for new REST endpoints and envelopes
+    - [x] Verify frontend agent counters/edges consume enriched envelopes
+
+- [ ] **Validate**: UX & performance gates
+        - [ ] Run frontend unit tests and manual dashboard verification with `playwright-mcp` following `docs/internal/sqlite-blackboard-store/dashboard-playwright-mcp.md` (`uv run python examples/03-the-dashboard/02-dashboard-edge-cases.py`) `[activity: run-tests]` *(unit tests passing; Playwright runner currently blocks on CSS module parsing — see latest execution notes)*
+        - [ ] Perform manual UX review of persistent history flow (`uv run python examples/03-the-dashboard/04_persistent_pizza_dashboard.py`) before dashboard checks `[activity: manual-test]`
+        - [ ] Confirm accessibility checks (keyboard navigation, screen-reader labels) for new components `[activity: accessibility-review]`
+
+- [ ] **Integration & End-to-End Validation**
+    - [ ] Confirm unit and contract tests pass for both in-memory and SQLite stores
+    - [ ] Execute integration tests covering orchestrator/API/dashboard interplay with SQLite backend
+    - [ ] Run end-to-end workflow (publish artifacts, query history, filter in dashboard) against SQLite store
+    - [ ] Validate performance targets for write throughput and dashboard pagination `[ref: docs/internal/sqlite-blackboard-store/patterns.md]`
+    - [ ] Perform security review for data-at-rest, visibility enforcement, and endpoint access controls `[ref: docs/internal/sqlite-blackboard-store/domain.md]`
+    - [ ] Verify acceptance criteria from research packet (parity, historical access, UX enhancements)
+    - [ ] Ensure documentation updated (README, CLI help, dashboard docs) for new backend option
+    - [ ] Run build, lint, and formatting checks; confirm deployment scripts handle SQLite configuration
+    - [ ] Record final specification compliance report before release
+
+- [ ] **Integration & End-to-End Validation**
+    - [ ] [All unit tests passing per component if multi-component]
+    - [ ] [Integration tests for component interactions]
+    - [ ] [End-to-end tests for complete user flows]
+    - [ ] [Performance tests meet requirements] `[ref: SDD/Section 10 "Quality Requirements"]`
+    - [ ] [Security validation passes] `[ref: SDD/Section 10 "Security Requirements"]`
+    - [ ] [Acceptance criteria verified against PRD] `[ref: PRD acceptance criteria sections]`
+    - [ ] [Test coverage meets standards]
+    - [ ] [Documentation updated for any API/interface changes]
+    - [ ] [Build and deployment verification]
+    - [ ] [All PRD requirements implemented]
+    - [ ] [Implementation follows SDD design]
+
+### 📝 Phase 3 Design Notes (Added after implementation review)
+
+During integration we discovered the front-end needs richer metadata than the original research anticipated:
+
+- Agent counters and graph edges require historical consumption data; this led us to design `artifact_consumptions` persistence and an `embed_meta` path in the store interface.
+- Stores (in-memory & SQLite) will expose filtering through a shared `FilterConfig` and can return either lean `ArtifactRecord` objects or enriched envelopes with consumption/run metadata.
+- REST endpoints will surface the enriched envelopes directly so clients do not reimplement the joins. A dedicated `/api/v1/agents/{id}/history-summary` endpoint will provide filtered produced/consumed counts.
+- Historical Blackboard replaces the Event Log. Payload viewing will use the shared JSON inspector component.
+
+These tasks are tracked in Phase 3 to guide future contributors implementing new stores (e.g., PostgresStore, RedisStore).