@yemi33/minions 0.1.2373 → 0.1.2374

package/docs/design-inbox-entries-schema.md

@@ -0,0 +1,106 @@
+# inbox_entries Table Schema
+
+## Overview
+The `inbox_entries` table is the persistent store for operator and agent inbox messages within the Minions orchestration engine. It provides a time-ordered, unread-tracking message queue decoupled from the runtime work-item/dispatch lifecycle.
+
+## Schema Specification
+
+### Column Definitions
+
+| Column | Type | Constraints | Purpose |
+|--------|------|-----------|---------|
+| `id` | TEXT | PRIMARY KEY | Unique message identifier; generated as `<recipient>-<timestamp>-<sender>-<seq>` format |
+| `recipient_session_id` | TEXT | NOT NULL | Session ID of the inbox owner (operator or agent); routes messages to specific actor |
+| `sender_id` | TEXT | NOT NULL | Identifier of the message originator; structured as `<agent-id>` or `'system'` or `'operator'` |
+| `sender_name` | TEXT | NOT NULL | Human-readable sender name for UI display; persisted at write-time for offline/archived reference |
+| `sender_type` | TEXT | NOT NULL | Enum: `'agent'`, `'system'`, `'operator'`; constrains allowed sender_id values |
+| `interaction_id` | TEXT | NOT NULL | Group ID linking related messages (e.g., work-item dispatch feedback, PR review batch); enables threading and deduplication |
+| `sequence` | INTEGER | NOT NULL, DEFAULT 0 | Per-interaction message counter; disambiguates bursts and enables ordering within same millisecond |
+| `summary` | TEXT | NOT NULL | Title/subject line; used for list views and user scanning; always present even if content is truncated |
+| `content` | TEXT | NOT NULL | Full message body; may contain Markdown, code blocks, or structured JSON sidecars (not yet parsed at table level) |
+| `unread` | INTEGER | NOT NULL, DEFAULT 1 | Boolean flag (0/1); toggled by reader, cleared on archive/delete; enables fast unread-count queries |
+| `sent_at` | INTEGER | NOT NULL | Unix timestamp (milliseconds) of message creation; primary sort key for time-ordered display |
+| `read_at` | INTEGER | Nullable | Unix timestamp (milliseconds) when recipient first read the message; null until marked read |
+| `notified_at` | INTEGER | Nullable | Unix timestamp (milliseconds) of last notification event sent to operator; null if no notification dispatched |
+
+### Constraints & Relationships
+
+- **PK constraint:** `id` is globally unique across all inboxes; collision risk is negligible given the composite structure (recipient + timestamp + sender + seq).
+- **FK (soft):** `recipient_session_id` does NOT enforce a referential constraint to `sessions` table (sessions may be archived). Queries SHOULD handle missing recipients gracefully.
+- **FK (soft):** `sender_id` is informational only; no validation that the sender exists in the `config.agents` or active sessions. Invalid senders are logged but do not block writes.
+- **No composite index on (recipient_session_id, sent_at):** See Performance section.
+
+## Architectural Rationale
+
+### Design Decisions
+
+1. **Separate inbox table vs. inline dispatch/work-item attachments.**
+   - Inboxes can outlive their source work items (e.g., a feedback message remains useful after the WI is done).
+   - Decoupling allows independent inbox lifecycle (archive, export, cleanup) without cascading work-item deletes.
+   - Supports rich async consolidation (team notes, KB articles) that don't belong to any single WI.
+
+2. **`unread` flag instead of soft-delete or status enum.**
+   - Fast count queries for badge (`SELECT COUNT(*) WHERE recipient_session_id = ? AND unread = 1`).
+   - Unread state is mutable; allows mark-as-read UX without data loss.
+   - Simplifies archive/export (no tombstone filtering needed).
+
+3. **`sender_type` as an explicit column.**
+   - Allows schema validation (e.g., "agent" senders must have valid IDs; "system" is always valid).
+   - Enables filtered views ("show only agent messages" or "system alerts only") without string parsing.
+
+4. **`interaction_id` for grouping.**
+   - Enables undo/redo on related messages (e.g., canceling all build-failure notifications from one dispatch).
+   - Supports consolidation: multiple small notifications can be collapsed into one summary message.
+
+5. **Persistent `sender_name` at write-time.**
+   - Operator may rename agents or disable them; storing the name ensures historical accuracy.
+   - UI doesn't need to perform runtime lookups on archived inboxes.
+
+6. **Unix timestamps (milliseconds) instead of ISO strings.**
+   - Enables range queries (`sent_at BETWEEN ? AND ?`) without parsing.
+   - Smaller column footprint; easier to sort.
+   - Downside: not human-readable; UI must convert to locale strings.
+
+### Entity Relationships
+
+- **inbox_entries ← dispatch WI feedback:** A failed dispatch may create multiple inbox entries (error summary, retry recommendation, logs snippet). Threaded via `interaction_id`.
+- **inbox_entries ← PR reviews:** Humans and auto-review agents post comments; summarized as inbox alerts to project owners via `interaction_id` per-PR.
+- **inbox_entries ← consolidation:** `engine/consolidation.js` reads inbox entries to extract `agent:` frontmatter and route findings into per-agent knowledge files.
+
+## Performance Considerations
+
+### Indexing Strategy
+
+**Current indexes (recommended):**
+- `idx_recipient_sent` on `(recipient_session_id, sent_at DESC)` — enables efficient "unread count" and "latest N messages" queries without table scans.
+- `idx_interaction` on `(interaction_id)` — enables grouping/dedup queries (e.g., "was this interaction already notified?").
+- PK `id` is implicit index.
+
+**Query patterns:**
+- **Unread count per session:** `SELECT COUNT(*) WHERE recipient_session_id = ? AND unread = 1` → must scan `idx_recipient_sent` to avoid full table.
+- **Latest messages:** `SELECT * WHERE recipient_session_id = ? ORDER BY sent_at DESC LIMIT 20` → seeks on `idx_recipient_sent` DESC.
+- **Thread view:** `SELECT * WHERE interaction_id = ? ORDER BY sent_at` → seeks on `idx_interaction`, then sorts in application or via secondary index.
+- **Cleanup:** `DELETE WHERE sent_at < ? AND unread = 0` → full table scan (acceptable; cleanup is infrequent, low-priority background task).
+
+**Missing indexes (future optimization):**
+- `idx_sender_id` — if queries like "messages from this agent" are common, consider adding.
+- `idx_unread` — if badge ("N unread") is a hot query, consider combining with recipient (`idx_recipient_unread` on `(recipient_session_id, unread)`).
+
+### Size Estimates & Cleanup
+
+- **Row size:** ~500–2000 bytes (summary + content; content can be verbose with logs/code snippets).
+- **Growth:** ~5–50 entries/day (depends on agent activity and verbose flagging).
+- **Retention:** No automatic cleanup; operators may archive/export and manually delete aged entries. Engine does NOT auto-prune.
+- **Recommended**: Implement a dashboard setting to soft-delete entries older than N days (default 30) with an undo/export option.
+
+## Migration & Backward Compatibility
+
+- **Migration 018 or later** creates `inbox_entries` as part of SQL state cutover. Legacy `inbox/` Markdown files are NOT imported; operators must re-send or manually recreate critical messages.
+- **No schema versioning needed** for now; table is immutable-append-only (reads/writes only).
+
+## Related Code
+
+- **Writes:** `engine/inbox-store.js` (primary mutator; uses `getDb()` and `emitStateEvent`).
+- **Reads:** `engine/queries.js#getInboxForSession()`, dashboard `/api/inbox`, `engine/consolidation.js`.
+- **UI:** `dashboard/pages/inbox.html`, `dashboard/js/inbox.js`.
+

package/engine/db/migrations/019-inbox-entries.js

@@ -0,0 +1,37 @@
+// engine/db/migrations/019-inbox-entries.js
+//
+// Creates the `inbox_entries` table: persistent store for operator and agent
+// inbox messages, decoupled from the runtime work-item/dispatch lifecycle.
+//
+// Schema: columns, types, constraints, and indexing strategy documented in
+// docs/design-inbox-entries-schema.md
+
+module.exports = {
+  version: 19,
+  description: 'inbox_entries: operator/agent message queue with full schema + indexes',
+  up(db) {
+    db.exec(`
+      CREATE TABLE inbox_entries (
+        id                  TEXT PRIMARY KEY,
+        recipient_session_id TEXT NOT NULL,
+        sender_id           TEXT NOT NULL,
+        sender_name         TEXT NOT NULL,
+        sender_type         TEXT NOT NULL,
+        interaction_id      TEXT NOT NULL,
+        sequence            INTEGER NOT NULL DEFAULT 0,
+        summary             TEXT NOT NULL,
+        content             TEXT NOT NULL,
+        unread              INTEGER NOT NULL DEFAULT 1,
+        sent_at             INTEGER NOT NULL,
+        read_at             INTEGER,
+        notified_at         INTEGER
+      );
+
+      CREATE INDEX idx_inbox_recipient_sent 
+        ON inbox_entries(recipient_session_id, sent_at DESC);
+
+      CREATE INDEX idx_inbox_interaction 
+        ON inbox_entries(interaction_id);
+    `);
+  },
+};

package/engine/inbox-store.js

@@ -0,0 +1,276 @@
+// engine/inbox-store.js — SQL-backed inbox_entries table handlers.
+//
+// Provides read/write helpers for the operator and agent inbox message queue.
+// Inboxes are decoupled from the runtime work-item/dispatch lifecycle and
+// support threading via interaction_id, unread tracking, and time-ordered display.
+//
+// Schema: docs/design-inbox-entries-schema.md
+
+const crypto = require('node:crypto');
+const { getDb, withTransaction } = require('./db');
+
+// In-process monotonic counter. Millisecond timestamps are too coarse to
+// disambiguate bursts of createInboxEntry() calls for the same
+// recipient+sender pair (e.g. rapid consolidation/notification writes), and
+// the caller-supplied `sequence` field is not guaranteed to be unique either
+// (it defaults to 0 and callers are not required to increment it). The
+// counter guarantees every id generated within this process is distinct even
+// when Date.now() does not advance between calls.
+let _inboxIdCounter = 0;
+
+// Generate a likely-unique inbox entry ID:
+// <recipient>-<timestamp>-<sender>-<seq>-<counter>-<random>
+// `sequence` is preserved for callers that pass a meaningful value; the
+// monotonic counter + random suffix are what actually make the id
+// collision-proof, independent of clock resolution or caller behavior.
+function _generateInboxId(recipientSessionId, senderId, sequence) {
+  const now = Date.now();
+  _inboxIdCounter = (_inboxIdCounter + 1) % 0x1000000;
+  const rand = crypto.randomBytes(4).toString('hex');
+  return `${recipientSessionId}-${now}-${senderId}-${sequence}-${_inboxIdCounter.toString(36)}-${rand}`;
+}
+
+// Read all inbox entries for a given recipient session, ordered by time descending.
+function _readInboxEntriesFromSql(db, { recipientSessionId } = {}) {
+  let query = 'SELECT * FROM inbox_entries';
+  const params = [];
+  if (recipientSessionId) {
+    query += ' WHERE recipient_session_id = ?';
+    params.push(recipientSessionId);
+  }
+  query += ' ORDER BY sent_at DESC';
+  return db.prepare(query).all(...params);
+}
+
+// Read unread count for a recipient.
+function getInboxUnreadCount(recipientSessionId) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+  const row = db.prepare('SELECT COUNT(*) AS count FROM inbox_entries WHERE recipient_session_id = ? AND unread = 1')
+    .get(recipientSessionId);
+  return row ? row.count : 0;
+}
+
+// Read all inbox entries for a recipient session.
+function getInboxForSession(recipientSessionId, { limit = null, offset = 0 } = {}) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+  let query = 'SELECT * FROM inbox_entries WHERE recipient_session_id = ? ORDER BY sent_at DESC';
+  const params = [recipientSessionId];
+  if (limit) {
+    query += ' LIMIT ? OFFSET ?';
+    params.push(limit, offset);
+  }
+  return db.prepare(query).all(...params);
+}
+
+// Read a single inbox entry by ID.
+function getInboxEntryById(id) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+  return db.prepare('SELECT * FROM inbox_entries WHERE id = ?').get(id);
+}
+
+// Create a new inbox entry. Returns the entry object.
+function createInboxEntry({
+  recipientSessionId,
+  senderId,
+  senderName,
+  senderType,
+  interactionId,
+  summary,
+  content,
+  sequence = 0,
+}) {
+  if (!recipientSessionId || !senderId || !senderName || !senderType || !interactionId || !summary || !content) {
+    throw new Error('inbox-store: missing required fields for createInboxEntry');
+  }
+  if (!['agent', 'system', 'operator'].includes(senderType)) {
+    throw new Error(`inbox-store: invalid senderType "${senderType}"`);
+  }
+
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  const MAX_ID_COLLISION_RETRIES = 5;
+
+  return withTransaction(db, () => {
+    const now = Date.now();
+    const insert = db.prepare(`
+      INSERT INTO inbox_entries
+        (id, recipient_session_id, sender_id, sender_name, sender_type, interaction_id,
+         sequence, summary, content, unread, sent_at, read_at, notified_at)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 1, ?, NULL, NULL)
+    `);
+
+    // _generateInboxId() is collision-proof in practice (monotonic
+    // per-process counter + random suffix), but retry-on-conflict is kept as
+    // defense-in-depth against an id collision from any other source (e.g. a
+    // pre-existing row imported/replayed from another process) rather than
+    // letting a rare UNIQUE-constraint failure crash the caller.
+    let id;
+    for (let attempt = 0; attempt < MAX_ID_COLLISION_RETRIES; attempt++) {
+      id = _generateInboxId(recipientSessionId, senderId, sequence);
+      try {
+        insert.run(id, recipientSessionId, senderId, senderName, senderType, interactionId, sequence, summary, content, now);
+        break;
+      } catch (e) {
+        const isUniqueViolation = e && (e.code === 'ERR_SQLITE_ERROR' || e.errcode === 1555 || /UNIQUE constraint failed/.test(e.message || ''));
+        if (!isUniqueViolation || attempt === MAX_ID_COLLISION_RETRIES - 1) throw e;
+        // Regenerate and retry — the monotonic counter guarantees the next
+        // attempt produces a different id.
+      }
+    }
+
+    return getInboxEntryById(id);
+  });
+}
+
+// Mark one or more inbox entries as read.
+function markInboxEntriesAsRead(entryIds) {
+  if (!Array.isArray(entryIds) || !entryIds.length) return { wrote: false };
+
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    const now = Date.now();
+    const placeholders = entryIds.map(() => '?').join(',');
+    const query = `UPDATE inbox_entries SET unread = 0, read_at = ? WHERE id IN (${placeholders})`;
+    const stmt = db.prepare(query);
+    const result = stmt.run(now, ...entryIds);
+    return { wrote: result.changes > 0 };
+  });
+}
+
+// Mark one or more inbox entries as unread.
+function markInboxEntriesAsUnread(entryIds) {
+  if (!Array.isArray(entryIds) || !entryIds.length) return { wrote: false };
+
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    const placeholders = entryIds.map(() => '?').join(',');
+    const query = `UPDATE inbox_entries SET unread = 1, read_at = NULL WHERE id IN (${placeholders})`;
+    const stmt = db.prepare(query);
+    const result = stmt.run(...entryIds);
+    return { wrote: result.changes > 0 };
+  });
+}
+
+// Mark all inbox entries for a recipient as read.
+function markAllInboxAsRead(recipientSessionId) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    const now = Date.now();
+    const result = db.prepare(
+      'UPDATE inbox_entries SET unread = 0, read_at = ? WHERE recipient_session_id = ? AND unread = 1'
+    ).run(now, recipientSessionId);
+    return { wrote: result.changes > 0 };
+  });
+}
+
+// Delete an inbox entry by ID.
+function deleteInboxEntry(id) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    const result = db.prepare('DELETE FROM inbox_entries WHERE id = ?').run(id);
+    return { wrote: result.changes > 0 };
+  });
+}
+
+// Delete multiple inbox entries by ID.
+function deleteInboxEntries(entryIds) {
+  if (!Array.isArray(entryIds) || !entryIds.length) return { wrote: false };
+
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    const placeholders = entryIds.map(() => '?').join(',');
+    const query = `DELETE FROM inbox_entries WHERE id IN (${placeholders})`;
+    const result = db.prepare(query).run(...entryIds);
+    return { wrote: result.changes > 0 };
+  });
+}
+
+// Delete all unread entries older than a given timestamp (in milliseconds).
+// Used for cleanup/archival.
+function deleteInboxEntriesOlderThan(timestampMs, { unreadOnly = false } = {}) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    let query = 'DELETE FROM inbox_entries WHERE sent_at < ?';
+    const params = [timestampMs];
+    if (unreadOnly) {
+      query += ' AND unread = 1';
+    }
+    const result = db.prepare(query).run(...params);
+    return { wrote: result.changes > 0, deletedCount: result.changes };
+  });
+}
+
+// Retrieve inbox entries grouped by interaction_id.
+function getInboxByInteraction(interactionId) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+  return db.prepare('SELECT * FROM inbox_entries WHERE interaction_id = ? ORDER BY sent_at ASC')
+    .all(interactionId);
+}
+
+// Retrieve all unread entries for all recipients.
+function getAllUnreadInboxEntries() {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+  return db.prepare('SELECT * FROM inbox_entries WHERE unread = 1 ORDER BY sent_at DESC').all();
+}
+
+// Update the notified_at timestamp for an entry (marks when a notification was sent).
+function markInboxEntryNotified(id) {
+  let db;
+  try { db = getDb(); }
+  catch (e) { throw new Error(`inbox-store: SQLite unavailable (${e.message})`); }
+
+  return withTransaction(db, () => {
+    const now = Date.now();
+    const result = db.prepare('UPDATE inbox_entries SET notified_at = ? WHERE id = ?').run(now, id);
+    return { wrote: result.changes > 0 };
+  });
+}
+
+module.exports = {
+  // Read operations
+  getInboxUnreadCount,
+  getInboxForSession,
+  getInboxEntryById,
+  getInboxByInteraction,
+  getAllUnreadInboxEntries,
+
+  // Write operations
+  createInboxEntry,
+  markInboxEntriesAsRead,
+  markInboxEntriesAsUnread,
+  markAllInboxAsRead,
+  markInboxEntryNotified,
+  deleteInboxEntry,
+  deleteInboxEntries,
+  deleteInboxEntriesOlderThan,
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yemi33/minions",
-  "version": "0.1.2373",
+  "version": "0.1.2374",
   "description": "Multi-agent AI dev team that runs from ~/.minions/ — five autonomous agents share a single engine, dashboard, and knowledge base",
   "bin": {
     "minions": "bin/minions.js"