@johpaz/hive-sdk 0.0.18 → 0.1.3

package/docs/HIVE-HARNESS.md

@@ -0,0 +1,113 @@
+# Hive Harness — durable task execution
+
+The `harness` module (`@johpaz/hive-sdk/harness`) is the SDK's durable-execution
+layer: a HiveDB-backed job queue with crash recovery, checkpointable runs,
+retry with backoff, idempotent submission, goal verification, and proof
+packets. It's what lets a host app (a `hive-app`, or a production service like
+Hive Cloud) survive a process restart mid-task without losing work or
+double-executing a tool call.
+
+It is deliberately **not** wired into `AgentRunner` automatically, and it has
+no built-in notion of "chat" vs "project task" vs any other app-specific job
+type — job `type` and run `kind` are plain strings. The host app defines its
+own vocabulary and registers executors for it. This is the same infrastructure
+that powers `hive`'s durable-queue harness, generalized so any SDK consumer
+can reuse it instead of re-implementing crash-safe job execution from scratch.
+
+## Architecture
+
+| Piece | File | Responsibility |
+|---|---|---|
+| `JobDoc` / `HarnessRunDoc` / `ProofPacketDoc` | `collections.ts` | HiveDB document shapes |
+| `db-helpers` | `db-helpers.ts` | `nextId`, `updateDoc`, `findByAny` — primitives HiveDB's `Collection` doesn't provide directly |
+| `job-store` | `job-store.ts` | Durable job persistence: claim/lease/complete/fail/retry, all via OCC |
+| `run-store` | `run-store.ts` | Checkpoint + lease for a single durable run (messages, iteration/token counters, pending tool calls) |
+| `durable-queue` | `durable-queue.ts` | `DurableLaneQueue` — FIFO+priority per lane, global concurrency cap, executor registry |
+| `goal-verifier` | `goal-verifier.ts` | `verifyGoal()` — deterministic check tool or LLM verifier, single goal or a list of acceptance criteria |
+| `run-epoch` | `run-epoch.ts` | Fixed-worker epoch fingerprint (provider/model/app-version/tool-catalog) |
+| `proof-packet` | `proof-packet.ts` | Compressed evidence artifact for a completed run |
+| `reconcile` | `reconcile.ts` | `reconcileOnBoot()` — crash repair + retention cap, call once at startup |
+
+## Durable queue semantics
+
+- **Lanes**: a lane (e.g. a session id, or `task:<id>`) runs at most one job
+  at a time, FIFO within the lane, ordered by `priority` then creation order.
+- **Global concurrency**: `maxGlobalConcurrency` caps how many jobs run at
+  once across all lanes (default 4). Types listed in `nonRetryableTypes`
+  (default `["chat_turn"]`) bypass this cap — a busy batch of background jobs
+  must not make an interactive/user-facing job type stop responding.
+- **Leases**: a claimed job gets a lease (default 30 min); the queue renews
+  it every 30s while executing. A lease that expires (crashed process) is
+  reclaimed to `pending` or marked `interrupted` once `attempts >=
+  max_attempts` — checked by `reconcileOnBoot()` at startup and by the
+  queue's periodic maintenance tick thereafter.
+- **Executors**: register one per job type with `registerExecutor(type, fn)`.
+  An executor receives the `JobDoc`, an `AbortSignal` (fired on cancel or
+  `taskTimeoutMs`), and any live callbacks passed to `enqueue()`.
+
+## Retry & backoff
+
+Two independent retry mechanisms:
+
+1. **Crash retries** (`attempts` / `max_attempts`) — bumped on every claim,
+   checked by `reclaimOrInterrupt` after a lease expires. This is about
+   *the process dying*, not the job failing logically.
+2. **Logical-failure retries** (`retry_count` / `JobRetryPolicy`) — when an
+   executor returns `{ok: false, retryable: true}` (the default unless set
+   `false`), `failJobOrRetry` reschedules the job with exponential backoff +
+   jitter instead of failing it immediately:
+
+   ```ts
+   delay = min(maxDelayMs, initialDelayMs * backoffMultiplier ** retryCount)
+         * (1 + jitter * random())
+   ```
+
+   Once `retryCount >= policy.maxRetries`, the job fails terminally. Types in
+   `nonRetryableTypes` never take this path — a failed interactive turn
+   should surface to the user immediately, not silently retry later.
+
+## Idempotency
+
+`createJob`/`enqueue` accept an optional `idempotency_key`. A repeated key
+returns the existing job (whatever its status — pending, running, completed,
+or terminally failed) instead of creating a duplicate, so a retried HTTP
+request from a caller doesn't double-enqueue work.
+
+## Goal verification & acceptance criteria
+
+`verifyGoal()` answers "was this met" for a single goal or — when
+`acceptance` criteria are supplied — for each criterion independently (its
+own optional `checkTool`, or an LLM judgment against its own description).
+The overall verdict is the conjunction of all criteria. The harness has no
+built-in tool registry: pass a `runCheckTool` callback that resolves a
+`checkTool` name to something the host app can actually execute.
+
+## Proof packets
+
+`buildProofPacket()` persists a compressed evidence artifact once a run
+finishes: intended outcome, per-criterion results, checks run, evidence
+snippets, known limits, and the run's fixed-worker epoch. Useful as an
+audit trail without having to replay the full run transcript.
+
+## Setup
+
+```ts
+import {
+  ensureHarnessIndexes,
+  reconcileOnBoot,
+  initDurableQueue,
+  registerExecutor,
+  getBootId,
+} from "@johpaz/hive-sdk/harness";
+
+await ensureHarnessIndexes();     // idempotent — safe every boot
+await reconcileOnBoot(getBootId());
+
+registerExecutor("my_job_type", async (job, signal) => {
+  // ... do the work, honoring `signal` for cancellation/timeout
+  return { ok: true, result: "done" };
+});
+
+const queue = initDurableQueue({ maxGlobalConcurrency: 4 });
+await queue.enqueue({ lane: "session-1", type: "my_job_type", run_id: "r1", payload: {} });
+```

package/package.json

@@ -1,7 +1,6 @@
 {
   "name": "@johpaz/hive-sdk",
-  "version": "0.0.18",
-  "private": false,
+  "version": "0.1.3",
   "description": "Hive SDK — The Agent Harness SDK. Build, deploy, and scale AI agent applications with multi-channel support, context engineering, and swarm orchestration.",
   "license": "MIT",
   "homepage": "https://github.com/johpaz/hive-sdk#readme",
@@ -28,6 +27,33 @@
   ],
   "main": "./packages/core/src/index.ts",
   "types": "./packages/core/src/index.ts",
+  "exports": {
+    ".": "./packages/core/src/index.ts",
+    "./agent": "./packages/core/src/agent/index.ts",
+    "./agent/providers": "./packages/core/src/agent/providers/index.ts",
+    "./agent/selectors": "./packages/core/src/agent/selectors/index.ts",
+    "./tools": "./packages/core/src/tools/index.ts",
+    "./skills": "./packages/core/src/skills/index.ts",
+    "./storage": "./packages/core/src/storage/index.ts",
+    "./swarm": "./packages/core/src/swarm/index.ts",
+    "./swarm/strategies": "./packages/core/src/swarm/strategies/index.ts",
+    "./swarm/presets": "./packages/core/src/swarm/presets/index.ts",
+    "./ace": "./packages/core/src/ace/index.ts",
+    "./ethics": "./packages/core/src/ethics/index.ts",
+    "./canvas": "./packages/core/src/canvas/index.ts",
+    "./config": "./packages/core/src/config/index.ts",
+    "./mcp": "./packages/core/src/mcp/index.ts",
+    "./mcp/transports": "./packages/core/src/mcp/transports/index.ts",
+    "./memory": "./packages/core/src/memory/index.ts",
+    "./multimodal": "./packages/core/src/multimodal/index.ts",
+    "./security": "./packages/core/src/security/index.ts",
+    "./state": "./packages/core/src/state/index.ts",
+    "./utils": "./packages/core/src/utils/index.ts",
+    "./gateway": "./packages/core/src/gateway/index.ts",
+    "./api": "./packages/core/src/api/index.ts",
+    "./harness": "./packages/core/src/harness/index.ts",
+    "./package.json": "./package.json"
+  },
   "bin": {
     "hives": "./packages/cli/bin/hives"
   },
@@ -42,15 +68,21 @@
     "prepublish": "echo 'No build needed - Bun runs TypeScript directly'"
   },
   "dependencies": {
+    "@johpaz/hive-db": "^0.4.0",
     "@anthropic-ai/sdk": "^0.74.0",
     "@google/genai": "^1.43.0",
     "@modelcontextprotocol/sdk": "latest",
     "@sapphire/snowflake": "latest",
+    "@slack/bolt": "latest",
+    "@whiskeysockets/baileys": "latest",
     "async-mutex": "^0.5.0",
     "cron-parser": "^5.5.0",
     "croner": "^10.0.1",
+    "discord.js": "latest",
     "docx": "^9.6.1",
+    "grammy": "latest",
     "groq-sdk": "^0.37.0",
+    "jsonwebtoken": "^9.0.3",
     "jszip": "^3.10.1",
     "pdf-lib": "^1.17.1",
     "mammoth": "^1.12.0",
@@ -58,12 +90,14 @@
     "openai": "^6.18.0",
     "pdfjs-dist": "^5.6.205",
     "pptxgenjs": "^4.0.1",
+    "qrcode-terminal": "latest",
     "toon-format-parser": "^1.1.0",
     "xlsx": "^0.18.5",
     "zod": "latest"
   },
   "devDependencies": {
     "@types/bun": "^1.3.13",
+    "@types/jsonwebtoken": "^9.0.10",
     "typescript": "6.0.2"
   }
 }

package/packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hive/cli",
-  "version": "0.0.15",
+  "version": "0.1.3",
   "description": "Hive CLI — init, run, test, trace",
   "license": "MIT",
   "type": "module",

package/packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hive/core",
-  "version": "0.0.15",
+  "version": "0.1.3",
   "description": "Hive Core — Agentes AI con Context Engineering, FTS5, ACE, Swarm",
   "license": "MIT",
   "type": "module",
@@ -29,18 +29,25 @@
     "./state": "./src/state/index.ts",
     "./utils": "./src/utils/index.ts",
     "./gateway": "./src/gateway/index.ts",
-    "./api": "./src/api/index.ts"
+    "./api": "./src/api/index.ts",
+    "./harness": "./src/harness/index.ts"
   },
   "dependencies": {
+    "@johpaz/hive-db": "^0.4.0",
     "@anthropic-ai/sdk": "^0.74.0",
     "@google/genai": "^1.43.0",
     "@modelcontextprotocol/sdk": "latest",
     "@sapphire/snowflake": "latest",
+    "@slack/bolt": "latest",
+    "@whiskeysockets/baileys": "latest",
     "async-mutex": "^0.5.0",
     "cron-parser": "^5.5.0",
     "croner": "^10.0.1",
+    "discord.js": "latest",
     "docx": "^9.6.1",
+    "grammy": "latest",
     "groq-sdk": "^0.37.0",
+    "jsonwebtoken": "^9.0.3",
     "jszip": "^3.10.1",
     "pdf-lib": "^1.17.1",
     "mammoth": "^1.12.0",
@@ -48,10 +55,14 @@
     "openai": "^6.18.0",
     "pdfjs-dist": "^5.6.205",
     "pptxgenjs": "^4.0.1",
+    "qrcode-terminal": "latest",
     "toon-format-parser": "^1.1.0",
     "xlsx": "^0.18.5",
     "zod": "latest"
   },
+  "devDependencies": {
+    "@types/jsonwebtoken": "^9.0.10"
+  },
   "peerDependencies": {
     "typescript": "^5.0.0"
   }

package/packages/core/src/ace/Tracer.ts

@@ -89,7 +89,7 @@ export function recordLLMUsage(opts: {
   Promise.resolve().then(async () => {
     try {
       const { recordUsage } = await import("../storage/usage.ts")
-      recordUsage({
+      await recordUsage({
         provider: opts.provider,
         model: opts.model,
         inputTokens: opts.inputTokens,

package/packages/core/src/agent/AgentRunner.ts

@@ -78,6 +78,16 @@ export interface AgentLoopOptions {
   signal?: AbortSignal
   /** Clean text for FTS5 and tracing (extracted from userMessage if multimodal) */
   rawUserMessage?: string
+  /**
+   * Per-call provider API key override, taking precedence over the DB-stored
+   * key resolved by `resolveProviderConfig`. Required for safe multi-tenant
+   * hosting (e.g. hive-cloud): without it, concurrent calls for different
+   * tenants using the same provider would race on `process.env[...]`, since
+   * that env var is process-global.
+   */
+  apiKey?: string
+  /** Per-call provider base URL override (self-hosted/proxy endpoints), same rationale as `apiKey`. */
+  baseUrl?: string
 }
 
 export interface StepEvent {
@@ -115,6 +125,8 @@ export async function* runAgent(
     agent.provider_id || "openai",
     agent.model_id || "gpt-4o-mini"
   )
+  if (opts.apiKey) providerCfg.apiKey = opts.apiKey
+  if (opts.baseUrl) providerCfg.baseUrl = opts.baseUrl
 
   const cleanModel = providerCfg.model.replace(new RegExp(`^${providerCfg.provider}\\/`), "")
   log.info(`[agent-loop] Starting: agent=${agentName} thread=${opts.threadId} provider=${providerCfg.provider}/${cleanModel}`)

package/packages/core/src/agent/ContextCompiler.ts

@@ -138,9 +138,9 @@ export async function compileContext(opts: {
 
   // [STEP-2] STRATEGY 1: WRITE — Load scratchpad (persistent notes)
   log.info(`[context-compiler] [STEP-2] Loading scratchpad...`)
-  let scratchpadNotes: ReturnType<typeof getScratchpad> = []
+  let scratchpadNotes: Awaited<ReturnType<typeof getScratchpad>> = []
   try {
-    scratchpadNotes = getScratchpad(threadId)
+    scratchpadNotes = await getScratchpad(threadId)
     log.info(`[context-compiler] [STEP-2] ✅ Loaded ${scratchpadNotes.length} scratchpad notes`)
   } catch (err) {
     log.error(`[context-compiler] [STEP-2] ❌ FAILED loading scratchpad: ${JSON.stringify(err)}`)
@@ -257,7 +257,7 @@ export async function compileContext(opts: {
 
   try {
     // Load minimal skills (always available)
-    minimalSkills = getMinimalSkills()
+    minimalSkills = await getMinimalSkills()
     log.info(`[context-compiler] [STEP-8b] ✅ Loaded ${minimalSkills.length} minimal skills`)
 
     // Discover additional skills via FTS5 (coordinator only)
@@ -268,7 +268,7 @@ export async function compileContext(opts: {
         : Array.isArray(inputForSkills)
           ? inputForSkills.filter(p => p.type === "text").map(p => (p as any).text).join("\n")
           : String(inputForSkills)
-      discoveredSkills = selectSkills(textMessage)
+      discoveredSkills = await selectSkills(textMessage)
       log.info(`[context-compiler] [STEP-8b] ✅ Discovered ${discoveredSkills.length} additional skills via FTS5`)
     }
   } catch (err) {

package/packages/core/src/agent/ConversationStore.ts

@@ -212,33 +212,43 @@ export function saveSummary(
   `).run(threadId, summary, messagesCovered, lastMessageId)
 }
 
+import { getHiveDB } from "../storage/HiveDBStorage.ts";
+
+interface ScratchpadDoc {
+  threadId: string;
+  key: string;
+  value: string;
+  updatedAt: number;
+}
+
+function scratchpadDocId(threadId: string, key: string): string {
+  return `${threadId}:${key}`;
+}
+
 // ─── Scratchpad ───────────────────────────────────────────────────────────────
 
-export function saveScratchpadNote(
+export async function saveScratchpadNote(
   threadId: string,
   key: string,
   value: string,
-  source?: string
-): void {
-  const db = getDb()
-  db.query(`
-    INSERT INTO scratchpad (thread_id, key, value, source)
-    VALUES (?, ?, ?, ?)
-    ON CONFLICT(thread_id, key) DO UPDATE SET
-      value      = excluded.value,
-      source     = excluded.source,
-      updated_at = unixepoch()
-  `).run(threadId, key, value, source ?? null)
+  _source?: string
+): Promise<void> {
+  const db = await getHiveDB();
+  const col = db.collection<ScratchpadDoc>("scratchpad");
+  await col.put(scratchpadDocId(threadId, key), { threadId, key, value, updatedAt: Date.now() });
 }
 
-export function getScratchpad(threadId: string): Array<{ key: string; value: string }> {
-  const db = getDb()
-  return db.query(
-    "SELECT key, value FROM scratchpad WHERE thread_id = ? ORDER BY updated_at DESC"
-  ).all(threadId) as Array<{ key: string; value: string }>
+export async function getScratchpad(threadId: string): Promise<Array<{ key: string; value: string }>> {
+  const db = await getHiveDB();
+  const col = db.collection<ScratchpadDoc>("scratchpad");
+  const entries = await col.scan();
+  return entries
+    .filter(e => e.doc.threadId === threadId)
+    .map(e => ({ key: e.doc.key, value: e.doc.value }));
 }
 
-export function deleteScratchpadNote(threadId: string, key: string): void {
-  const db = getDb()
-  db.query("DELETE FROM scratchpad WHERE thread_id = ? AND key = ?").run(threadId, key)
+export async function deleteScratchpadNote(threadId: string, key: string): Promise<void> {
+  const db = await getHiveDB();
+  const col = db.collection<ScratchpadDoc>("scratchpad");
+  await col.delete(scratchpadDocId(threadId, key));
 }

package/packages/core/src/agent/selectors/PlaybookSelector.ts

@@ -1,19 +1,20 @@
 /**
- * FTS5-based Playbook Rules Selector (ACE Curator)
- * 
- * This module allows the Context Compiler to inject relevant evolved rules
- * into the agent prompt based on semantic relevance to the current message.
+ * HiveDB-based Playbook Rules Selector (ACE Curator)
+ *
+ * Uses HiveDB hybrid search over the playbook index.
  */
 
-import { getDb } from "../../storage/SQLiteStorage.ts"
+import { getHiveDB } from "../../storage/HiveDBStorage.ts"
 import { logger } from "../../utils/logger.ts"
+import type { HivePlaybookDoc } from "../../storage/hiveSeed.ts"
+import type { IndexDoc } from "@johpaz/hive-db"
 
 const log = logger.child("playbook-selector")
 
 // ─── Types ───────────────────────────────────────────────────────────────────────
 
 export interface PlaybookRule {
-    id: number
+    id: string
     rule: string
     category: string
     applicable_to?: string
@@ -21,25 +22,27 @@ export interface PlaybookRule {
 
 // ─── Configuration ─────────────────────────────────────────────────────────────
 
-/** Maximum rules to inject per context window */
 const MAX_RULES_PER_TURN = 5
 
-/** Minimum bm25 score threshold for rules */
-const MIN_RELEVANCE_THRESHOLD = -10 // Relaxed for better matching
+const MIN_RELEVANCE_THRESHOLD = 0.5
 
 // ─── Selection Logic ───────────────────────────────────────────────────────────
 
-/**
- * Select relevant rules from the Playbook based on semantic matching
- */
-export function selectPlaybookRules(message: string): PlaybookRule[] {
-    const db = getDb()
+function toRule(id: string, doc: HivePlaybookDoc): PlaybookRule {
+    return {
+        id,
+        rule: doc.rule,
+        category: doc.category,
+        applicable_to: doc.applicableTo ? doc.applicableTo.join(",") : undefined,
+    }
+}
+
+export async function selectPlaybookRules(message: string): Promise<PlaybookRule[]> {
+    const db = await getHiveDB()
     const startTime = performance.now()
 
-    // Clean query — use prefix matching for consistency with skill-selector and tool-selector
     const keywords = message
         .toLowerCase()
-        // Keep only letters, numbers, spaces (strips ALL FTS5 special syntax)
         .replace(/[^\p{L}\p{N}\s]/gu, " ")
         .split(/\s+/)
         .filter(w => w.length > 3)
@@ -47,32 +50,29 @@ export function selectPlaybookRules(message: string): PlaybookRule[] {
 
     if (keywords.length === 0) return []
 
-    // Use prefix matching for better recall (e.g., "program*" matches "programar", "programación")
-    const ftsQuery = keywords.map(w => `${w}*`).join(" OR ")
+    const query = keywords.join(" ")
 
     try {
-        // Query FTS table
-        const ftsResults = db.query(`
-            SELECT rowid, bm25(playbook_fts) as score
-            FROM playbook_fts
-            WHERE playbook_fts MATCH ?
-            ORDER BY score ASC
-            LIMIT ?
-        `).all(ftsQuery, MAX_RULES_PER_TURN) as Array<{ rowid: number; score: number }>
-
-        const relevantIds = ftsResults
+        const hits = await db.queryHybrid({
+            text: query,
+            k: MAX_RULES_PER_TURN,
+            boosts: { body: 5.0, tags: 2.0, name: 1.0 },
+        })
+
+        const relevantIds = hits
             .filter(r => r.score >= MIN_RELEVANCE_THRESHOLD)
-            .map(r => r.rowid)
+            .map(r => r.id)
 
         if (relevantIds.length === 0) return []
 
-        // Fetch full rules
-        const rules = db.query(`
-            SELECT id, rule, category, applicable_to
-            FROM playbook
-            WHERE id IN (${relevantIds.map(() => '?').join(',')})
-            AND active = 1
-        `).all(...relevantIds) as PlaybookRule[]
+        const playbookCol = db.collection<HivePlaybookDoc>("playbook")
+        const rules: PlaybookRule[] = []
+        for (const id of relevantIds) {
+            const entry = await playbookCol.get(id)
+            if (entry && entry.doc.active) {
+                rules.push(toRule(id, entry.doc))
+            }
+        }
 
         const timing = performance.now() - startTime
         log.info(`[playbook-selector] Selected ${rules.length} rules in ${timing.toFixed(2)}ms`)
@@ -89,56 +89,30 @@ export function selectPlaybookRules(message: string): PlaybookRule[] {
 
 // ─── Sync Logic ───────────────────────────────────────────────────────────────
 
-/**
- * Sync active playbook rules to FTS5 virtual table
- */
 export async function syncPlaybookToFTS(): Promise<void> {
-    const db = getDb()
+    const db = await getHiveDB()
 
     try {
-        // Step 1: Get active rules
-        const rules = db.query(`
-            SELECT id, rule, category, applicable_to
-            FROM playbook
-            WHERE active = 1
-        `).all() as Array<{
-            id: number
-            rule: string
-            category: string
-            applicable_to: string
-        }>
+        const playbookCol = db.collection<HivePlaybookDoc>("playbook")
+        const entries = await playbookCol.scan()
+        const rules = entries.map(e => ({ id: e.id, doc: e.doc })).filter(r => r.doc.active)
 
         if (rules.length === 0) {
             log.debug(`[playbook-selector] No rules in playbook to sync`)
+            return
         }
 
-        // Step 2: Atomic transaction for FTS5 sync
-        const syncTransaction = db.transaction(() => {
-            // Verify table exists
-            const tableCheck = db.query("SELECT name FROM sqlite_master WHERE type='table' AND name='playbook_fts'").get()
-            if (!tableCheck) {
-                throw new Error("playbook_fts table does not exist!")
-            }
-
-            // A: Clear existing data
-            db.run("DELETE FROM playbook_fts")
-
-            // B: Prepare insertion
-            const insert = db.prepare(`
-                INSERT INTO playbook_fts(rowid, rule, category, applicable_to)
-                VALUES (?, ?, ?, ?)
-            `)
-
-            // C: Re-populate
-            for (const item of rules) {
-                insert.run(item.id, item.rule, item.category, item.applicable_to)
-            }
-        })
+        const docs: IndexDoc[] = rules.map(r => ({
+            id: r.id,
+            name: r.doc.category,
+            body: r.doc.rule,
+            tags: r.doc.applicableTo ? r.doc.applicableTo.join(" ") : "",
+            filters: [{ field: "type", value: "playbook" }],
+        }))
 
-        // Execute transaction
-        syncTransaction()
+        await db.upsertBatch(docs)
 
-        log.info(`[playbook-selector] Atomic sync complete: ${rules.length} rules indexed in FTS5`)
+        log.info(`[playbook-selector] Atomic sync complete: ${rules.length} rules indexed in HiveDB`)
 
     } catch (err) {
         log.error(`[playbook-selector] Transactional sync failed:`, err)