@swarmclawai/swarmclaw 1.3.4 → 1.3.6

package/README.md

@@ -204,6 +204,25 @@ Read the full setup guide in [`SWARMDOCK.md`](./SWARMDOCK.md), browse the public
 
 ## Release Notes
 
+### v1.3.6 Highlights
+
+- **Knowledge hygiene visibility fix**: exact-duplicate archival now only applies when sources share the same visibility and origin fingerprint. Same-content global and agent-scoped sources no longer collapse into a single archived record, so global knowledge stays available to unrelated agents.
+- **Release gate hardening**: the default test matrix now includes the 1.3.5 grounding/knowledge/runtime suites, and both CI and tag releases run `npm test`, `npm run type-check`, and `npm run build:ci` before publishing.
+
+### v1.3.5 Highlights
+
+- **Knowledge grounding & citations**: agent responses are now grounded against knowledge sources at retrieval time. Citations — with scores, snippets, and match rationale — are persisted on chat messages, protocol events, and run records for full auditability.
+- **Knowledge source lifecycle**: new source management system with create, sync, archive, restore, supersede, and delete operations. Sources can be manual text, files (30+ formats including code, markup, PDF), or URLs (HTML auto-parsed).
+- **Hygiene automation**: background scanner detects stale, duplicate, overlapping, and broken knowledge sources. Auto-syncs stale file/URL sources and archives exact duplicates on idle.
+- **Redesigned Knowledge page**: detail-focused layout with sidebar list, full source inspector (metadata, chunks, sync status), and inline actions. Search/browse toggle, tag filtering, and archive visibility controls.
+- **Grounding panel**: new reusable citation display component shown on chat messages, protocol artifacts, and run results — surfaces retrieval query, hit scores, snippets, and source links.
+- **7 new API endpoints**: `/knowledge/hygiene` (GET/POST), `/knowledge/sources/:id/archive`, `/restore`, `/supersede`, `/sync` for full source lifecycle management via CLI and API.
+- **Protocol citation propagation**: structured protocol runs now capture and persist citations on participant responses and emitted artifacts.
+- **Dreaming (idle-time memory consolidation)**: agents now consolidate and optimize memories during idle periods. Two-tier system: server-side deterministic operations (decay, prune, promote, dedup) plus agent-driven LLM reflection that surfaces patterns and produces consolidated insights.
+- **Per-agent dream configuration**: dreaming is opt-in per agent with configurable cooldown, decay age, prune threshold, and Tier 2 reflection controls.
+- **Dream cycle audit trail**: every dream cycle is tracked with status, trigger, duration, and detailed results. Viewable in the memory UI and via CLI.
+- **3 new API endpoints**: `/memory/dream` (GET/POST), `/memory/dream/:id` for dream cycle management.
+
 ### v1.3.4 Highlights
 
 - **Bug fix — custom provider loading under Turbopack (#32)**: converted all CommonJS `require()` calls across the codebase to ES module imports, fixing "Unknown provider: custom-\<id\>" errors and other potential Turbopack compatibility issues. Affected modules: providers, provider health, subagent swarm, prompt builder, chat finalization, CLI utils, and OpenClaw connectors. Thanks to @psywolf85 for the initial fix.
@@ -235,82 +254,7 @@ Read the full setup guide in [`SWARMDOCK.md`](./SWARMDOCK.md), browse the public
 - **Extended approval workflows**: new `agent_create`, `budget_change`, and `delegation_enable` approval categories with configurable policies in settings. When enabled, agent creation returns a pending approval instead of creating the agent directly.
 - **Shared validation schemas**: Zod schemas in `src/lib/validation/schemas.ts` are now safe for client-side import (server-only DAG validation moved to `server-schemas.ts`), enabling form-level pre-validation.
 
-### v1.2.9 Highlights
-
-- **SwarmDock marketplace connector**: SwarmClaw agents can now register on SwarmDock, auto-bid on matching work, receive assignments as board tasks, and submit results back through the connector runtime.
-- **Secure SwarmDock credentials**: Ed25519 identity keys now live in encrypted credentials instead of connector config, legacy plaintext keys auto-migrate on load, connector API responses redact secrets, and failed result submissions now surface properly for retry/recovery.
-- **Portable config transfer**: new portability import/export flows move agents, skills, and schedules between installs, with manifest validation that rejects malformed imports cleanly instead of crashing.
-- **Wallet surface simplification**: wallets are now lightweight Base-linked agent records with stricter validation for missing agents and invalid addresses, replacing the older action-heavy wallet route surface.
-- **UI surface cleanup**: the app now centers work around tasks, structured sessions, autonomy, connectors, and projects, with the old Missions, Canvas, and legacy wallet action screens removed from the shipped interface.
-
-### v1.2.8 Highlights
-
-- **Linux/WSL compatibility**: subprocess spawning now uses `$SHELL` instead of hardcoded `/bin/zsh`, fixing `ENOENT` errors on Linux and WSL systems.
-- **nvm compatibility**: stripped `npm_config_prefix` from subprocess environment, fixing node PATH resolution for nvm users.
-- **Dev-mode daemon fix**: prevented duplicate daemon spawn failure when daemon runs in-process during development.
-- **Gateway sheet stability**: fixed infinite render loop when opening a gateway profile with a disconnected gateway.
-- **Auto-provision gateway on deploy**: "Deploy on this host" now automatically creates a gateway profile and credential, so agents can connect immediately without a manual save step.
-- **Credential cleanup on gateway delete**: deleting a gateway profile now cleans up its associated credential when no other gateway or agent references it.
-
-### v1.2.7 Highlights
-
-- **Tool primitives**: consolidated 50+ narrow tools into 6 action-based primitives — `execute`, `files`, `memory`, `platform`, `skills`, and `credential-env` — with skill teach files so agents learn usage patterns on demand.
-- **Type system decomposition**: split the monolithic 3500-line types file into 16 focused domain modules (`agent.ts`, `session.ts`, `message.ts`, `run.ts`, etc.) with full backward-compatible re-exports.
-- **Lightweight direct chat**: message classifier detects simple conversational turns and fast-paths them with minimal prompt assembly and reduced thinking budget.
-- **Prompt mode resolution**: root sessions receive full system prompts while delegated and lightweight turns get streamlined minimal prompts.
-- **Execute tool config UI**: inspector panel now has separate configuration sections for the execute tool (sandbox/host backend, network, timeout) and browser sandbox.
-- **Chat store pagination**: proper `messageStartIndex` tracking, improved queued-message deduplication, and active-turn transcript merging.
-- **Per-session WebSocket notifications**: message mutations now broadcast to session-specific topics for more targeted UI updates.
-- **Tool capability policy refactoring**: consolidated matching logic with extension ID canonicalization for reliable policy resolution.
-- **Validation schemas**: new `AgentExecuteConfigSchema` for Zod-based execute config validation.
-- **Bug fix — custom provider resolution**: fixed "Unknown provider" error when using custom providers in chat execution.
-- **Lint baseline maintained** at 364 violations (no regressions).
-
-### v1.2.5 Highlights
-
-- **Working memory hierarchy**: agents maintain structured working state (facts, plans, decisions, blockers, evidence) that persists across turns and survives context compaction.
-- **Execution brief**: each chat turn receives a concise briefing document synthesized from working state for faster, more focused agent reasoning.
-- **RunContext**: persistent structured context tracks session lineage, delegation chain, and accumulated working memory across run lifecycle.
-- **Unified message flow**: consolidated message handling and execution routing into a single coherent pipeline.
-- **Delegation advisory**: new advisory layer for structured capability analysis during delegation decisions.
-- **Real-time session sync**: session create, update, and delete events now push over WebSocket, replacing poll-only refresh for the chat list.
-- **HMR resilience**: module-level state in the WebSocket client, fallback polling, and API request dedup now survives Next.js hot-module reloads.
-- **Type safety sweep**: eliminated 16 `any` types across 7 API routes with proper narrowing and error guards.
-- **Bug fix — setup wizard crash**: replaced client-side `crypto.randomUUID()` with browser-safe alternative, fixing fresh-install failures with Ollama and other providers.
-- **Bug fix — custom provider validation**: connection-test endpoint now recognizes custom providers from storage instead of rejecting them as "Unsupported provider."
-- **Bug fix — session cwd normalization**: `updateChatSession` now expands `~` paths consistently with session creation.
-
-### v1.2.4 Highlights
-
-- **Custom providers in agent config**: agent setup and inline model switching now merge saved custom provider configs into the selectable provider list, so custom providers show up reliably even when the built-in provider feed is stale or incomplete.
-- **Custom provider save-only flow**: the Providers screen no longer forces connection tests or live model discovery for custom providers; operators can save the endpoint, linked key, and manual model list directly.
-- **Custom provider runtime routing**: saved custom-provider model lists and linked credentials now flow through the agent UI and runtime resolution paths consistently, including legacy `provider_configs` records normalized on load.
-
-### v1.2.3 Highlights
-
-- **Standalone asset staging repair**: `swarmclaw server` now copies `.next/static` and `public/` into the Next.js standalone runtime after the first build, preventing blank UI loads and 503s for CSS, JS, and image assets.
-- **OpenClaw SSH port fix**: remote OpenClaw deploys now preserve well-known SSH ports like `22` instead of clamping them to `1024`.
-- **OpenClaw image source fix**: generated remote deploy bundles and default upgrade actions now use the official `ghcr.io/openclaw/openclaw:latest` image instead of the missing Docker Hub shorthand.
-- **Standalone self-healing**: server startup now repairs older incomplete standalone bundles by staging missing runtime assets before launching `server.js`.
-
-### v1.2.2 Highlights
-
-- **Modular chat execution pipeline**: decomposed the monolithic chat-execution module into 6 focused stages (preflight, preparation, stream execution, partial persistence, finalization, types) for maintainability and testability.
-- **Repository pattern adoption**: extracted ~15 repository modules from `storage.ts`, giving each domain (agents, sessions, missions, credentials, tasks, etc.) its own data-access layer.
-- **Runtime state encapsulation**: moved process-local state (active sessions, dev servers) from storage into `runtime-state.ts` with proper HMR singleton usage.
-- **Streaming state improvements**: stable assistant render IDs, better live-row display logic, and smoother streaming phase transitions in the chat UI.
-- **8 new skills**: coding-agent, github, nano-banana-pro, nano-pdf, openai-image-gen, resourceful-problem-solving, skill-creator, summarize.
-- **Lint baseline improvements**: reduced lint violations from 414 to 396 (-18).
-
-### v1.2.1 Highlights
-
-- **System health endpoint**: new `/api/system/status` route returns lightweight health summary for external monitoring and uptime checks.
-- **Memory abstracts**: ~100-token LLM summaries attached to memories for efficient proactive recall without loading full content.
-- **Structured logging**: migrated 40+ files from `console.*` to the `log` module for consistent, level-aware logging across the codebase.
-- **Lint baseline improvements**: reduced lint violations from 440 to 414 (-26) through targeted fixes across server and UI code.
-- **Daemon housekeeping**: pruning for subagent processes, orchestrator state, connector sessions, and usage records to prevent resource leaks.
-- **SKILL.md v2.0.0**: comprehensive CLI documentation covering 40+ command groups with examples and usage patterns.
-- **New dev scripts**: added `type-check`, `test`, and `format` scripts to `package.json` for streamlined development workflows.
+*For older release notes (v1.2.x and earlier), see [swarmclaw.ai/docs/release-notes](https://swarmclaw.ai/docs/release-notes).*
 
 
 ## What SwarmClaw Focuses On

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@swarmclawai/swarmclaw",
-  "version": "1.3.4",
+  "version": "1.3.6",
   "description": "Self-hosted AI runtime for OpenClaw, delegation, autonomy, runtime skills, crypto wallets, and chat platform connectors.",
   "license": "MIT",
   "publishConfig": {
@@ -62,7 +62,7 @@
     "benchmark:autonomy": "node ./scripts/benchmark-autonomy-harness.mjs",
     "benchmark:agent-regression": "node --import tsx ./scripts/run-agent-regression-suite.ts",
     "type-check": "tsc --noEmit",
-    "test": "npm run test:cli && npm run test:setup && npm run test:openclaw",
+    "test": "npm run test:cli && npm run test:setup && npm run test:openclaw && npm run test:runtime",
     "format": "eslint --fix",
     "lint": "eslint",
     "lint:fix": "eslint --fix",
@@ -72,6 +72,7 @@
     "test:cli": "node --test src/cli/*.test.js bin/*.test.js scripts/postinstall.test.mjs",
     "test:setup": "tsx --test src/app/api/setup/check-provider/route.test.ts src/lib/server/provider-model-discovery.test.ts src/components/auth/setup-wizard/utils.test.ts src/components/auth/setup-wizard/types.test.ts src/hooks/setup-done-detection.test.ts src/lib/setup-defaults.test.ts",
     "test:openclaw": "tsx --test src/lib/openclaw/openclaw-agent-id.test.ts src/lib/openclaw/openclaw-endpoint.test.ts src/lib/server/agents/agent-runtime-config.test.ts src/lib/server/build-llm.test.ts src/lib/server/connectors/connector-routing.test.ts src/lib/server/connectors/openclaw.test.ts src/lib/server/gateway/protocol.test.ts src/lib/server/llm-response-cache.test.ts src/lib/server/mcp-conformance.test.ts src/lib/server/openclaw/agent-resolver.test.ts src/lib/server/openclaw/deploy.test.ts src/lib/server/openclaw/skills-normalize.test.ts src/lib/server/session-tools/openclaw-nodes.test.ts src/lib/server/tasks/task-quality-gate.test.ts src/lib/server/tasks/task-validation.test.ts src/lib/server/tool-capability-policy.test.ts src/lib/providers/openclaw-exports.test.ts src/app/api/openclaw/dashboard-url/route.test.ts",
+    "test:runtime": "tsx --test src/lib/server/knowledge-sources.test.ts src/lib/server/chat-execution/chat-execution-grounding.test.ts src/lib/server/protocols/protocol-service.test.ts src/lib/server/runtime/run-ledger.test.ts",
     "test:e2e": "tsx .workbench/browser-e2e/run.ts",
     "test:mcp:conformance": "node --import tsx ./scripts/mcp-conformance-check.ts",
     "prepack": "npm run build:ci",

package/skills/swarmclaw.md

@@ -73,6 +73,23 @@ Agents have persistent memory across sessions:
 - Memories are automatically surfaced in context when relevant
 - Store important learnings proactively -- don't wait to be asked
 
+### Dreaming
+
+Agents with dreaming enabled automatically consolidate memories during idle periods. You can also trigger a dream manually:
+
+#### Check dream status
+```json
+{ "tool": "memory", "action": "list", "category": "dream_reflection" }
+```
+
+#### Manual dream trigger
+Use the platform API to trigger a dream cycle:
+```json
+{ "tool": "execute", "command": "curl -s -X POST http://localhost:3456/api/memory/dream -H 'Content-Type: application/json' -d '{\"agentId\":\"YOUR_AGENT_ID\"}'" }
+```
+
+Dream cycles produce `dream_reflection` and `consolidated_insight` memories that help maintain a clean, coherent memory store over time.
+
 ### Delegation
 
 Agents can delegate work to other agents:

package/src/app/api/agents/[id]/dream/route.ts

@@ -0,0 +1,45 @@
+import { NextResponse } from 'next/server'
+import { notFound } from '@/lib/server/collection-helpers'
+import { safeParseBody } from '@/lib/server/safe-parse-body'
+import { log } from '@/lib/server/logger'
+
+const TAG = 'api-agent-dream'
+
+export async function GET(_req: Request, { params }: { params: Promise<{ id: string }> }) {
+  try {
+    const { id } = await params
+    const { loadAgents } = await import('@/lib/server/storage')
+    const agents = loadAgents()
+    const agent = agents[id]
+    if (!agent) return notFound()
+    const { resolveDreamConfig } = await import('@/lib/server/memory/dream-service')
+    const { listDreamCycles } = await import('@/lib/server/memory/dream-cycles')
+    return NextResponse.json({
+      ok: true,
+      config: resolveDreamConfig(agent),
+      recentCycles: listDreamCycles(id, 10),
+    })
+  } catch (err: unknown) {
+    log.error(TAG, 'GET failed:', err)
+    return NextResponse.json({ ok: false, error: String((err as Error)?.message || err) }, { status: 500 })
+  }
+}
+
+export async function PATCH(req: Request, { params }: { params: Promise<{ id: string }> }) {
+  try {
+    const { id } = await params
+    const { data: body, error } = await safeParseBody(req)
+    if (error) return error
+    const { updateAgent } = await import('@/lib/server/agents/agent-service')
+    const patch: Record<string, unknown> = {}
+    if ('dreamEnabled' in body) patch.dreamEnabled = body.dreamEnabled
+    if ('dreamConfig' in body) patch.dreamConfig = body.dreamConfig
+    const updated = updateAgent(id, patch)
+    if (!updated) return notFound()
+    const { resolveDreamConfig } = await import('@/lib/server/memory/dream-service')
+    return NextResponse.json({ ok: true, config: resolveDreamConfig(updated) })
+  } catch (err: unknown) {
+    log.error(TAG, 'PATCH failed:', err)
+    return NextResponse.json({ ok: false, error: String((err as Error)?.message || err) }, { status: 500 })
+  }
+}

package/src/app/api/knowledge/[id]/route.ts

@@ -1,73 +1,72 @@
 import { NextResponse } from 'next/server'
 import { notFound } from '@/lib/server/collection-helpers'
-import { getMemoryDb } from '@/lib/server/memory/memory-db'
+import {
+  deleteKnowledgeSource,
+  getKnowledgeSourceDetail,
+  updateKnowledgeSource,
+} from '@/lib/server/knowledge-sources'
+import type { KnowledgeSourceKind } from '@/types'
+
+function inferKind(body: Record<string, unknown>): KnowledgeSourceKind | undefined {
+  if (body.kind === 'file' || body.kind === 'url' || body.kind === 'manual') return body.kind
+  if (typeof body.sourcePath === 'string' && body.sourcePath.trim()) return 'file'
+  if (typeof body.sourceUrl === 'string' && body.sourceUrl.trim() && typeof body.content !== 'string') return 'url'
+  return undefined
+}
 
 export async function GET(_req: Request, { params }: { params: Promise<{ id: string }> }) {
   const { id } = await params
-  const db = getMemoryDb()
-  const entry = db.get(id)
-  if (!entry || entry.category !== 'knowledge') {
-    return notFound()
-  }
-  return NextResponse.json(entry)
+  const detail = await getKnowledgeSourceDetail(id)
+  if (!detail) return notFound()
+  return NextResponse.json(detail)
 }
 
 export async function PUT(req: Request, { params }: { params: Promise<{ id: string }> }) {
   const { id } = await params
-  const db = getMemoryDb()
-  const existing = db.get(id)
-  if (!existing || existing.category !== 'knowledge') {
-    return notFound()
-  }
-
   const body = await req.json().catch(() => null)
   if (!body || typeof body !== 'object' || Array.isArray(body)) {
     return NextResponse.json({ error: 'Invalid JSON body.' }, { status: 400 })
   }
 
-  const { title, content, tags, scope, agentIds } = body as Record<string, unknown>
-
-  const updates: Record<string, unknown> = {}
-  if (typeof title === 'string' && title.trim()) {
-    updates.title = title.trim()
-  }
-  if (typeof content === 'string') {
-    updates.content = content
-  }
+  const payload = body as Record<string, unknown>
 
-  const existingMeta = (existing.metadata || {}) as Record<string, unknown>
-  const metaUpdates: Record<string, unknown> = { ...existingMeta }
+  try {
+    const detail = await updateKnowledgeSource(id, {
+      kind: inferKind(payload),
+      title: typeof payload.title === 'string' ? payload.title : undefined,
+      content: typeof payload.content === 'string' ? payload.content : undefined,
+      tags: Array.isArray(payload.tags) ? payload.tags.filter((tag): tag is string => typeof tag === 'string') : undefined,
+      scope: payload.scope === 'agent' ? 'agent' : payload.scope === 'global' ? 'global' : undefined,
+      agentIds: Array.isArray(payload.agentIds) ? payload.agentIds.filter((agentId): agentId is string => typeof agentId === 'string') : undefined,
+      sourceLabel: typeof payload.sourceLabel === 'string'
+        ? payload.sourceLabel
+        : typeof payload.source === 'string'
+          ? payload.source
+          : undefined,
+      sourceUrl: typeof payload.sourceUrl === 'string' ? payload.sourceUrl : undefined,
+      sourcePath: typeof payload.sourcePath === 'string'
+        ? payload.sourcePath
+        : typeof payload.filePath === 'string'
+          ? payload.filePath
+          : undefined,
+      metadata: payload.metadata && typeof payload.metadata === 'object' && !Array.isArray(payload.metadata)
+        ? payload.metadata as Record<string, unknown>
+        : undefined,
+    })
 
-  if (Array.isArray(tags)) {
-    const normalizedTags = (tags as unknown[]).filter(
-      (t): t is string => typeof t === 'string' && t.trim().length > 0,
+    if (!detail) return notFound()
+    return NextResponse.json(detail)
+  } catch (error) {
+    return NextResponse.json(
+      { error: error instanceof Error ? error.message : 'Failed to update knowledge source.' },
+      { status: 400 },
     )
-    metaUpdates.tags = normalizedTags
-  }
-
-  if (scope === 'global' || scope === 'agent') {
-    metaUpdates.scope = scope
-    metaUpdates.agentIds = scope === 'agent' && Array.isArray(agentIds)
-      ? (agentIds as unknown[]).filter((id): id is string => typeof id === 'string')
-      : []
   }
-
-  updates.metadata = metaUpdates
-
-  const updated = db.update(id, updates)
-  if (!updated) {
-    return notFound()
-  }
-  return NextResponse.json(updated)
 }
 
 export async function DELETE(_req: Request, { params }: { params: Promise<{ id: string }> }) {
   const { id } = await params
-  const db = getMemoryDb()
-  const existing = db.get(id)
-  if (!existing || existing.category !== 'knowledge') {
-    return notFound()
-  }
-  db.delete(id)
+  const deleted = await deleteKnowledgeSource(id)
+  if (!deleted) return notFound()
   return NextResponse.json({ deleted: id })
 }

package/src/app/api/knowledge/hygiene/route.ts

@@ -0,0 +1,13 @@
+import { NextResponse } from 'next/server'
+import {
+  getKnowledgeHygieneSummary,
+  runKnowledgeHygieneMaintenance,
+} from '@/lib/server/knowledge-sources'
+
+export async function GET() {
+  return NextResponse.json(await getKnowledgeHygieneSummary())
+}
+
+export async function POST() {
+  return NextResponse.json(await runKnowledgeHygieneMaintenance())
+}

package/src/app/api/knowledge/route.ts

@@ -1,22 +1,49 @@
 import { NextResponse } from 'next/server'
-import { addKnowledge, searchKnowledge, listKnowledge } from '@/lib/server/memory/memory-db'
+import {
+  createKnowledgeSource,
+  listKnowledgeSourceSummaries,
+  searchKnowledgeHits,
+} from '@/lib/server/knowledge-sources'
+import type { KnowledgeSourceKind } from '@/types'
+
+function parseTags(raw: string | null): string[] | undefined {
+  if (!raw) return undefined
+  const tags = raw.split(',').map((tag) => tag.trim()).filter(Boolean)
+  return tags.length > 0 ? tags : undefined
+}
+
+function parseLimit(raw: string | null): number | undefined {
+  if (!raw) return undefined
+  return Math.max(1, Math.min(500, Number.parseInt(raw, 10) || 50))
+}
+
+function parseBool(raw: string | null): boolean {
+  if (!raw) return false
+  const value = raw.trim().toLowerCase()
+  return value === '1' || value === 'true' || value === 'yes'
+}
+
+function inferKind(body: Record<string, unknown>): KnowledgeSourceKind {
+  if (body.kind === 'file' || body.kind === 'url' || body.kind === 'manual') return body.kind
+  if (typeof body.sourcePath === 'string' && body.sourcePath.trim()) return 'file'
+  if (typeof body.sourceUrl === 'string' && body.sourceUrl.trim() && typeof body.content !== 'string') return 'url'
+  return 'manual'
+}
 
 export async function GET(req: Request) {
   const { searchParams } = new URL(req.url)
-  const q = searchParams.get('q')
-  const tagsParam = searchParams.get('tags')
-  const limitParam = searchParams.get('limit')
-
-  const tags = tagsParam ? tagsParam.split(',').map((t) => t.trim()).filter(Boolean) : undefined
-  const limit = limitParam ? Math.max(1, Math.min(500, Number.parseInt(limitParam, 10) || 50)) : undefined
+  const query = searchParams.get('q')
+  const tags = parseTags(searchParams.get('tags'))
+  const limit = parseLimit(searchParams.get('limit'))
+  const includeArchived = parseBool(searchParams.get('includeArchived'))
 
-  if (q) {
-    const results = searchKnowledge(q, tags, limit)
-    return NextResponse.json(results)
+  if (query && query.trim()) {
+    const hits = await searchKnowledgeHits({ query, tags, limit, includeArchived })
+    return NextResponse.json(hits)
   }
 
-  const entries = listKnowledge(tags, limit)
-  return NextResponse.json(entries)
+  const sources = await listKnowledgeSourceSummaries({ tags, limit, includeArchived })
+  return NextResponse.json(sources)
 }
 
 export async function POST(req: Request) {
@@ -25,36 +52,37 @@ export async function POST(req: Request) {
     return NextResponse.json({ error: 'Invalid JSON body.' }, { status: 400 })
   }
 
-  const { title, content, tags, scope, agentIds, source, sourceUrl } = body as Record<string, unknown>
+  const payload = body as Record<string, unknown>
 
-  if (typeof title !== 'string' || !title.trim()) {
-    return NextResponse.json({ error: 'title is required.' }, { status: 400 })
-  }
-  if (typeof content !== 'string') {
-    return NextResponse.json({ error: 'content is required.' }, { status: 400 })
-  }
+  try {
+    const detail = await createKnowledgeSource({
+      kind: inferKind(payload),
+      title: typeof payload.title === 'string' ? payload.title : undefined,
+      content: typeof payload.content === 'string' ? payload.content : undefined,
+      tags: Array.isArray(payload.tags) ? payload.tags.filter((tag): tag is string => typeof tag === 'string') : undefined,
+      scope: payload.scope === 'agent' ? 'agent' : 'global',
+      agentIds: Array.isArray(payload.agentIds) ? payload.agentIds.filter((id): id is string => typeof id === 'string') : undefined,
+      sourceLabel: typeof payload.sourceLabel === 'string'
+        ? payload.sourceLabel
+        : typeof payload.source === 'string'
+          ? payload.source
+          : undefined,
+      sourceUrl: typeof payload.sourceUrl === 'string' ? payload.sourceUrl : undefined,
+      sourcePath: typeof payload.sourcePath === 'string'
+        ? payload.sourcePath
+        : typeof payload.filePath === 'string'
+          ? payload.filePath
+          : undefined,
+      metadata: payload.metadata && typeof payload.metadata === 'object' && !Array.isArray(payload.metadata)
+        ? payload.metadata as Record<string, unknown>
+        : undefined,
+    })
 
-  const normalizedTags = Array.isArray(tags)
-    ? (tags as unknown[]).filter((t): t is string => typeof t === 'string' && t.trim().length > 0)
-    : undefined
-
-  const normalizedScope = scope === 'agent' ? 'agent' as const : 'global' as const
-  const normalizedAgentIds = Array.isArray(agentIds)
-    ? (agentIds as unknown[]).filter((id): id is string => typeof id === 'string')
-    : []
-
-  const normalizedSource = typeof source === 'string' && source.trim() ? source.trim() : undefined
-  const normalizedSourceUrl = typeof sourceUrl === 'string' && sourceUrl.trim() ? sourceUrl.trim() : undefined
-
-  const entry = addKnowledge({
-    title: title.trim(),
-    content,
-    tags: normalizedTags,
-    scope: normalizedScope,
-    agentIds: normalizedAgentIds,
-    source: normalizedSource,
-    sourceUrl: normalizedSourceUrl,
-  })
-
-  return NextResponse.json(entry)
+    return NextResponse.json(detail)
+  } catch (error) {
+    return NextResponse.json(
+      { error: error instanceof Error ? error.message : 'Failed to create knowledge source.' },
+      { status: 400 },
+    )
+  }
 }

package/src/app/api/knowledge/sources/[id]/archive/route.ts

@@ -0,0 +1,15 @@
+import { NextResponse } from 'next/server'
+import { notFound } from '@/lib/server/collection-helpers'
+import { archiveKnowledgeSource } from '@/lib/server/knowledge-sources'
+
+export async function POST(req: Request, { params }: { params: Promise<{ id: string }> }) {
+  const { id } = await params
+  const body = await req.json().catch(() => ({}))
+  const detail = await archiveKnowledgeSource(id, {
+    reason: typeof body?.reason === 'string' ? body.reason : null,
+    duplicateOfSourceId: typeof body?.duplicateOfSourceId === 'string' ? body.duplicateOfSourceId : null,
+    supersededBySourceId: typeof body?.supersededBySourceId === 'string' ? body.supersededBySourceId : null,
+  })
+  if (!detail) return notFound()
+  return NextResponse.json(detail)
+}

package/src/app/api/knowledge/sources/[id]/restore/route.ts

@@ -0,0 +1,10 @@
+import { NextResponse } from 'next/server'
+import { notFound } from '@/lib/server/collection-helpers'
+import { restoreKnowledgeSource } from '@/lib/server/knowledge-sources'
+
+export async function POST(_req: Request, { params }: { params: Promise<{ id: string }> }) {
+  const { id } = await params
+  const detail = await restoreKnowledgeSource(id)
+  if (!detail) return notFound()
+  return NextResponse.json(detail)
+}

package/src/app/api/knowledge/sources/[id]/route.ts

@@ -0,0 +1 @@
+export { GET, PUT, DELETE } from '@/app/api/knowledge/[id]/route'

package/src/app/api/knowledge/sources/[id]/supersede/route.ts

@@ -0,0 +1,26 @@
+import { NextResponse } from 'next/server'
+import { notFound } from '@/lib/server/collection-helpers'
+import { supersedeKnowledgeSource } from '@/lib/server/knowledge-sources'
+
+export async function POST(req: Request, { params }: { params: Promise<{ id: string }> }) {
+  const { id } = await params
+  const body = await req.json().catch(() => null)
+  const supersededBySourceId = typeof body === 'object' && body && !Array.isArray(body) && typeof body.supersededBySourceId === 'string'
+    ? body.supersededBySourceId
+    : ''
+
+  if (!supersededBySourceId.trim()) {
+    return NextResponse.json({ error: 'supersededBySourceId is required.' }, { status: 400 })
+  }
+
+  try {
+    const detail = await supersedeKnowledgeSource(id, supersededBySourceId)
+    if (!detail) return notFound()
+    return NextResponse.json(detail)
+  } catch (error) {
+    return NextResponse.json(
+      { error: error instanceof Error ? error.message : 'Failed to supersede knowledge source.' },
+      { status: 400 },
+    )
+  }
+}

package/src/app/api/knowledge/sources/[id]/sync/route.ts

@@ -0,0 +1,17 @@
+import { NextResponse } from 'next/server'
+import { notFound } from '@/lib/server/collection-helpers'
+import { syncKnowledgeSource } from '@/lib/server/knowledge-sources'
+
+export async function POST(_req: Request, { params }: { params: Promise<{ id: string }> }) {
+  const { id } = await params
+  try {
+    const detail = await syncKnowledgeSource(id)
+    if (!detail) return notFound()
+    return NextResponse.json(detail)
+  } catch (error) {
+    return NextResponse.json(
+      { error: error instanceof Error ? error.message : 'Failed to sync knowledge source.' },
+      { status: 400 },
+    )
+  }
+}

package/src/app/api/knowledge/sources/route.ts

@@ -0,0 +1 @@
+export { GET, POST } from '@/app/api/knowledge/route'

package/src/app/api/knowledge/upload/route.ts

@@ -3,28 +3,7 @@ import fs from 'fs'
 import path from 'path'
 import { genId } from '@/lib/id'
 import { UPLOAD_DIR } from '@/lib/server/storage'
-
-const TEXT_EXTS = new Set([
-  '.txt', '.md', '.markdown', '.csv', '.tsv', '.json', '.jsonl',
-  '.html', '.htm', '.xml', '.yaml', '.yml', '.toml', '.ini', '.cfg',
-  '.js', '', '.tsx', '.jsx', '.py', '.go', '.rs', '.java', '.c', '.cpp', '.h',
-  '.rb', '.php', '.sh', '.bash', '.zsh', '.sql', '.r', '.swift', '.kt',
-  '.env', '.log', '.conf', '.properties', '.gitignore', '.dockerignore',
-])
-
-function isTextFile(filename: string): boolean {
-  const ext = path.extname(filename).toLowerCase()
-  return TEXT_EXTS.has(ext) || ext === ''
-}
-
-function deriveTitle(filename: string): string {
-  const name = path.basename(filename, path.extname(filename))
-  return name
-    .replace(/[-_]+/g, ' ')
-    .replace(/([a-z])([A-Z])/g, '$1 $2')
-    .replace(/\b\w/g, (c) => c.toUpperCase())
-    .trim() || 'Uploaded Document'
-}
+import { deriveKnowledgeTitle, extractKnowledgeTextFromBuffer } from '@/lib/server/knowledge-import'
 
 export async function POST(req: Request) {
   const filename = req.headers.get('x-filename') || 'document.txt'
@@ -44,35 +23,8 @@ export async function POST(req: Request) {
   const filePath = path.join(UPLOAD_DIR, safeName)
   fs.writeFileSync(filePath, buf)
 
-  // Extract text content
-  let content = ''
-  const ext = path.extname(filename).toLowerCase()
-
-  if (ext === '.pdf') {
-    // Try dynamic import of pdf-parse if available
-    try {
-      // @ts-ignore — pdf-parse is an optional dependency
-      const pdfParse = (await import(/* webpackIgnore: true */ 'pdf-parse')).default
-      const result = await pdfParse(buf)
-      content = result.text || ''
-    } catch {
-      // pdf-parse not installed — read as raw text fallback
-      content = '[PDF document — install pdf-parse for text extraction]\n\nFile saved at: ' + filePath
-    }
-  } else if (isTextFile(filename)) {
-    content = buf.toString('utf-8')
-  } else {
-    // Binary file — can't extract text
-    content = `[Binary file: ${filename}]\n\nFile saved at: ${filePath}`
-  }
-
-  // Truncate very long content to prevent memory issues
-  const MAX_CONTENT = 500_000
-  if (content.length > MAX_CONTENT) {
-    content = content.slice(0, MAX_CONTENT) + '\n\n[... truncated at 500k characters]'
-  }
-
-  const title = deriveTitle(filename)
+  const content = await extractKnowledgeTextFromBuffer(buf, filename, filePath)
+  const title = deriveKnowledgeTitle(filename)
   const url = `/api/uploads/${safeName}`
 
   return NextResponse.json({