@theokit/sdk 1.6.2 → 1.8.0

package/claude-template/dot-claude/skills/theokit-eval/SKILL.md

@@ -0,0 +1,144 @@
+---
+user-invocable: false
+paths:
+  - "**/*eval*"
+  - "**/*Eval*"
+  - "**/*scorer*"
+description: TheoKit SDK Eval suite API reference — Eval.create, scorers, datasets, EvalRun
+---
+
+# TheoKit Eval Suite
+
+Eval-as-code primitive for production deploy gates. Run evals against real LLM
+providers to measure quality, latency, and cost before shipping.
+
+## Quick start
+
+```typescript
+import { Eval, Scorers } from "@theokit/sdk";
+
+const run = await Eval.create({
+  name: "qa-smoke",
+  dataset: [
+    { input: "Reply with the word: ok.", expected: "ok" },
+    { input: "Say jazz in one word.", expected: "jazz" },
+  ],
+  scorers: [
+    Scorers.containsExpected({ caseSensitive: false }),
+    Scorers.regex(/[a-zA-Z]/),
+  ],
+  agent: {
+    apiKey: process.env.OPENROUTER_API_KEY,
+    model: { id: "openai/gpt-4o-mini" },
+    local: { cwd: process.cwd(), sandboxOptions: { enabled: false } },
+  },
+  concurrency: 4,
+}).run();
+
+console.log(run.aggregate.meanScore);     // 0.95
+console.log(run.aggregate.passRatio);     // 1.0
+console.log(run.aggregate.tokensInTotal); // 142
+console.log(run.aggregate.durationMsP95); // 1830
+```
+
+## Built-in scorers (`Scorers`)
+
+| Scorer | What it checks |
+|---|---|
+| `Scorers.exactMatch({ caseSensitive? })` | `output.trim() === expected.trim()` — refuses empty `expected` |
+| `Scorers.containsExpected({ caseSensitive? })` | `output.includes(expected)` — refuses empty `expected` |
+| `Scorers.regex(pattern)` | `pattern.test(output)` — test patterns against adversarial output to avoid ReDoS |
+| `Scorers.jsonShape(zodSchema, { strict? })` | `JSON.parse(output)` + Zod validation — caps output at 1 MB before parse |
+| `Scorers.llmJudge({ model, apiKey, criteria, rubric? })` | Second LLM scores against criteria — requires SEPARATE `apiKey` |
+
+### Custom scorer
+
+A scorer is an async function returning a number between 0 and 1:
+
+```typescript
+const myScorer = async (row: { input: string; output: string; expected?: string }) => {
+  return row.output.length < 100 ? 1.0 : 0.5;
+};
+```
+
+## Dataset
+
+The `dataset` field accepts an array of objects with `input` and optional `expected`:
+
+```typescript
+interface EvalDatasetRow {
+  input: string;
+  expected?: string;
+}
+```
+
+Recommended ceiling: ~10k rows (v1 materializes in memory). For larger evals,
+partition into multiple `Eval.create` calls.
+
+## `EvalRun` shape
+
+```typescript
+interface EvalRun {
+  id: string;
+  name: string;
+  startedAt: number;
+  endedAt: number;
+  durationMs: number;
+  aggregate: EvalAggregate;
+  rows: ReadonlyArray<EvalRowResult>;
+  metadata?: Record<string, unknown>;
+}
+
+interface EvalAggregate {
+  meanScore: number;
+  medianScore: number;
+  passRatio: number;          // rows where meanScore >= 0.5
+  perScorer: Record<string, { mean; median; min; max }>;
+  totalRows: number;
+  errorRows: number;
+  durationMsP50: number;
+  durationMsP95: number;
+  tokensInTotal: number;
+  tokensOutTotal: number;
+}
+```
+
+`EvalRun` is plain JSON — `JSON.stringify(run)` works directly.
+
+## Concurrency
+
+`concurrency` defaults to 4. Allowed range: `[1, 64]` (integer). 0 and
+Infinity are rejected at `Eval.create` time.
+
+## Concurrent runs
+
+Per-process single-flight per `name`. Two `Eval.run` calls with the same
+`name` running simultaneously throw `EvalAlreadyRunningError`. Include model
+id in the name for matrix runs.
+
+## CLI integration
+
+The `theokit eval` CLI invokes `Eval.run` internally. User-authored
+`eval.config.{ts,mjs}` files are forward-compatible.
+
+## Telemetry
+
+When `agent.telemetry.enabled === true`, `Eval.run` emits a parent `eval.run`
+OTel span; `agent.send` / `llm.call` spans nest under it.
+
+## Cost forecasting
+
+```
+aggregate.tokensInTotal  x provider_input_price
++ aggregate.tokensOutTotal x provider_output_price
+```
+
+With `llmJudge`, add ~1 judge call per row. 1000 rows with `gpt-4o-mini`
+costs roughly $3.00 total (base + judge).
+
+## Errors
+
+| Error | When |
+|---|---|
+| `EvalAlreadyRunningError` | Same `name` already running in this process |
+| `ConfigurationError` | Invalid concurrency, missing required fields |

package/claude-template/dot-claude/skills/theokit-gateways/SKILL.md

@@ -0,0 +1,209 @@
+---
+user-invocable: false
+description: Gateway architecture and 10 platform adapters (Telegram, Slack, Discord, WhatsApp, Teams, Email, SMS, Mattermost, LINE, Matrix).
+paths:
+  - "**/*gateway*"
+  - "**/*Gateway*"
+  - "**/*telegram*"
+  - "**/*slack*"
+  - "**/*discord*"
+  - "**/*whatsapp*"
+---
+
+# TheoKit SDK -- Gateways
+
+Quick reference for the gateway architecture and all 10 platform adapters.
+
+## Architecture
+
+The gateway system uses a base adapter pattern. Each platform adapter extends `BasePlatformAdapter` from `@theokit/gateway`. The core package defines:
+
+- `BasePlatformAdapter` -- abstract class with `connect`, `disconnect`, `sendMessage`, `onInbound`.
+- `GatewayMessageEvent` -- discriminated union of all platform inbound events (keyed by `platform`).
+- `OutboundMessage` -- what `sendMessage` accepts.
+- `SendResult` -- `{ ok, messageId?, error? }`.
+- Session router for multi-platform agent dispatch.
+
+## BasePlatformAdapter
+
+```typescript
+abstract class BasePlatformAdapter {
+  abstract readonly platform: PlatformName;
+  abstract connect(): Promise<boolean>;
+  abstract disconnect(): Promise<void>;
+  abstract sendMessage(out: OutboundMessage): Promise<SendResult>;
+  abstract onInbound(handler: (event: GatewayMessageEvent) => Promise<void>): () => void;
+  async startTyping(channelId: string): Promise<void> { /* noop */ }
+  async stopTyping(channelId: string): Promise<void> { /* noop */ }
+}
+```
+
+## GatewayMessageEvent (common fields)
+
+```typescript
+interface BaseMessageEvent {
+  id: string;
+  platform: PlatformName;
+  sender: { id: string; username?: string; displayName?: string };
+  channel: { id: string; type: "dm" | "group" | "thread"; topicId?: string };
+  text: string;
+  receivedAt: number;
+  replyTo?: string;
+}
+```
+
+## PlatformName
+
+```typescript
+type PlatformName =
+  | "telegram" | "discord" | "slack" | "whatsapp" | "teams"
+  | "email" | "sms" | "mattermost" | "line" | "matrix";
+```
+
+## Platform adapters
+
+### 1. Telegram (`@theokit/gateway-telegram`)
+
+```bash
+pnpm add @theokit/gateway-telegram @theokit/gateway grammy
+```
+
+```typescript
+import { TelegramAdapter } from "@theokit/gateway-telegram";
+
+const adapter = new TelegramAdapter({ botToken: process.env.TELEGRAM_BOT_TOKEN! });
+await adapter.connect();
+```
+
+Platform-specific: `event.telegram.chatId`, `event.telegram.messageId`, `event.telegram.threadId?`.
+
+### 2. Discord (`@theokit/gateway-discord`)
+
+```bash
+pnpm add @theokit/gateway-discord @theokit/gateway discord.js
+```
+
+```typescript
+import { DiscordAdapter } from "@theokit/gateway-discord";
+
+const adapter = new DiscordAdapter({ botToken: process.env.DISCORD_BOT_TOKEN! });
+await adapter.connect();
+```
+
+Platform-specific: `event.discord.guildId`, `event.discord.channelId`, `event.discord.messageId`.
+
+### 3. Slack (`@theokit/gateway-slack`)
+
+```bash
+pnpm add @theokit/gateway-slack @theokit/gateway @slack/bolt @slack/web-api
+```
+
+```typescript
+import { SlackAdapter } from "@theokit/gateway-slack";
+
+const adapter = new SlackAdapter({
+  botToken: process.env.SLACK_BOT_TOKEN!,
+  appToken: process.env.SLACK_APP_TOKEN!,
+  requireMention: true,  // default; public channels need @bot mention
+});
+await adapter.connect();
+```
+
+Platform-specific: `event.slack.teamId`, `event.slack.channelId`, `event.slack.ts`, `event.slack.threadTs?`.
+
+### 4. WhatsApp (`@theokit/gateway-whatsapp`)
+
+```bash
+pnpm add @theokit/gateway-whatsapp @theokit/gateway
+```
+
+Two backends: `"cloud"` (Meta Cloud API) and `"web"` (whatsapp-web.js bridge).
+
+Platform-specific: `event.whatsapp.wamid`, `event.whatsapp.backend`, `event.whatsapp.phoneNumberId?`.
+
+### 5. Teams (`@theokit/gateway-teams`)
+
+```bash
+pnpm add @theokit/gateway-teams @theokit/gateway botbuilder
+```
+
+Platform-specific: `event.teams.activityId`, `event.teams.conversationId`, `event.teams.conversationType`, `event.teams.tenantId?`.
+
+### 6. Email (`@theokit/gateway-email`)
+
+```bash
+pnpm add @theokit/gateway-email @theokit/gateway
+```
+
+Uses IMAP for inbound and SMTP for outbound.
+
+Platform-specific: `event.email.messageId`, `event.email.subject`, `event.email.fromAddress`, `event.email.recipients`.
+
+### 7. SMS (`@theokit/gateway-sms`)
+
+```bash
+pnpm add @theokit/gateway-sms @theokit/gateway
+```
+
+Backends: `"twilio"`, `"plivo"`, `"vonage"`. Inbound via webhook server.
+
+Platform-specific: `event.sms.backend`, `event.sms.from` (E.164), `event.sms.to` (E.164).
+
+### 8. Mattermost (`@theokit/gateway-mattermost`)
+
+```bash
+pnpm add @theokit/gateway-mattermost @theokit/gateway
+```
+
+Platform-specific: `event.mattermost.postId`, `event.mattermost.channelId`, `event.mattermost.rootId?`.
+
+### 9. LINE (`@theokit/gateway-line`)
+
+```bash
+pnpm add @theokit/gateway-line @theokit/gateway
+```
+
+Inbound via webhook with signature verification. Reply tokens are cached (one-shot, 60s TTL).
+
+Platform-specific: `event.line.sourceType`, `event.line.sourceId`, `event.line.messageId`.
+
+### 10. Matrix (`@theokit/gateway-matrix`)
+
+```bash
+pnpm add @theokit/gateway-matrix @theokit/gateway matrix-js-sdk
+```
+
+Platform-specific: `event.matrix.roomId`, `event.matrix.eventId`, `event.matrix.memberCount`.
+
+## Common usage pattern
+
+```typescript
+import { Agent } from "@theokit/sdk";
+
+const adapter = new TelegramAdapter({ botToken: process.env.TELEGRAM_BOT_TOKEN! });
+await adapter.connect();
+
+adapter.onInbound(async (event) => {
+  const agent = await Agent.getOrCreate(`${event.platform}-${event.sender.id}`, {
+    apiKey: process.env.THEOKIT_API_KEY!,
+    model: { id: "google/gemini-2.0-flash-001" },
+    local: { cwd: process.cwd() },
+    memory: { enabled: true, namespace: "bot", scope: "user", userId: event.sender.id },
+  });
+
+  await adapter.startTyping(event.channel.id);
+  const run = await agent.send(event.text);
+  const result = await run.wait();
+  await adapter.stopTyping(event.channel.id);
+
+  await adapter.sendMessage({
+    channel: event.channel,
+    text: result.result ?? "No response.",
+    replyTo: event.id,
+  });
+});
+```
+
+## Error mapping convention
+
+Each adapter maps platform errors to canonical `SendResult.error.code` values: `rate_limit`, `channel_not_found`, `no_permission`, `auth_error`, `message_too_long`, `platform_error`.

package/claude-template/dot-claude/skills/theokit-memory/SKILL.md

@@ -0,0 +1,176 @@
+---
+user-invocable: false
+description: Memory API, embedding providers, dreaming, active recall, and backends for @theokit/sdk.
+paths:
+  - "**/*memory*"
+  - "**/*Memory*"
+  - "**/*embed*"
+---
+
+# TheoKit SDK -- Memory
+
+Quick reference for durable memory, embedding providers, dreaming, and backends.
+
+## Enabling memory on an agent
+
+```typescript
+import { Agent } from "@theokit/sdk";
+
+const agent = await Agent.create({
+  apiKey: process.env.THEOKIT_API_KEY!,
+  model: { id: "google/gemini-2.0-flash-001" },
+  local: { cwd: process.cwd() },
+  memory: {
+    enabled: true,
+    namespace: "my-app",
+    userId: "user-123",
+    scope: "user",
+  },
+});
+
+await (await agent.send("Remember: my preferred test runner is Vitest.")).wait();
+```
+
+## MemoryOptions
+
+```typescript
+interface MemoryOptions {
+  enabled: boolean;
+  namespace?: string;   // separates application domains
+  userId?: string;      // isolates user memories
+  scope?: "agent" | "user" | "team";
+  storePath?: string;   // relative to workspace; cannot escape
+}
+```
+
+## Scopes
+
+| Scope | Use for |
+|---|---|
+| `"agent"` | Durable state for one agent ID. Default scope. |
+| `"user"` | Stable user preferences across agent instances. Requires `userId`. |
+| `"team"` | Shared team facts safe for every authorized caller. |
+
+## Safety rules
+
+- Memory MUST NOT store API keys, bearer tokens, passwords, or credential material.
+- Local `storePath` is resolved relative to the workspace. Path traversal raises `ConfigurationError`.
+- Memory is durable by `{ namespace, userId, scope }`, not by JavaScript process.
+
+## SDKMemoryManager
+
+```typescript
+interface SDKMemoryManager {
+  // Reserved for explicit inspection and deletion APIs.
+}
+```
+
+The agent uses memory during runs automatically. Public management APIs are narrow until deletion and audit semantics are finalized.
+
+## Memory backends (v1.2+)
+
+```typescript
+import { Memory } from "@theokit/sdk";
+
+const memory = await Memory.create({
+  cwd: process.cwd(),
+  index: {
+    backend: "sqlite-vec",  // default
+    embedding: { provider: "openai", model: "text-embedding-3-small" },
+  },
+});
+
+// Or use LanceDB for >100k facts:
+const memory = await Memory.create({
+  cwd: process.cwd(),
+  index: {
+    backend: "lance",
+    embedding: { provider: "openai", model: "text-embedding-3-small" },
+  },
+});
+```
+
+- `@lancedb/lancedb` is an optional peer dep. If missing with `backend: "lance"`, throws `ConfigurationError(code: "lance_backend_unavailable")`.
+- Embedding dimension validated when opening existing index. Mismatch throws `ConfigurationError(code: "embedding_dimension_mismatch")`.
+
+## Embedding providers (ADR D11)
+
+Locked provider union: `openai`, `mistral`, `openrouter`, `voyage`, `deepinfra`.
+
+```typescript
+index: {
+  embedding: {
+    provider: "openai",
+    model: "text-embedding-3-small",
+  },
+}
+```
+
+Each provider adapter implements:
+
+```typescript
+interface EmbeddingAdapter {
+  id: string;
+  model: string;
+  dimension: number;
+  embed(texts: string[]): Promise<number[][]>;
+}
+```
+
+## Memory migration CLI (v1.2+)
+
+Migrate SQLite memory index to LanceDB without data loss:
+
+```bash
+# Dry-run (preview, no writes)
+pnpm exec theokit-migrate-memory --cwd . --dry-run
+
+# Real migration
+pnpm exec theokit-migrate-memory --cwd .
+```
+
+Options: `--cwd <path>`, `--dry-run`, `--keep-sqlite`, `--batch-size <n>`.
+
+Algorithm: read SQLite, write to staging `lance-new/`, validate count + sample text match, atomic rename, prompt to delete SQLite.
+
+## Dreaming (consolidation sweeps)
+
+```typescript
+await Memory.runDreamingSweep({
+  cwd: process.cwd(),
+  embedding: { provider: "openai", model: "text-embedding-3-small" },
+});
+```
+
+Dreaming consolidates redundant facts into compressed summaries. In v1.x, consolidation is deterministic only (no LLM-mediated narrative). LLM narrative mode is deferred.
+
+## Active recall
+
+Active recall queries memory during `agent.send` to inject relevant facts into the LLM context. Configured via the memory options on the agent. Query modes:
+
+- `"embedding"` -- cosine similarity search against the vector index.
+- `"keyword"` -- keyword-based search.
+- `"hybrid"` -- combines embedding and keyword results.
+
+## Semantic cache (related)
+
+`Cache.semantic` from `@theokit/sdk` reuses embedding adapters for LLM response caching with cosine-similarity matching. See the cache documentation for details.
+
+## Resume behavior
+
+Memory is durable by `{ namespace, userId, scope }`, not by process. Recreating or resuming an agent with the same memory config can recall durable facts. Inline secrets and MCP servers are NOT persisted through memory.
+
+## Conversation storage (pluggable persistence)
+
+```typescript
+import { Agent, InMemoryConversationStorage } from "@theokit/sdk";
+
+const agent = await Agent.create({
+  apiKey, model,
+  conversationStorage: new InMemoryConversationStorage(),
+});
+```
+
+Built-in adapters: `FileSystemConversationStorage`, `InMemoryConversationStorage`. Custom adapters implement `ConversationStorageAdapter`.
+
+When using custom storage, `Agent.resume` requires the adapter to be passed again -- silent FS fallback is rejected with `ConfigurationError(code: "conversation_storage_required")`.