khotan-data 0.0.1 → 0.1.1

package/AGENTS.md

@@ -0,0 +1,54 @@
+# khotan-data
+
+Production-grade data sync, ETL, and webhook components for Next.js + Drizzle + Postgres.
+
+## Agent Skills
+
+This package ships agent skills that teach coding agents how to use khotan-data.
+Skills are installed into consumer projects via the CLI:
+
+```bash
+npx khotan init --yes          # Installs all skills during setup
+npx khotan add skill-setup     # Or install individually
+```
+
+The installer auto-detects which coding agents are present (Cursor, Claude Code, Codex, Copilot, Kiro, Roo) and installs to all detected agent directories. It also places an `AGENTS.md` router at the project root.
+
+## Skill Templates
+
+Source templates live in `src/cli/templates/`:
+
+| Template | Skill name | Teaches |
+|----------|-----------|---------|
+| `skill-setup.md` | khotan-setup | Project initialization, factory config, database setup |
+| `skill-plug.md` | khotan-plug | Plug authoring, auth strategies, typed endpoints |
+| `skill-dashboard.md` | khotan-dashboard | Hub dashboard, plug debugger UI |
+| `skill-webhook.md` | khotan-webhook | Wires, Catch, Pass, webhook flow |
+| `agent-skill.md` | khotan-probe | Probe CLI for debugging plugs |
+
+## Agent Detection
+
+`src/cli/agent-detect.ts` handles multi-agent installation:
+
+| Agent | Marker directory | Skill path |
+|-------|-----------------|------------|
+| Cursor | `.cursor/` | `.cursor/skills/{name}/SKILL.md` |
+| Claude Code | `.claude/` | `.claude/skills/{name}/SKILL.md` |
+| Codex | `.agents/` | `.agents/skills/{name}/SKILL.md` |
+| Copilot | `.github/` | `.github/skills/{name}/SKILL.md` |
+| Kiro | `.kiro/` | `.kiro/skills/{name}/SKILL.md` |
+| Roo | `.roo/` | `.roo/rules/{name}/SKILL.md` |
+
+If no agent directories are detected, defaults to Cursor + Claude Code.
+
+## Package Architecture
+
+See `openspec/` for full specifications. Key source files:
+
+| File | Purpose |
+|------|---------|
+| `src/factory.ts` | Runtime core — plug registration, API routing, debug endpoints |
+| `src/cli/` | CLI commands (init, add, generate, migrate, probe) |
+| `src/cli/registry.ts` | Component/block registry |
+| `src/cli/agent-detect.ts` | Agent detection + multi-path skill installer |
+| `src/cli/compare.ts` | Schema inference + diff engine for probe --compare |

package/README.md

@@ -7,11 +7,127 @@ Built for **Next.js + Drizzle + Postgres** projects. Think better-auth for data
 ## Install
 
 ```bash
-npm install khotan-data
+npm i khotan-data
 ```
 
 Requires `drizzle-orm` as a peer dependency (you almost certainly already have it).
 
+## CLI
+
+Scaffold components into your Next.js + Drizzle project:
+
+```bash
+# Initialize khotan config
+npx khotan init
+
+# Full setup (drizzle + shadcn + config in one go)
+npx khotan init --full
+
+# Add components (reusable building blocks — never create pages)
+npx khotan add schema    # Drizzle table definitions (plugs, flows, runs, resources, mappings)
+npx khotan add cache     # Durable key/value caches for workflows and relays
+npx khotan add plug      # Fetch wrapper with auth, retry, pagination
+npx khotan add inflow    # Workflow-backed flow for pulling data in
+npx khotan add outflow   # Workflow-backed flow for pushing data out
+npx khotan add relay     # Workflow-backed flow for moving data between plugs
+npx khotan add hub       # Dashboard UI + API route + config (requires shadcn)
+
+# Add blocks (sample pages composed from components)
+npx khotan add config-page-1   # /config page that renders the KhotanHub dashboard
+
+# Options
+npx khotan add schema --force   # Overwrite existing files
+npx khotan add hub --yes        # Auto-accept dependency install prompts
+```
+
+## Factory (Runtime Engine)
+
+Register plugs, caches, flows, and resources — the factory upserts them on boot and serves a REST API:
+
+```typescript
+import { khotan, drizzleAdapter, toNextJsHandler } from "khotan-data/factory";
+import { db } from "@/db";
+import { shopifyPlug } from "@/lib/khotan/plugs/shopify";
+import { shopifyProductsInflow } from "@/lib/khotan/flows/shopify-products";
+import { shopifyProductsSnapshotCache } from "@/lib/khotan/caches/shopify-products-snapshot";
+
+const khotanData = khotan({
+  adapter: drizzleAdapter(db),
+  resources: [
+    { name: "products", mapping: { connectField: "sku" } },
+  ],
+  caches: [
+    shopifyProductsSnapshotCache,
+  ],
+  plugs: [
+    {
+      name: "shopify",
+      plug: shopifyPlug,
+      flows: [
+        shopifyProductsInflow,
+      ],
+    },
+  ],
+});
+
+// Next.js App Router: app/api/khotan/[...all]/route.ts
+export const { GET, POST, PUT, DELETE } = toNextJsHandler(khotanData.handler);
+
+// Start a flow through Khotan so run tracking + Workflow IDs are recorded
+await khotanData.flow("products-inflow", { plugName: "shopify" }).start({
+  runType: "delta",
+});
+```
+
+## Caches
+
+Use first-class caches when a flow, relay, catch, or pass needs durable state between runs.
+
+```typescript
+import { cache } from "@/lib/khotan/caches/cache";
+
+export const shopifyProductsSnapshotCache = cache({
+  name: "shopify-products-snapshot",
+  scope: {
+    plug: "shopify",
+    resource: "products",
+    flow: "shopify-products-inflow",
+  },
+  ttl: "6h",
+});
+```
+
+Inside workflows, use `khotanCache(ctx, "name")` for snapshots, cursors, and dedupe markers:
+
+```typescript
+import { khotanCache } from "khotan-data/factory";
+
+async function shopifyProductsWorkflow(ctx: InflowContext) {
+  "use workflow";
+
+  async function syncProducts() {
+    "use step";
+    const snapshotCache = khotanCache(ctx, "shopify-products-snapshot");
+    const previous =
+      (await snapshotCache.get<Array<Record<string, unknown>>>("latest")) ?? [];
+
+    const response = await shopifyPlug.get<{ data?: Array<Record<string, unknown>> }>("/products");
+    const records = Array.isArray(response.data) ? response.data : [];
+
+    await snapshotCache.set("latest", records);
+
+    return {
+      extracted: records.length,
+      transformed: records.length,
+      created: records.length,
+      metadata: { previousCount: previous.length },
+    };
+  }
+
+  return syncProducts();
+}
+```
+
 ## Quick Start
 
 ```typescript