khotan-data 0.0.1 → 0.1.0

package/dist/templates/skill-plug.md

@@ -0,0 +1,193 @@
+---
+name: khotan-plug
+description: >
+  Create and configure khotan Plugs — HTTP clients for external APIs with
+  auth, retry, pagination, and typed endpoints. Use when connecting to a
+  new API, defining endpoint contracts, configuring authentication, or
+  creating a typed API client.
+---
+
+Create and configure khotan Plugs — HTTP clients for external APIs with auth, retry, pagination, and typed endpoints. Use when connecting to a new API, defining endpoint contracts, configuring authentication, or creating a typed API client.
+
+## Scaffold
+
+```bash
+npx khotan add plug --yes
+```
+
+Creates `{outputDir}/plugs/plug.ts` (the Plug runtime) and `plug.example.ts` (typed contract example).
+
+## Creating a Plug
+
+```typescript
+import { plug, bearer, apiKey, basic, custom } from "./plug";
+
+export const stripePlug = plug({
+  name: "stripe",
+  baseUrl: "https://api.stripe.com/v1",
+  auth: bearer(process.env.STRIPE_KEY!),
+  retry: { attempts: 3, backoff: 1000 },
+  timeout: 30000,
+});
+```
+
+## Auth Strategies
+
+| Function | Usage |
+|----------|-------|
+| `bearer(token)` | `Authorization: Bearer <token>` — static string or async function |
+| `basic(user, pass)` | `Authorization: Basic <base64>` |
+| `apiKey(name, value)` | Custom header (default) or query param with `{ in: "query" }` |
+| `custom(fn)` | Full control: `(headers) => { headers.set(...) }` |
+| `tokenExchange(config)` | OAuth-style: exchanges variables for bearer token, caches, auto-refreshes on 401 |
+
+### Token Exchange Example
+
+```typescript
+const auth = tokenExchange({
+  tokenUrl: "/oauth/token",
+  buildBody: (vars) => ({ grant_type: "client_credentials", client_id: vars.clientId, client_secret: vars.clientSecret }),
+  parseToken: (res) => ({ token: res.access_token, expiresIn: res.expires_in }),
+});
+```
+
+## Vars (Runtime Variables)
+
+For variables managed via the Hub UI instead of env vars:
+
+```typescript
+export const myPlug = plug({
+  baseUrl: "https://api.example.com",
+  auth: bearer(() => ""), // overridden by vars
+  vars: [
+    { key: "apiKey", label: "API Key", type: "text", secret: true },
+    { key: "orgId", label: "Org ID", type: "text", defaultValue: "org_demo" },
+    { key: "_token", label: "", type: "text", hidden: true },
+  ] as const,
+});
+```
+
+Vars are encrypted in the database via `KHOTAN_SECRET` and injected per-request. Hidden vars (prefixed `_`) are internal storage (cached tokens, etc). `defaultValue` seeds the database the first time the plug is initialized, and later Hub/CLI edits override that stored value.
+
+## Typed Endpoints
+
+Define Zod schemas inline on the plug:
+
+```typescript
+import { z } from "zod";
+
+export const myPlug = plug({
+  baseUrl: "https://api.example.com",
+  auth: bearer(process.env.API_KEY!),
+  endpoints: {
+    listProducts: {
+      method: "GET",
+      path: "/products",
+      query: z.object({ page: z.number().optional(), limit: z.number().optional() }),
+      responses: { 200: z.object({ data: z.array(z.object({ id: z.string(), name: z.string() })), total: z.number() }) },
+    },
+    createProduct: {
+      method: "POST",
+      path: "/products",
+      body: z.object({ name: z.string(), price: z.number() }),
+      responses: { 201: z.object({ id: z.string() }) },
+    },
+  },
+});
+```
+
+Endpoints power the plug debugger UI, `khotan plug --compare`, and typed clients.
+
+## Typed Client (Contract Pattern)
+
+For separate contract definition + type-safe calls:
+
+```typescript
+import { defineContract, createPlugClient } from "khotan-data/plug";
+
+const contract = defineContract({
+  listProducts: {
+    method: "GET",
+    path: "/products",
+    query: z.object({ page: z.number().optional() }),
+    responses: { 200: z.object({ data: z.array(ProductSchema), total: z.number() }) },
+  },
+});
+
+const client = createPlugClient(contract, myPlug);
+const result = await client.listProducts({ query: { page: 1 } });
+// result.status — 200
+// result.body.data — typed as Product[]
+```
+
+## Hooks
+
+```typescript
+const myPlug = plug({
+  // ...
+  hooks: {
+    beforeRequest: async (ctx) => {
+      // ctx.vars, ctx.setVars, ctx.headers, ctx.url
+    },
+    afterResponse: async (response, ctx) => {
+      // inspect/transform response
+    },
+    onUnauthorized: async (ctx) => {
+      // refresh token, update vars
+    },
+  },
+});
+```
+
+## Registering in Factory
+
+In `{outputDir}/khotan.ts`:
+
+```typescript
+plugs: [
+  {
+    name: "stripe",
+    plug: stripePlug,
+    flows: [
+      { name: "charges-inflow", type: "inflow", schedule: "0 * * * *", resource: "orders" },
+    ],
+    // Optional: wires, catches, passes
+  },
+],
+```
+
+## Making Requests
+
+```typescript
+// Direct
+const products = await myPlug.get("/products", { params: { limit: "10" } });
+const created = await myPlug.post("/products", { body: { name: "Widget" } });
+
+// With vars (factory injects these automatically in wire/debug contexts)
+const data = await myPlug.get("/items", { vars: { apiKey: "..." } });
+```
+
+## Debugging
+
+Use `khotan plug` to test plugs against the running dev server. `khotan probe` remains as a legacy alias:
+
+```bash
+npx khotan plug myPlug --info                    # See endpoints
+npx khotan plug myPlug GET /products             # Fire request
+npx khotan plug myPlug --endpoint listProducts --compare  # Check schema
+```
+
+Set `KHOTAN_DEBUG=1` for verbose `[khotan:auth]` and `[khotan:request]` console logs.
+
+## Managing Vars
+
+Use the CLI to inspect and update stored plug variables:
+
+```bash
+npx khotan plug vars --list
+npx khotan plug vars myPlug
+npx khotan plug vars myPlug set --json '{"apiKey":"secret","orgId":"org_live"}'
+npx khotan plug vars myPlug clear
+```
+
+Variable reads mask `secret` fields automatically. Hub and CLI both talk to the same `/api/khotan/variables/*` routes.

package/dist/templates/skill-setup.md

@@ -0,0 +1,119 @@
+---
+name: khotan-setup
+description: >
+  Set up khotan-data in a Next.js + Drizzle + Postgres project. Use when
+  initializing khotan in a new project, adding the database schema,
+  configuring the factory, or troubleshooting missing setup steps.
+---
+
+Set up khotan-data in a Next.js + Drizzle + Postgres project. Use when initializing khotan in a new project, adding the database schema, configuring the factory, or troubleshooting missing setup steps.
+
+## Quick Start
+
+```bash
+npm install khotan-data
+npx khotan init
+npx khotan add schema --yes
+npx khotan migrate
+npx khotan add plug --yes
+```
+
+## What Init Creates
+
+`npx khotan init` scaffolds three files (never overwrites existing):
+
+| File | Purpose |
+|------|---------|
+| `khotan.config.ts` | CLI config — sets `outputDir` (default: `src/khotan` or `khotan`) |
+| `{outputDir}/khotan.ts` | Factory config — register plugs, resources, adapter |
+| `src/app/api/khotan/[...all]/route.ts` | Catch-all API route |
+
+Use `npx khotan init --full` for greenfield projects — also installs drizzle-orm, postgres, drizzle-kit, and shadcn.
+
+## Factory Config Pattern
+
+Edit `{outputDir}/khotan.ts` after init:
+
+```typescript
+import { khotan, drizzleAdapter } from "khotan-data/factory";
+import { db } from "@/db";
+import { stripeChargesInflow } from "./flows/stripe-charges";
+
+const khotanData = khotan({
+  adapter: drizzleAdapter(db),
+  resources: [
+    { name: "products", connectField: "sku" },
+    { name: "orders", connectField: "order_number" },
+  ],
+  plugs: [
+    {
+      name: "stripe",
+      plug: stripePlug,
+      flows: [
+        stripeChargesInflow,
+      ],
+    },
+  ],
+});
+
+export default khotanData;
+```
+
+The factory auto-upserts plugs, flows, and resources to the database on first API request.
+
+## Route Handler
+
+The catch-all route delegates all HTTP methods to the factory:
+
+```typescript
+import { toNextJsHandler } from "khotan-data/factory";
+import khotanData from "@/lib/khotan/khotan";
+
+export const { GET, POST, PUT, PATCH, DELETE } = toNextJsHandler(khotanData.handler);
+```
+
+## Database Setup
+
+```bash
+npx khotan add schema --yes   # Scaffolds Drizzle table definitions
+npx khotan migrate             # Generates + applies migrations (needs DATABASE_URL)
+npx khotan migrate --push      # Or push directly without migration files
+```
+
+Tables created: `khotan_plugs`, `khotan_resources`, `khotan_flows`, `khotan_wires`, `khotan_runs`, `khotan_mappings`.
+
+The schema command auto-detects your Drizzle schema directory, updates `drizzle.config.ts` glob pattern, and adds the barrel re-export.
+
+## Environment Variables
+
+| Variable | Required | Purpose |
+|----------|----------|---------|
+| `DATABASE_URL` | Yes | Postgres connection (used by Drizzle) |
+| `KHOTAN_SECRET` | For variables | AES-256-GCM key for encrypting plug vars |
+| `KHOTAN_DEBUG` | For debugging | Enables `/debug/*` routes and the `plug` CLI (`probe` alias) |
+| `KHOTAN_WEBHOOK_URL` | For webhooks | Public URL for wire callbacks |
+
+## Next.js Config
+
+Add to `next.config.ts` if using local/tarball install:
+
+```typescript
+const nextConfig = {
+  serverExternalPackages: ["khotan-data"],
+};
+```
+
+## Verify Setup
+
+```bash
+curl http://localhost:3000/api/khotan/plugs     # Should list registered plugs
+curl http://localhost:3000/api/khotan/flows      # Should list flows
+curl http://localhost:3000/api/khotan/resources   # Should list resources
+```
+
+## Troubleshooting
+
+- **Empty plug list**: Factory upserts on first request — hit any endpoint first, then check `/plugs`
+- **"Cannot find module khotan-data"**: Add to `serverExternalPackages` in next.config.ts
+- **Migration fails**: Ensure `DATABASE_URL` is set and Postgres is reachable
+- **Init won't overwrite**: By design — delete the file manually if you need to re-scaffold

package/dist/templates/skill-webhook.md

@@ -0,0 +1,196 @@
+---
+name: khotan-webhook
+description: >
+  Set up webhook subscriptions and event processing with khotan Wires,
+  Catch, and Pass. Use when receiving webhooks from external services,
+  registering callback URLs, processing incoming events durably, or
+  forwarding events between services.
+---
+
+Set up webhook subscriptions and event processing with khotan Wires, Catch, and Pass. Use when receiving webhooks from external services, registering callback URLs, processing incoming events durably, or forwarding events between services.
+
+## Wire (Webhook Subscriptions)
+
+```bash
+npx khotan add wire --yes
+```
+
+Scaffolds `{outputDir}/wires/wire.ts` (the builder) and `src/components/khotan/wire.tsx` (UI panel).
+
+### Creating a Wire
+
+```typescript
+import { wire } from "./wire";
+
+export const stripeWire = wire({
+  events: ["invoice.paid", "charge.succeeded"],
+
+  async onSubscribe(ctx) {
+    const res = await ctx.plug.post<{ id: string; secret: string }>(
+      "/webhook_endpoints",
+      {
+        body: {
+          url: ctx.callbackUrl,
+          enabled_events: ctx.events,
+        },
+      },
+    );
+    await ctx.setWireVars({ webhookSecret: res.secret });
+    return { remoteId: res.id };
+  },
+
+  async onUnsubscribe(ctx) {
+    await ctx.plug.delete(`/webhook_endpoints/${ctx.remoteId}`);
+  },
+
+  async onVerify(ctx) {
+    const signature = ctx.headers["stripe-signature"];
+    // Verify HMAC using ctx.wireVars.webhookSecret and ctx.body (raw text)
+    return isValidSignature(signature, ctx.body, ctx.wireVars.webhookSecret);
+  },
+});
+```
+
+### Hook Contexts
+
+**onSubscribe** receives:
+- `ctx.plug` — Plug with vars/auth auto-injected (BoundPlug)
+- `ctx.callbackUrl` — The URL to register with the external service
+- `ctx.events` — Event types to subscribe to
+- `ctx.wireVars` / `ctx.setWireVars()` — Persist wire-specific data (secrets, tokens)
+- Must return `{ remoteId: string }`
+
+**onUnsubscribe** receives:
+- `ctx.plug` — BoundPlug
+- `ctx.remoteId` — The ID returned from onSubscribe
+- `ctx.wireVars` / `ctx.setWireVars()`
+
+**onVerify** receives:
+- `ctx.headers` — Incoming request headers
+- `ctx.body` — Raw request body (for signature verification)
+- `ctx.wireVars` — Wire-specific vars
+- Must return `boolean`
+
+### Registering Wires
+
+In `{outputDir}/khotan.ts`:
+
+```typescript
+import { stripeWire } from "./wires/stripe-wire";
+
+plugs: [
+  {
+    name: "stripe",
+    plug: stripePlug,
+    wires: [stripeWire],
+  },
+],
+```
+
+### Programmatic Wire API
+
+```typescript
+const w = khotanData.wire("stripe");
+await w.create("https://your-domain.com/api/khotan/webhook/stripe");
+await w.get();           // Get wire status
+await w.delete(wireId);  // Disconnect
+```
+
+### Webhook Callback URL
+
+Wires register callbacks at: `{webhookUrl}/api/khotan/webhook/{plugName}`
+
+For local dev, use ngrok or similar tunnel and set `KHOTAN_WEBHOOK_URL`.
+
+## Catch (Durable Event Processing)
+
+```bash
+npx khotan add catch --yes
+```
+
+Process webhook events durably via Vercel Workflow:
+
+```typescript
+import { catchEvent } from "./webhooks/catch";
+
+const processInvoice = catchEvent(async (ctx) => {
+  "use workflow";
+
+  async function persist() {
+    "use step";
+    // Write to database — retried on failure
+    await db.insert(invoices).values(ctx.event);
+  }
+
+  await persist();
+});
+```
+
+Register on the source plug:
+
+```typescript
+{ name: "stripe", plug: stripePlug, wires: [stripeWire], catches: [processInvoice] }
+```
+
+## Pass (Event Forwarding)
+
+```bash
+npx khotan add pass --yes
+```
+
+Forward webhook events to another service:
+
+```typescript
+import { pass } from "./webhooks/pass";
+
+const forwardToSlack = pass({
+  to: "slack",  // Destination plug name (must be registered)
+  workflow: async (ctx) => {
+    "use workflow";
+    // ctx.event — the incoming webhook payload
+    // ctx.destVars — destination plug variables
+    async function forward() {
+      "use step";
+      await ctx.destPlug.post("/messages", {
+        body: { text: `New event: ${ctx.event.type}` },
+      });
+    }
+    await forward();
+  },
+});
+```
+
+Register on the source plug:
+
+```typescript
+{ name: "stripe", plug: stripePlug, wires: [stripeWire], passes: [forwardToSlack] }
+```
+
+## Webhook Flow
+
+```
+External Service → POST /api/khotan/webhook/:plugName
+  → onVerify (signature check)
+  → Parse event type
+  → Start catch workflows (durable processing)
+  → Start pass workflows (event forwarding)
+  → Return { received: true }
+```
+
+## Dependencies
+
+- **Wire**: Requires `plug` and `schema` components
+- **Catch**: Requires `wire`; needs `workflow` package for Vercel Workflow
+- **Pass**: Requires `wire` and `plug`; needs `workflow` package
+
+## Hub Integration
+
+The WirePanel in the Hub UI lets users connect/disconnect webhooks from the browser. It calls `POST /api/khotan/wires/:plugName` with the callback URL.
+
+## Debugging Webhooks
+
+1. Check wire status: `GET /api/khotan/wires/:plugName`
+2. Verify `onVerify` logic: check wire vars contain the signing secret
+3. Check factory logs: `KHOTAN_DEBUG=1` enables `[khotan:wire]` log lines
+4. 401 on webhook receive = `onVerify` returning false
+5. 500 on webhook = missing `workflow` package or catch/pass misconfigured