khotan-data 0.0.1 → 0.1.1

package/dist/templates/skill-plug.md

@@ -0,0 +1,216 @@
+---
+name: khotan-plug
+description: >
+  Create and configure khotan Plugs — HTTP clients for external APIs with
+  auth, retry, pagination, and typed endpoints. Use when connecting to a
+  new API, defining endpoint contracts, configuring authentication, or
+  creating a typed API client.
+---
+
+Create and configure khotan Plugs — HTTP clients for external APIs with auth, retry, pagination, and typed endpoints. Use when connecting to a new API, defining endpoint contracts, configuring authentication, or creating a typed API client.
+
+## Scaffold
+
+```bash
+npx khotan add plug --yes
+```
+
+Creates `{outputDir}/plugs/plug.ts` (the Plug runtime) and `plug.example.ts` (typed contract example).
+
+## Creating a Plug
+
+```typescript
+import { plug, bearer, apiKey, basic, custom } from "./plug";
+
+export const stripePlug = plug({
+  name: "stripe",
+  baseUrl: "https://api.stripe.com/v1",
+  auth: bearer(process.env.STRIPE_KEY!),
+  retry: { attempts: 3, backoff: 1000 },
+  timeout: 30000,
+});
+```
+
+## Auth Strategies
+
+| Function | Usage |
+|----------|-------|
+| `bearer(token)` | `Authorization: Bearer <token>` — static string or async function |
+| `basic(user, pass)` | `Authorization: Basic <base64>` |
+| `apiKey(name, value)` | Custom header (default) or query param with `{ in: "query" }` |
+| `custom(fn)` | Full control: `(headers) => { headers.set(...) }` |
+| `tokenExchange(config)` | OAuth-style: exchanges variables for bearer token, caches, auto-refreshes on 401 |
+
+### Token Exchange Example
+
+```typescript
+const auth = tokenExchange({
+  tokenUrl: "/oauth/token",
+  buildBody: (vars) => ({ grant_type: "client_credentials", client_id: vars.clientId, client_secret: vars.clientSecret }),
+  parseToken: (res) => ({ token: res.access_token, expiresIn: res.expires_in }),
+});
+```
+
+## Vars (Runtime Variables)
+
+For variables managed via the Hub UI instead of env vars:
+
+```typescript
+export const myPlug = plug({
+  baseUrl: "https://api.example.com",
+  auth: bearer(() => ""), // overridden by vars
+  vars: [
+    { key: "apiKey", label: "API Key", type: "text", secret: true },
+    { key: "orgId", label: "Org ID", type: "text", defaultValue: "org_demo" },
+    { key: "_token", label: "", type: "text", hidden: true },
+  ] as const,
+});
+```
+
+Vars are encrypted in the database via `KHOTAN_SECRET` and injected per-request. Hidden vars (prefixed `_`) are internal storage (cached tokens, etc). `defaultValue` seeds the database the first time the plug is initialized, and later Hub/CLI edits override that stored value.
+
+## Typed Endpoints
+
+Define Zod schemas inline on the plug:
+
+```typescript
+import { z } from "zod";
+
+export const myPlug = plug({
+  baseUrl: "https://api.example.com",
+  auth: bearer(process.env.API_KEY!),
+  endpoints: {
+    listProducts: {
+      method: "GET",
+      path: "/products",
+      query: z.object({ page: z.number().optional(), limit: z.number().optional() }),
+      responses: { 200: z.object({ data: z.array(z.object({ id: z.string(), name: z.string() })), total: z.number() }) },
+    },
+    createProduct: {
+      method: "POST",
+      path: "/products",
+      body: z.object({ name: z.string(), price: z.number() }),
+      responses: { 201: z.object({ id: z.string() }) },
+    },
+  },
+});
+```
+
+Endpoints power the plug debugger UI, `khotan plug --compare`, and typed clients.
+
+## Preferred Pattern
+
+Keep each integration in a single app-owned plug file when possible:
+
+```typescript
+import { z } from "zod";
+import { plug, basic } from "./plug";
+
+const ProductSchema = z.object({
+  id: z.string(),
+  sku: z.string(),
+  name: z.string(),
+});
+
+export type Product = z.infer<typeof ProductSchema>;
+
+export const myPlug = plug({
+  name: "my-service",
+  baseUrl: "https://api.example.com",
+  auth: basic(process.env.API_USER!, process.env.API_KEY!),
+  endpoints: {
+    listProducts: {
+      method: "GET",
+      path: "/products",
+      query: z.object({ page: z.number().optional(), limit: z.number().optional() }),
+      responses: { 200: z.array(ProductSchema) },
+    },
+  },
+});
+```
+
+This keeps the runtime plug, debugger metadata, `khotan plug --compare`, and any exported types in one place.
+
+## Hooks
+
+```typescript
+const myPlug = plug({
+  // ...
+  hooks: {
+    beforeRequest: async (ctx) => {
+      // ctx.vars, ctx.setVars, ctx.headers, ctx.url
+    },
+    afterResponse: async (response, ctx) => {
+      // inspect/transform response
+    },
+    onUnauthorized: async (ctx) => {
+      // refresh token, update vars
+    },
+  },
+});
+```
+
+## Registering in Factory
+
+In `{outputDir}/khotan.ts`:
+
+```typescript
+plugs: [
+  {
+    name: "stripe",
+    plug: stripePlug,
+    flows: [
+      { name: "charges-inflow", type: "inflow", schedule: "0 * * * *", resource: "orders" },
+    ],
+    // Optional: wires, catches, passes
+  },
+],
+```
+
+## Making Requests
+
+```typescript
+// Direct
+const products = await myPlug.get("/products", { params: { limit: "10" } });
+const created = await myPlug.post("/products", { body: { name: "Widget" } });
+
+// With vars (factory injects these automatically in wire/debug contexts)
+const data = await myPlug.get("/items", { vars: { apiKey: "..." } });
+```
+
+## Debugging
+
+Use `khotan plug` to test plugs against the running dev server. `khotan probe` remains as a legacy alias:
+
+```bash
+npx khotan plug myPlug --info                    # See endpoints
+npx khotan plug myPlug GET /products             # Fire request
+npx khotan plug myPlug --endpoint listProducts --compare  # Check schema
+```
+
+Set `KHOTAN_DEBUG=1` for verbose `[khotan:auth]` and `[khotan:request]` console logs.
+
+### Recommended Plug Workflow
+
+1. Create the plug file and auth/hook setup.
+2. Add a small set of typed endpoints directly on the plug (`listProducts`, `getProduct`, etc).
+3. Run the app with `KHOTAN_DEBUG=1`.
+4. Use `npx khotan plug myPlug --info` to confirm the endpoints are visible to the debugger.
+5. Use `npx khotan plug myPlug --endpoint listProducts --compare` against the live API.
+6. Tighten schemas until the compare output matches the real payload shape you care about.
+7. Only then build inflows, relays, outflows, or webhook handlers on top of those endpoints.
+
+The package does not paginate or delta-sync for you automatically inside user flows. Your app code decides which typed endpoints to call, what page size to use, when to stop, and how to implement full, test, partial, backfill, reconcile, or delta runs.
+
+## Managing Vars
+
+Use the CLI to inspect and update stored plug variables:
+
+```bash
+npx khotan plug vars --list
+npx khotan plug vars myPlug
+npx khotan plug vars myPlug set --json '{"apiKey":"secret","orgId":"org_live"}'
+npx khotan plug vars myPlug clear
+```
+
+Variable reads mask `secret` fields automatically. Hub and CLI both talk to the same `/api/khotan/variables/*` routes.

package/dist/templates/skill-setup.md

@@ -0,0 +1,161 @@
+---
+name: khotan-setup
+description: >
+  Set up khotan-data in a Next.js + Drizzle + Postgres project. Use when
+  initializing khotan in a new project, adding the database schema,
+  configuring the factory, or troubleshooting missing setup steps.
+---
+
+Set up khotan-data in a Next.js + Drizzle + Postgres project. Use when initializing khotan in a new project, adding the database schema, configuring the factory, or troubleshooting missing setup steps.
+
+## Quick Start
+
+```bash
+npm install khotan-data
+npx khotan init
+npx khotan add schema --yes
+npx khotan migrate
+npx khotan add plug --yes
+```
+
+## What Init Creates
+
+`npx khotan init` scaffolds three files (never overwrites existing):
+
+| File | Purpose |
+|------|---------|
+| `khotan.config.ts` | CLI config — sets `outputDir` (default: `src/khotan` or `khotan`) |
+| `{outputDir}/khotan.ts` | Factory config — register plugs, resources, adapter |
+| `src/app/api/khotan/[...all]/route.ts` | Catch-all API route |
+
+Use `npx khotan init --full` for greenfield projects — also installs drizzle-orm, postgres, drizzle-kit, and shadcn.
+
+## Factory Config Pattern
+
+Edit `{outputDir}/khotan.ts` after init:
+
+```typescript
+import { khotan, drizzleAdapter } from "khotan-data/factory";
+import { db } from "@/db";
+import { stripeChargesInflow } from "./flows/stripe-charges";
+
+const khotanData = khotan({
+  adapter: drizzleAdapter(db),
+  resources: [
+    { name: "products", mapping: { connectField: "sku" } },
+    { name: "orders", mapping: { connectField: "order_number" } },
+  ],
+  plugs: [
+    {
+      name: "stripe",
+      plug: stripePlug,
+      flows: [
+        stripeChargesInflow,
+      ],
+    },
+  ],
+});
+
+export default khotanData;
+```
+
+The factory auto-upserts plugs, flows, and resources to the database on first API request.
+
+## Route Handler
+
+The catch-all route delegates all HTTP methods to the factory:
+
+```typescript
+import { toNextJsHandler } from "khotan-data/factory";
+import khotanData from "@/lib/khotan/khotan";
+
+export const { GET, POST, PUT, PATCH, DELETE } = toNextJsHandler(khotanData.handler);
+```
+
+## Database Setup
+
+```bash
+npx khotan add schema --yes   # Scaffolds Drizzle table definitions
+npx khotan migrate             # Generates + applies migrations (needs DATABASE_URL)
+npx khotan migrate --push      # Or push directly without migration files
+```
+
+Tables created: `khotan_plugs`, `khotan_resources`, `khotan_flows`, `khotan_wires`, `khotan_runs`, `khotan_mappings`.
+
+The schema command auto-detects your Drizzle schema directory, updates `drizzle.config.ts` glob pattern, and adds the barrel re-export.
+
+## Environment Variables
+
+| Variable | Required | Purpose |
+|----------|----------|---------|
+| `DATABASE_URL` | Yes | Postgres connection (used by Drizzle) |
+| `KHOTAN_SECRET` | For variables | AES-256-GCM key for encrypting plug vars |
+| `KHOTAN_DEBUG` | For debugging | Enables `/debug/*` routes and the `plug` CLI (`probe` alias) |
+| `KHOTAN_WEBHOOK_URL` | For webhooks | Public URL for wire callbacks |
+| `CRON_SECRET` | For production cron | Protects the built-in `/api/khotan/cron` dispatcher route |
+
+## Next.js Config
+
+Add to `next.config.ts` if using local/tarball install:
+
+```typescript
+const nextConfig = {
+  serverExternalPackages: ["khotan-data"],
+};
+```
+
+## Verify Setup
+
+```bash
+curl http://localhost:3000/api/khotan/plugs     # Should list registered plugs
+curl http://localhost:3000/api/khotan/flows      # Should list flows
+curl http://localhost:3000/api/khotan/resources   # Should list resources
+```
+
+## Scheduled Flows On Vercel
+
+Khotan flow `schedule` values are runtime source-of-truth metadata. On Vercel, prefer a single dispatcher CRON instead of defining one platform CRON per flow.
+
+Add one entry to `vercel.json`:
+
+```json
+{
+  "crons": [
+    { "path": "/api/khotan/cron", "schedule": "* * * * *" }
+  ]
+}
+```
+
+Then define schedules only on your flows in `{outputDir}/khotan.ts`:
+
+```typescript
+{
+  name: "products-inflow",
+  type: "inflow",
+  schedule: "0 * * * *",
+  resource: "products",
+}
+```
+
+The dispatcher route evaluates which flows are due on each tick and starts them through the normal run-tracking path. If `CRON_SECRET` is set, Vercel should call the route with `Authorization: Bearer <CRON_SECRET>`.
+
+## Typical Build Order
+
+After init and schema setup, the usual path to a working sync is:
+
+1. Add or author a plug file for the external service.
+2. Define a few typed endpoints directly on the plug with Zod response schemas.
+3. Start the app with `KHOTAN_DEBUG=1`.
+4. Verify the plug is visible with `npx khotan plug --list` and `npx khotan plug myPlug --info`.
+5. Hit live endpoints with `npx khotan plug myPlug --endpoint listProducts --compare` until the schemas match the real API shape you intend to use.
+6. Register the plug in `{outputDir}/khotan.ts` with resources and flows.
+7. Only after endpoint verification, build inflows, relays, outflows, or webhook handlers on top of those live-checked endpoints.
+
+This keeps sync logic grounded in real API payloads before you write pagination, mapping, or transformation code.
+
+## Troubleshooting
+
+- **Empty plug list**: Factory upserts on first request — hit any endpoint first, then check `/plugs`
+- **"Cannot find module khotan-data"**: Add to `serverExternalPackages` in next.config.ts
+- **Migration fails**: Ensure `DATABASE_URL` is set and Postgres is reachable
+- **Init won't overwrite**: By design — delete the file manually if you need to re-scaffold

package/dist/templates/skill-webhook.md

@@ -0,0 +1,196 @@
+---
+name: khotan-webhook
+description: >
+  Set up webhook subscriptions and event processing with khotan Wires,
+  Catch, and Pass. Use when receiving webhooks from external services,
+  registering callback URLs, processing incoming events durably, or
+  forwarding events between services.
+---
+
+Set up webhook subscriptions and event processing with khotan Wires, Catch, and Pass. Use when receiving webhooks from external services, registering callback URLs, processing incoming events durably, or forwarding events between services.
+
+## Wire (Webhook Subscriptions)
+
+```bash
+npx khotan add wire --yes
+```
+
+Scaffolds `{outputDir}/wires/wire.ts` (the builder) and `src/components/khotan/wire.tsx` (UI panel).
+
+### Creating a Wire
+
+```typescript
+import { wire } from "./wire";
+
+export const stripeWire = wire({
+  events: ["invoice.paid", "charge.succeeded"],
+
+  async onSubscribe(ctx) {
+    const res = await ctx.plug.post<{ id: string; secret: string }>(
+      "/webhook_endpoints",
+      {
+        body: {
+          url: ctx.callbackUrl,
+          enabled_events: ctx.events,
+        },
+      },
+    );
+    await ctx.setWireVars({ webhookSecret: res.secret });
+    return { remoteId: res.id };
+  },
+
+  async onUnsubscribe(ctx) {
+    await ctx.plug.delete(`/webhook_endpoints/${ctx.remoteId}`);
+  },
+
+  async onVerify(ctx) {
+    const signature = ctx.headers["stripe-signature"];
+    // Verify HMAC using ctx.wireVars.webhookSecret and ctx.body (raw text)
+    return isValidSignature(signature, ctx.body, ctx.wireVars.webhookSecret);
+  },
+});
+```
+
+### Hook Contexts
+
+**onSubscribe** receives:
+- `ctx.plug` — Plug with vars/auth auto-injected (BoundPlug)
+- `ctx.callbackUrl` — The URL to register with the external service
+- `ctx.events` — Event types to subscribe to
+- `ctx.wireVars` / `ctx.setWireVars()` — Persist wire-specific data (secrets, tokens)
+- Must return `{ remoteId: string }`
+
+**onUnsubscribe** receives:
+- `ctx.plug` — BoundPlug
+- `ctx.remoteId` — The ID returned from onSubscribe
+- `ctx.wireVars` / `ctx.setWireVars()`
+
+**onVerify** receives:
+- `ctx.headers` — Incoming request headers
+- `ctx.body` — Raw request body (for signature verification)
+- `ctx.wireVars` — Wire-specific vars
+- Must return `boolean`
+
+### Registering Wires
+
+In `{outputDir}/khotan.ts`:
+
+```typescript
+import { stripeWire } from "./wires/stripe-wire";
+
+plugs: [
+  {
+    name: "stripe",
+    plug: stripePlug,
+    wires: [stripeWire],
+  },
+],
+```
+
+### Programmatic Wire API
+
+```typescript
+const w = khotanData.wire("stripe");
+await w.create("https://your-domain.com/api/khotan/webhook/stripe");
+await w.get();           // Get wire status
+await w.delete(wireId);  // Disconnect
+```
+
+### Webhook Callback URL
+
+Wires register callbacks at: `{webhookUrl}/api/khotan/webhook/{plugName}`
+
+For local dev, use ngrok or similar tunnel and set `KHOTAN_WEBHOOK_URL`.
+
+## Catch (Durable Event Processing)
+
+```bash
+npx khotan add catch --yes
+```
+
+Process webhook events durably via Vercel Workflow:
+
+```typescript
+import { catchEvent } from "./webhooks/catch";
+
+const processInvoice = catchEvent(async (ctx) => {
+  "use workflow";
+
+  async function persist() {
+    "use step";
+    // Write to database — retried on failure
+    await db.insert(invoices).values(ctx.event);
+  }
+
+  await persist();
+});
+```
+
+Register on the source plug:
+
+```typescript
+{ name: "stripe", plug: stripePlug, wires: [stripeWire], catches: [processInvoice] }
+```
+
+## Pass (Event Forwarding)
+
+```bash
+npx khotan add pass --yes
+```
+
+Forward webhook events to another service:
+
+```typescript
+import { pass } from "./webhooks/pass";
+
+const forwardToSlack = pass({
+  to: "slack",  // Destination plug name (must be registered)
+  workflow: async (ctx) => {
+    "use workflow";
+    // ctx.event — the incoming webhook payload
+    // ctx.destVars — destination plug variables
+    async function forward() {
+      "use step";
+      await ctx.destPlug.post("/messages", {
+        body: { text: `New event: ${ctx.event.type}` },
+      });
+    }
+    await forward();
+  },
+});
+```
+
+Register on the source plug:
+
+```typescript
+{ name: "stripe", plug: stripePlug, wires: [stripeWire], passes: [forwardToSlack] }
+```
+
+## Webhook Flow
+
+```
+External Service → POST /api/khotan/webhook/:plugName
+  → onVerify (signature check)
+  → Parse event type
+  → Start catch workflows (durable processing)
+  → Start pass workflows (event forwarding)
+  → Return { received: true }
+```
+
+## Dependencies
+
+- **Wire**: Requires `plug` and `schema` components
+- **Catch**: Requires `wire`; needs `workflow` package for Vercel Workflow
+- **Pass**: Requires `wire` and `plug`; needs `workflow` package
+
+## Hub Integration
+
+The WirePanel in the Hub UI lets users connect/disconnect webhooks from the browser. It calls `POST /api/khotan/wires/:plugName` with the callback URL.
+
+## Debugging Webhooks
+
+1. Check wire status: `GET /api/khotan/wires/:plugName`
+2. Verify `onVerify` logic: check wire vars contain the signing secret
+3. Check factory logs: `KHOTAN_DEBUG=1` enables `[khotan:wire]` log lines
+4. 401 on webhook receive = `onVerify` returning false
+5. 500 on webhook = missing `workflow` package or catch/pass misconfigured