deepline 0.0.1 → 0.1.1

package/README.md

@@ -0,0 +1,324 @@
+# Deepline V2 SDK + CLI
+
+TypeScript SDK and CLI for Deepline plays, tools, and enrichment.
+
+## Dev workflow
+
+### Prerequisites
+
+```bash
+bun run dev          # starts Next.js + Convex + Temporal + worker
+```
+
+Wait for health: `curl http://localhost:3000/api/v2/health`
+
+### Auth (one-time)
+
+```bash
+bun sdk/bin/deepline-dev.ts auth register
+```
+
+Opens browser for approval. Stores SDK CLI auth in `~/.local/deepline/<host-slug>/.env`.
+
+Check status: `bun sdk/bin/deepline-dev.ts auth status --json`
+
+### Making changes
+
+Edit any file in `sdk/src/` and re-run. No build step needed — `deepline-dev.ts` runs source directly via Bun.
+
+```bash
+# Edit sdk/src/cli/commands/play.ts
+# Then immediately:
+bun sdk/bin/deepline-dev.ts play run --file my-play.play.ts --watch
+```
+
+The root `package.json` has an alias: `bun run deepline` does the same thing.
+
+### When you need to build
+
+Only two cases require `npm run sdk:build`:
+
+1. **`npx --package=./sdk -- deepline`** — the path used by integration tests
+2. **Testing the published artifact** — verifying `dist/cli/index.mjs` works
+
+```bash
+cd sdk && bun run build    # or: npm run sdk:build from root
+```
+
+## Env files
+
+| File                                 | Set by              | Contains                                  |
+| ------------------------------------ | ------------------- | ----------------------------------------- |
+| `~/.local/deepline/<host-slug>/.env` | SDK `auth register` | `DEEPLINE_ORIGIN_URL`, `DEEPLINE_API_KEY` |
+
+`deepline-dev.ts` hard-sets `DEEPLINE_ORIGIN_URL=http://localhost:3000` so you never need to configure it manually for local dev.
+
+Config resolution order: explicit options > env vars > checkout `.env.worktree` > SDK v2 `.env` > prod fallback.
+
+## CLI commands
+
+```bash
+bun sdk/bin/deepline-dev.ts health
+bun sdk/bin/deepline-dev.ts auth register|status
+bun sdk/bin/deepline-dev.ts tools list|get|call
+bun sdk/bin/deepline-dev.ts play run --file <file.play.ts> [--input '{}'] [--csv file.csv] [--watch]
+bun sdk/bin/deepline-dev.ts play run --name <name> [--input '{}'] [--csv file.csv] [--watch]
+bun sdk/bin/deepline-dev.ts play get <file.play.ts|name> [--json]
+bun sdk/bin/deepline-dev.ts play versions --name <name> [--json]
+bun sdk/bin/deepline-dev.ts play tail --run-id <run-id> [--json]
+bun sdk/bin/deepline-dev.ts play tail --name <name> [--json]
+bun sdk/bin/deepline-dev.ts play runs --name <name> [--json]
+bun sdk/bin/deepline-dev.ts play publish <play-file.ts|name> [--latest|--revision-id <id>] [--json]
+```
+
+## Public play entrypoints
+
+These are the public ways to run plays. They intentionally map to the external
+surfaces we exercise in CI:
+
+1. CLI
+   - Repo-local Bun/source CLI: `bun sdk/bin/deepline-dev.ts play run ...`
+   - Repo script alias: `bun run deepline -- play run ...`
+   - Installed CLI: `deepline play run ...`
+2. Programmatic SDK convenience client
+   - `client.runPlay(...)`
+3. High-level SDK handles
+   - `ctx.play(name).run(...)`
+   - `ctx.play(name).runSync(...)`
+   - `ctx.runPlay(name, input)`
+4. Raw HTTP API
+   - `POST /api/v2/plays/run`
+   - `GET /api/v2/plays/run/:workflowId`
+5. In-play composition
+   - `ctx.runPlay(...)`
+
+Internal / advanced only:
+
+- `registerPlayArtifact()`
+- `stagePlayFiles()`
+- `startPlayRun()`
+- `getPlayStatus()`
+
+Those low-level primitives are still available for callers that explicitly need
+to own the package-submit-poll flow, but they are not the primary public SDK
+surface.
+
+## SDK usage (programmatic)
+
+```typescript
+import { Deepline, definePlay } from 'deepline';
+
+const ctx = await Deepline.connect();
+
+// Tools
+const tools = await ctx.tools.list();
+const result = await ctx.tools.execute({
+  tool: 'test_company_search',
+  input: {
+    domain: 'stripe.com',
+  },
+});
+const company = result.value;
+
+// Named play (already live)
+const job = await ctx.play('my-play').run({ domain: 'example.com' });
+const status = await job.status();
+const output = await job.get(); // polls until complete
+
+// Define + run a play
+const myPlay = definePlay('my-play', async (ctx, input: { domain: string }) => {
+  const company = await ctx.tools.execute({
+    id: 'company_search',
+    tool: 'test_company_search',
+    input: { domain: input.domain },
+    description: 'Look up company details by domain.',
+  });
+  return { company: company.result };
+});
+
+const detail = await myPlay.get();
+const result = await myPlay.runSync({ domain: 'example.com' });
+```
+
+## Testing
+
+```bash
+# Unit tests (no server needed)
+bun run vitest run tests/lib/plays tests/api/plays-get-route.test.ts
+
+# SDK + CLI integration (needs running dev server)
+bash scripts/test-sdk-play-cli.sh
+
+# V2 plays e2e — all free test tools, real Temporal (needs running dev server)
+bun tests/v2-plays/runner.ts --top 10
+bun tests/v2-plays/runner.ts --shard 1/4
+bun tests/v2-plays/runner.ts --file tests/v2-plays/plays/07-waterfall-basic.play.ts
+```
+
+## Repro: rate limit handling (end-to-end)
+
+Full sequence from zero to hitting + surviving rate limits via v2 plays.
+
+### 1. Install + auth
+
+```bash
+# Production
+bash <(curl -sS https://code.deepline.com/api/v2/sdk/install)
+
+# Local dev (no install needed)
+bun run dev   # start server
+bun sdk/bin/deepline-dev.ts auth register
+```
+
+### 2. Write a play that hammers the rate-limited test tool
+
+```bash
+cat > rate-limit-repro.play.ts << 'TS'
+import { definePlay } from 'deepline';
+
+// test_rate_limit has a 5 req/s shared window — this play fires 20 calls
+// to force 429s and prove the runtime retries + backs off correctly.
+export default definePlay('rate-limit-repro', async (ctx) => {
+  const items = Array.from({ length: 20 }, (_, i) => ({
+    key: `probe-${i}`,
+    row_number: i + 1,
+  }));
+
+  const results = await ctx.map('rate_limit_probes', items, { key: 'key' })
+  .step("result", (row, ctx) =>
+      ctx.tools.execute({
+        id: 'rate_limit_probe',
+        tool: 'test_rate_limit',
+        input: {
+        key: 'redis-shared-scenario',
+        row_number: row.row_number,
+      },
+        description: 'Exercise the shared rate-limit retry path.',
+      }))
+  .run();
+
+  return { total: results.length, results };
+});
+TS
+```
+
+### 3. Run it
+
+```bash
+# Dev
+bun sdk/bin/deepline-dev.ts play run --file rate-limit-repro.play.ts --watch
+
+# Production
+deepline play run --file rate-limit-repro.play.ts --watch
+```
+
+### 4. Fallback variant (multi-step with rate limits)
+
+Use `steps()` and `when()` to express row-level fallback sequences. Skipped
+conditional steps yield `null`.
+
+```bash
+cat > rate-limit-waterfall.play.ts << 'TS'
+import { definePlay, steps, when } from 'deepline';
+
+export default definePlay('rate-limit-waterfall', async (ctx) => {
+  const leads = Array.from({ length: 24 }, (_, i) => ({
+    lead_id: `lead-${String(i + 1).padStart(3, '0')}`,
+    row_number: i + 1,
+  }));
+
+  const probeFallback = steps<{ lead_id: string; row_number: number }>()
+    .step("primary_probe", (row, ctx) =>
+      ctx.tool({
+        id: 'rate_limit_probe',
+        tool: 'test_rate_limit',
+        input: {
+        key: 'redis-shared-scenario',
+        lead_id: row.lead_id,
+        row_number: row.row_number,
+      },
+        description: 'Exercise primary rate-limit behavior.',
+      }))
+    .step("backup_probe", when(
+      (row) => !row.primary_probe,
+      (row, ctx) =>
+        ctx.tool({
+          id: 'rate_limit_probe_backup',
+          tool: 'test_rate_limit',
+          input: {
+          key: 'redis-shared-scenario',
+          lead_id: row.lead_id,
+          row_number: row.row_number,
+        },
+          description: 'Exercise fallback rate-limit behavior.',
+        }),
+    ))
+    .return((row) => row.primary_probe ?? row.backup_probe ?? null);
+
+  // Fan out over leads — each gets a row-level fallback sequence
+  const results = await ctx.map('leads', leads, { key: 'lead_id' })
+    .step("probe", probeFallback)
+    .run();
+
+  return results;
+});
+TS
+```
+
+```bash
+bun sdk/bin/deepline-dev.ts play run --file rate-limit-waterfall.play.ts --watch
+```
+
+### What to expect
+
+- `test_rate_limit` has a Redis-backed rate limit: **5 req/s, 3 max concurrency**.
+- The SDK HTTP client auto-retries 429s with exponential backoff (1s → 2s → 4s, capped at 30s, up to 4 attempts).
+- The play rate limit scheduler (`PlayRateLimitScheduler`) paces tool calls using `queue_hint` metadata from rate limit definitions.
+- All 20/24 rows should complete — no terminal rate limit errors in the output.
+- The waterfall variant proves that short-circuiting (skipping later steps when earlier steps match) works correctly under rate pressure.
+
+### V1 equivalent
+
+The v1 repro scripts use `deepline enrich` (Python CLI) with `--with` JSON specs. The v2 plays replace this with typed TypeScript — same backend rate limit infrastructure, better DX.
+
+```bash
+# V1 (still works)
+scripts/repro-rate-limit-enrich.sh redis_429
+scripts/repro-rate-limit-enrich.sh redis_429_waterfall
+```
+
+## Transpiling native plays
+
+Convert V1 native `PlayDocument` definitions to V2 `definePlay()` files:
+
+```bash
+bun scripts/transpile-native-plays.ts                     # all -> src/lib/plays/generated/v2/
+bun scripts/transpile-native-plays.ts --play test_play     # single play
+bun scripts/transpile-native-plays.ts --stdout             # print to stdout
+```
+
+## File structure
+
+```
+sdk/
+  bin/deepline-dev.ts           # Dev entry (bun, no build, points at localhost)
+  src/
+    index.ts                    # Public exports
+    config.ts                   # Config + env resolution
+    http.ts                     # Fetch client with retries
+    client.ts                   # DeeplineClient (tools, plays API)
+    play.ts                     # definePlay, Deepline.connect, job handles
+    types.ts                    # TypeScript interfaces
+    errors.ts                   # DeeplineError, AuthError, RateLimitError
+    cli/
+      index.ts                  # CLI router
+      commands/
+        auth.ts                 # auth register/status
+        play.ts                 # play run/get/tail/runs/publish
+        tools.ts                # tools list/get/call
+        enrich.ts               # CSV enrichment (delegates to Python CLI)
+    plays/
+      bundle-play-file.ts       # esbuild bundler for .play.ts files
+      local-file-discovery.ts   # ctx.csv() file resolution + packaging
+  dist/                         # Build output (gitignored)
+```