@mevdragon/vidfarm-devcli 0.1.0

package/PLATFORM_SPEC.md

@@ -0,0 +1,898 @@
+# Vidfarm Platform Spec
+
+This document defines the initial platform architecture for Vidfarm running on a single Dockerized EC2 host.
+
+The goal is to let in-house developers build new video production templates quickly while the platform centrally owns auth, billing, job orchestration, customer state, and deployment.
+
+## Goals
+
+1. Support multiple video production patterns behind one consistent API.
+2. Treat every operation as an async job that immediately returns `job_id`.
+3. Let template developers write normal TypeScript/Node code with normal npm dependencies.
+4. Keep the first production deployment simple enough to run on one EC2 Docker host.
+5. Preserve a clean path to later split heavy workloads into isolated workers or separate services.
+
+## Non-Goals
+
+1. Public marketplace for third-party user-submitted templates.
+2. Multi-region or scale-to-zero serverless deployment in v1.
+3. Full microservice isolation for every template.
+4. Perfect cost attribution in v1. The first target is safe, conservative billing that protects gross margin.
+
+## Core Decision
+
+Vidfarm should run as one shared platform container on EC2.
+
+Templates should be packaged as internal code modules loaded by that platform, not as separately deployed HTTP services by default.
+Platform runtime code should live under `src/*`, while template implementation code should live outside the platform tree under `templates/<template-folder>/*`.
+
+This gives us:
+
+1. One auth and billing boundary.
+2. One job table and one queueing system.
+3. One deployment artifact for the normal case.
+4. Full npm freedom for template developers.
+5. A cleaner developer experience than forcing every template to become its own service.
+
+Templates may still opt into isolated execution later when they have special requirements such as native binaries, unusually high memory use, or independent scaling needs.
+
+## Initial Runtime Choice
+
+### Production Runtime
+
+- Node.js 22
+- TypeScript
+- Hono for the HTTP API
+- SQLite for v1 job state and queue state
+- S3 for customer files and generated artifacts
+- Remotion Lambda for final render workloads where appropriate
+
+### Why Hono
+
+Hono is a good fit for the control plane:
+
+1. Small and fast.
+2. Strong middleware model.
+3. Good TypeScript ergonomics.
+4. Easy to keep the HTTP layer thin while most complexity lives in jobs and template execution.
+
+### Why Not Bun for v1 Runtime
+
+Bun is not forbidden, but it should not be the production baseline for v1.
+
+Reasons:
+
+1. The platform already assumes Node-oriented Docker execution.
+2. AI SDK compatibility and native package behavior are more predictable on Node.
+3. Remotion and adjacent tooling are safer on the Node compatibility path.
+4. The hard part of this system is orchestration correctness, not JavaScript runtime speed.
+
+If desired, Bun can be evaluated later as a local development runner or for specific internal tools.
+
+## Supported Production Patterns
+
+The platform must support all of the following under one model:
+
+1. Pure AI multi-stage production.
+2. Remotion render pipelines.
+3. Hybrid research plus render pipelines.
+4. Animated storytelling pipelines.
+
+The common abstraction is:
+
+1. Customer hits a template endpoint.
+2. Platform validates auth and input.
+3. Platform creates an async job.
+4. Worker executes the requested operation.
+5. Customer polls job state or receives a webhook.
+
+## Architectural Overview
+
+```txt
+Client
+  -> Hono API
+  -> Auth / Billing / Template Registry / Job Creation
+  -> SQLite (jobs, logs, queue, rate-limit state, customer metadata pointers)
+  -> Worker Loop in same container
+  -> External providers (OpenAI, Gemini, OpenRouter, Perplexity, etc.)
+  -> S3 (workspace files, stage artifacts, final outputs)
+  -> Remotion Lambda (when render pipeline requires it)
+```
+
+Initial deployment is one process image with two logical responsibilities:
+
+1. API server
+2. Background worker / dispatcher
+
+These can run in the same container in v1. If needed later, they can be split into separate process types using the same codebase.
+
+## API Principles
+
+All template operations are async-first.
+
+Even if a task could finish quickly, the platform should still prefer job creation so the customer sees one consistent model.
+
+### Base Path
+
+```txt
+/templates/:templateId/*
+```
+
+### Core Endpoints
+
+```txt
+GET    /templates/:templateId
+POST   /templates/:templateId/config
+POST   /templates/:templateId/operations/:operationName
+GET    /templates/:templateId/jobs/:jobId
+GET    /templates/:templateId/jobs/:jobId/logs
+POST   /templates/:templateId/jobs/:jobId/cancel
+```
+
+### Request Headers
+
+```txt
+vidfarm-user-id: string
+vidfarm-api-key: string
+```
+
+### Job Creation Request Shape
+
+```json
+{
+  "tracer": "client-generated-string",
+  "payload": {}
+}
+```
+
+### Job Creation Response Shape
+
+```json
+{
+  "job_id": "job_xxx",
+  "tracer": "client-generated-string",
+  "status": "queued"
+}
+```
+
+## Template Model
+
+Templates are normal TypeScript packages with unrestricted internal structure and should live under a repo-level `templates/` directory, for example `templates/template_0000/*`.
+
+They may:
+
+1. Import npm libraries.
+2. Define helper modules.
+3. Bundle prompt files.
+4. Include Remotion compositions.
+5. Call provider SDKs.
+6. Run arbitrary internal orchestration logic.
+
+The framework should not force templates into a single-file callback model.
+
+### Template Contract
+
+The external API surface should be defined as operations, not just raw stage names.
+
+Suggested shape:
+
+```ts
+export const myTemplate = defineTemplate({
+  id: "ugc-voiceover-v1",
+  version: "1.0.0",
+  description: "Short-form UGC voiceover pipeline",
+  configSchema: z.object({
+    defaultProvider: z.enum(["openai", "gemini", "openrouter", "perplexity"]).default("openai")
+  }),
+
+  operations: {
+    scaffold: {
+      description: "Generate a script scaffold.",
+      inputSchema: z.object({
+        topic: z.string()
+      }),
+      workflow: "scaffoldWorkflow",
+      providerHint: "openai",
+      webhookSupport: true
+    },
+    render: {
+      description: "Submit final render work.",
+      inputSchema: z.object({
+        storyboardId: z.string()
+      }),
+      workflow: "renderWorkflow",
+      webhookSupport: true
+    }
+  },
+
+  jobs: {
+    async scaffoldWorkflow(ctx, input) {
+      return {
+        progress: 1,
+        output: {}
+      };
+    },
+    async renderWorkflow(ctx, input) {
+      return {
+        progress: 1,
+        output: {}
+      };
+    }
+  }
+});
+```
+
+### Why This Contract
+
+This separates:
+
+1. Public API entrypoints.
+2. Internal workflow implementation.
+3. Template metadata and validation.
+
+It gives template developers full control over their workflow logic while keeping the platform contract stable.
+
+## Template Execution Context
+
+Each operation or job should receive a framework-owned context object.
+
+Suggested capabilities:
+
+```ts
+interface TemplateJobContext {
+  env: "development" | "production";
+  customer: CustomerContext;
+  templateConfig: Record<string, unknown>;
+  logger: {
+    debug(message: string, metadata?: Record<string, unknown>): void;
+    info(message: string, metadata?: Record<string, unknown>): void;
+    warn(message: string, metadata?: Record<string, unknown>): void;
+    error(message: string, metadata?: Record<string, unknown>): void;
+    progress(progress: number, message: string, metadata?: Record<string, unknown>): void;
+  };
+  jobs: {
+    enqueueChild(input: {
+      operationName: string;
+      workflowName: string;
+      payload: Record<string, unknown>;
+      providerHint?: ProviderType;
+    }): Promise<{ jobId: string }>;
+  };
+  storage: {
+    putJson(key: string, value: unknown): Promise<{ key: string; url: string | null }>;
+    putText(key: string, value: string, contentType?: string): Promise<{ key: string; url: string | null }>;
+    putBuffer(
+      key: string,
+      value: Uint8Array,
+      options?: { contentType?: string; kind?: string; metadata?: Record<string, unknown> }
+    ): Promise<{ key: string; url: string | null }>;
+    getPublicUrl(key: string): string | null;
+  };
+  billing: {
+    record(input: {
+      type: "ai_generation" | "render" | "storage_write" | "cpu_estimate";
+      costUsd: number;
+      chargeUsd?: number;
+      metadata?: Record<string, unknown>;
+    }): Promise<void>;
+  };
+  providers: {
+    generateText(input: {
+      provider: ProviderType;
+      model: string;
+      prompt: string;
+      temperature?: number;
+    }): Promise<{ text: string }>;
+    generateImage(input: {
+      provider: ProviderType;
+      model: string;
+      prompt: string;
+      size?: string;
+    }): Promise<{ bytes: Uint8Array; contentType: string; revisedPrompt: string | null }>;
+    analyzeImageLayout(input: {
+      provider: ProviderType;
+      model: string;
+      imageUrl: string;
+      overlayText: string;
+    }): Promise<{
+      zone: "top" | "center" | "bottom";
+      align: "left" | "center" | "right";
+      maxWidthPercent: number;
+      justification: string;
+    }>;
+  };
+  remotion: {
+    render(input: {
+      compositionId: string;
+      serveUrl?: string;
+      entryPoint?: string;
+      outputKey?: string;
+      inputProps: Record<string, unknown>;
+    }): Promise<{ renderId: string; outputUrl: string | null; metadata: Record<string, unknown> }>;
+  };
+}
+```
+
+Framework-owned context capabilities should include:
+
+1. Resolving customer AI keys safely.
+2. Writing artifacts through a stable storage prefix.
+3. Enqueuing child jobs.
+4. Emitting logs and progress.
+5. Recording billable events.
+6. Submitting downstream renders through a Remotion adapter.
+7. Calling provider adapters through centralized rate-limit enforcement.
+
+## Environment Behavior
+
+The platform must clearly distinguish development from production.
+
+### Development
+
+Developer-owned API keys from local `.env` are allowed.
+
+This is for:
+
+1. Local testing.
+2. Template development.
+3. Dry-running internal workflows before deployment.
+
+### Production
+
+The platform must use customer-owned provider keys stored in the customer profile when the template requests external AI inference on behalf of that customer.
+
+Platform-controlled keys may still exist for:
+
+1. Platform-level fallback behavior.
+2. Internal moderation or diagnostics.
+3. Emergency operations.
+
+But customer-billed workloads should default to customer-owned keys when available.
+
+## Customer Profile Model
+
+Each customer profile should support:
+
+1. Multiple provider API keys.
+2. Multiple keys per provider.
+3. Workspace file storage references.
+4. Webhook destinations.
+5. Billing preferences and limits.
+
+Suggested provider key record:
+
+```ts
+interface CustomerProviderKey {
+  id: string;
+  provider: "openai" | "gemini" | "openrouter" | "perplexity";
+  encryptedSecret: string;
+  label?: string;
+  status: "active" | "paused" | "rate_limited" | "invalid";
+  lastUsedAt?: string;
+  cooldownUntil?: string;
+}
+```
+
+Customers may store multiple keys for the same provider. The platform should treat those keys as a small pooled resource that jobs must acquire before making outbound AI requests.
+
+## Queueing and Async Jobs
+
+The platform is async-native.
+
+Every operation should create a job record and return immediately.
+
+### Job State
+
+Suggested states:
+
+```txt
+queued
+running
+waiting_for_child
+waiting_for_human
+succeeded
+failed
+cancelled
+```
+
+### Job Data
+
+Each job record should track:
+
+1. `job_id`
+2. `template_id`
+3. `operation_name`
+4. `tracer`
+5. `status`
+6. `payload`
+7. `result`
+8. `error`
+9. `progress`
+10. `webhook_url`
+11. `parent_job_id`
+12. `customer_id`
+13. `reservation_id` or billing reference
+14. timestamps
+
+### Logs
+
+Logs must be stored as structured job events, not just raw text.
+
+Each event should support:
+
+1. timestamp
+2. level
+3. message
+4. machine-readable metadata
+5. progress update
+6. artifact references
+
+This lets the client render a live job timeline later.
+
+## SQLite-Backed AI Key Queue
+
+SQLite is not only the v1 job store. It is also the coordination layer for customer AI API key usage.
+
+The intended model is:
+
+1. A job becomes runnable.
+2. The worker identifies which provider and model the next step requires.
+3. The worker attempts to lease one eligible customer key from SQLite.
+4. If a lease is granted, the worker performs the outbound API call.
+5. The worker records usage, updates cooldown state if needed, and releases the lease.
+
+This gives the platform a lightweight queue for AI key access without needing Redis, SQS, or a separate lock service.
+
+### Why SQLite Is Acceptable in v1
+
+This is a reasonable design if all of the following remain true:
+
+1. One EC2 host is the active source of truth.
+2. The platform runs a moderate number of worker loops.
+3. SQLite is configured in WAL mode.
+4. Lease acquisition is done transactionally.
+5. Jobs are retry-safe and can be rescheduled when no key is available.
+
+### Core Idea
+
+The AI key queue is represented by a combination of:
+
+1. Customer provider key records.
+2. Active key lease records.
+3. Key usage and error events.
+4. Cooldown timestamps after rate-limit responses.
+
+There is no separate message broker for API key access. Eligibility is derived from database state at lease time.
+
+### Suggested Tables
+
+```sql
+create table customer_provider_keys (
+  id text primary key,
+  customer_id text not null,
+  provider text not null,
+  label text,
+  encrypted_secret text not null,
+  status text not null,
+  weight integer not null default 1,
+  last_used_at text,
+  cooldown_until text,
+  disabled_reason text,
+  created_at text not null,
+  updated_at text not null
+);
+
+create table provider_key_leases (
+  key_id text primary key,
+  lease_token text not null,
+  worker_id text not null,
+  job_id text not null,
+  leased_at text not null,
+  expires_at text not null
+);
+
+create table provider_key_usage_events (
+  id text primary key,
+  key_id text not null,
+  job_id text not null,
+  provider text not null,
+  model text,
+  event_type text not null,
+  input_tokens integer,
+  output_tokens integer,
+  cost_usd real,
+  created_at text not null
+);
+```
+
+Optional model capability table:
+
+```sql
+create table provider_key_capabilities (
+  key_id text not null,
+  model text not null,
+  primary key (key_id, model)
+);
+```
+
+### Lease Acquisition
+
+Workers must acquire a key lease before making any outbound provider request on behalf of a customer.
+
+Lease acquisition should happen inside a transaction using `BEGIN IMMEDIATE`.
+
+The query should exclude keys that are:
+
+1. Not active.
+2. In cooldown.
+3. Already leased and whose lease has not expired.
+4. Incompatible with the requested provider or model.
+
+Preferred selection order in v1:
+
+1. Least recently used eligible key.
+2. Higher weight first when weights differ.
+
+Illustrative flow:
+
+```txt
+BEGIN IMMEDIATE
+1. Select one eligible key
+2. Insert active lease row
+3. Commit
+```
+
+Illustrative query shape:
+
+```sql
+select k.id
+from customer_provider_keys k
+left join provider_key_leases l
+  on l.key_id = k.id
+  and l.expires_at > datetime('now')
+where k.customer_id = ?
+  and k.provider = ?
+  and k.status = 'active'
+  and (k.cooldown_until is null or k.cooldown_until <= datetime('now'))
+  and l.key_id is null
+order by k.last_used_at asc nulls first, k.weight desc
+limit 1;
+```
+
+If a key is found, the worker inserts a lease row with a short expiry such as 30 to 90 seconds.
+
+If no key is found, the worker must not busy-loop. It should reschedule the job for a future run.
+
+### Lease Semantics
+
+Lease rows should contain:
+
+1. `key_id`
+2. `lease_token`
+3. `worker_id`
+4. `job_id`
+5. `leased_at`
+6. `expires_at`
+
+The lease token should be required for release or extension so one worker cannot accidentally release another worker's lease.
+
+### Lease Expiry and Recovery
+
+If a worker crashes, its lease should naturally expire and the key should become eligible again.
+
+For long-running requests, the platform may optionally support lease extension heartbeats. This is useful when the provider call or downstream processing can exceed the default lease duration.
+
+### Success Path
+
+After a successful provider call, the worker should:
+
+1. Record a usage event.
+2. Update `last_used_at`.
+3. Clear any temporary rate-limit status when appropriate.
+4. Release the lease.
+
+### Rate-Limit Path
+
+If the provider returns a rate-limit response, the worker should:
+
+1. Record a `rate_limit` usage event.
+2. Put the key into cooldown by setting `cooldown_until`.
+3. Release the lease.
+4. Reschedule the job.
+
+Cooldown duration may initially be determined by:
+
+1. Provider response headers if available.
+2. Provider-specific backoff policy.
+3. Conservative defaults when headers are absent.
+
+### Auth Failure Path
+
+If the provider reports invalid credentials, the worker should:
+
+1. Record an `auth_error` usage event.
+2. Mark the key `invalid`.
+3. Release the lease.
+4. Retry with another key if one exists.
+5. Fail the job clearly if no valid key remains.
+
+### Scheduler Behavior
+
+The scheduler should treat jobs as runnable only when both are true:
+
+1. The job itself is ready to run.
+2. A compatible provider key can likely be leased now or soon.
+
+Recommended loop:
+
+1. Fetch queued jobs ordered by `run_after`.
+2. Attempt key lease acquisition for the next provider-dependent step.
+3. If lease succeeds, run the step and mark job `running`.
+4. If lease fails, move `run_after` forward instead of spinning.
+5. Retry later.
+
+This is what makes the AI key queue effectively a SQLite-backed coordination system rather than a separate infrastructure dependency.
+
+### Observability
+
+The platform should emit logs and metrics for:
+
+1. Lease acquisition success rate.
+2. Lease wait time.
+3. Key cooldown frequency by provider.
+4. Key invalidation frequency.
+5. Job deferrals caused by unavailable keys.
+
+These signals will tell us when SQLite remains sufficient and when key coordination needs a stronger backend.
+
+## Rate Limiting and Provider Routing
+
+Customer keys are not interchangeable infinite resources.
+
+The platform must route AI calls through a provider layer that understands:
+
+1. Provider type.
+2. Model name.
+3. Key-level rate limits.
+4. Backoff behavior.
+5. Retry policy.
+6. Temporary key disablement after provider errors.
+
+### Initial Approach
+
+Use SQLite-backed leasing for queue and key selection in v1.
+
+This is acceptable if:
+
+1. One EC2 host is the source of truth.
+2. SQLite is configured in WAL mode.
+3. Concurrency expectations remain moderate.
+4. Jobs are idempotent enough to survive retries.
+
+### Future Upgrade Path
+
+If platform concurrency or reliability needs outgrow SQLite, move the job and rate-limit state to Postgres before adopting many-worker horizontal scale.
+
+## Billing Model
+
+The platform owns billing enforcement, but templates should emit billing events through framework APIs.
+
+Templates should not hand-roll their own pricing logic in arbitrary ways.
+
+### Billing Principle
+
+Bill conservatively enough to avoid cloud-cost loss.
+
+Current target:
+
+```txt
+customer_charge_usd ~= platform_cost_usd * 2
+```
+
+This is a margin buffer, not a final finance system.
+
+### Billing Event Types
+
+The framework should support at least:
+
+1. External AI token usage.
+2. Remotion render usage.
+3. EC2 / CPU / memory approximation.
+4. Storage writes.
+5. Data egress or expensive file processing when relevant.
+
+### Billing API for Templates
+
+Templates should call framework helpers like:
+
+```ts
+await ctx.billing.record({
+  type: "ai_generation",
+  provider: "openai",
+  model: "gpt-4.1",
+  estimatedCostUsd: 0.024,
+  metadata: {},
+});
+```
+
+The framework should translate these events into customer-facing charges.
+
+## Webhooks
+
+Every job may include an optional webhook destination.
+
+The platform should emit webhook events for:
+
+1. `job.queued`
+2. `job.running`
+3. `job.progress`
+4. `job.succeeded`
+5. `job.failed`
+6. `job.cancelled`
+
+Webhook delivery should be:
+
+1. Signed
+2. Retried with backoff
+3. Persisted as delivery attempts
+
+## File and Artifact Storage
+
+S3 is the system of record for customer-uploaded files and large generated outputs.
+
+### Customer Workspace Convention
+
+Suggested logical prefix:
+
+```txt
+s3://bucket/customers/:customerId/workspace/...
+```
+
+### Job Artifact Convention
+
+Suggested logical prefix:
+
+```txt
+s3://bucket/templates/:templateId/users/:userId/jobs/:jobId/...
+```
+
+Artifacts may include:
+
+1. Prompt snapshots
+2. Storyboards
+3. Preview images
+4. Audio assets
+5. Subtitle files
+6. Render manifests
+7. Final video outputs
+
+This prefix is for template-generated outputs and intermediate artifacts. Keep it stable across storage backends so local development mirrors production object layout.
+
+## Remotion Integration
+
+Remotion should be treated as a specialized downstream execution path, not the center of the platform.
+
+Templates can invoke Remotion through a framework adapter.
+
+Suggested flow:
+
+1. Template job prepares structured render input.
+2. Template writes required assets through framework storage.
+3. Template calls `ctx.remotion.render(...)`.
+4. The adapter renders locally or via Lambda depending on environment config.
+5. Final artifact is attached back to the parent job result.
+
+This keeps Remotion as one implementation detail among several, rather than forcing the platform to be Remotion-first.
+
+## Isolation Policy
+
+Default mode is shared in-process execution inside the main platform runtime.
+
+### Shared Execution Is Correct By Default
+
+Use shared execution when the template:
+
+1. Uses standard Node dependencies.
+2. Fits within normal memory and CPU budgets.
+3. Does not need a custom OS image.
+4. Can safely coexist with other templates.
+
+### Isolated Execution Is an Escape Hatch
+
+Allow a template to declare isolated execution later if it needs:
+
+1. Heavy FFmpeg or native binary workloads.
+2. Custom Chromium or system library requirements.
+3. A stricter security or resource boundary.
+4. Independent scaling or scheduling.
+
+Even then, the main platform should still own:
+
+1. Auth
+2. Billing
+3. Job creation
+4. Customer state
+5. Webhook delivery
+
+## Suggested Repository Shape
+
+```txt
+/src
+/templates/template_0000
+/templates/template_0001
+/AWS_REMOTION_HANDOFF.md
+/PLATFORM_SPEC.md
+/SKILL.developer.md
+```
+
+## Suggested Internal Components
+
+### Platform API
+
+Responsibilities:
+
+1. Request validation
+2. Auth
+3. Template lookup
+4. Config updates
+5. Job creation
+6. Job status reads
+7. Webhook registration
+
+### Worker / Dispatcher
+
+Responsibilities:
+
+1. Pull queued jobs
+2. Acquire provider key leases
+3. Execute template jobs
+4. Persist logs and artifacts
+5. Update billing
+6. Deliver completion webhooks
+
+### Template Registry
+
+Responsibilities:
+
+1. Register approved in-house templates
+2. Expose metadata
+3. Resolve operations and jobs
+4. Enforce version compatibility
+
+## Security Notes
+
+1. Customer provider keys must be encrypted at rest.
+2. API keys must be hash-verified, not stored in plaintext.
+3. Template code is trusted internal code in v1, not untrusted tenant code.
+4. Webhook signatures must be mandatory.
+5. Customer file access must always be scoped by customer identity.
+
+## Operational Notes for v1
+
+1. Run one Docker image on one EC2 host.
+2. Keep API and worker in the same deployable unit initially.
+3. Use SQLite in WAL mode.
+4. Back up SQLite and treat it as transitional infrastructure.
+5. Store all large artifacts in S3, not on container disk.
+6. Assume template code is centrally reviewed before deployment.
+
+## Known Limits of v1
+
+1. SQLite is a reasonable starting point but not the final queueing backend for large-scale concurrency.
+2. Shared in-process template execution is simpler operationally but weaker as a hard isolation boundary.
+3. EC2 cost attribution will begin as approximation plus provider-cost tracking, not perfect real-time infrastructure metering.
+
+## Recommendation Summary
+
+The recommended initial Vidfarm platform is:
+
+1. One Node.js 22 Docker container on EC2.
+2. Hono as the HTTP API layer.
+3. SQLite as the initial jobs and rate-limit store.
+4. S3 for workspace and artifact storage.
+5. Remotion Lambda as a downstream rendering path.
+6. Templates implemented as normal internal TypeScript packages with full npm access.
+7. Public API defined as async operations that enqueue jobs.
+8. Optional isolated template execution added later only when justified by concrete workload needs.
+
+This is the simplest architecture that matches the product goals without prematurely turning each template into a separate service.