@mevdragon/vidfarm-devcli 0.1.0 → 0.2.0

package/.env.example

@@ -9,13 +9,20 @@ API_KEY_SALT="replace-with-a-second-long-random-secret"
 WEBHOOK_SECRET="replace-with-a-third-long-random-secret"
 
 WORKER_POLL_MS="1500"
-WORKER_BATCH_SIZE="4"
+WORKER_BATCH_SIZE="2"
+WORKER_MAX_CONCURRENT_JOBS="1"
 DEFAULT_JOB_DELAY_SECONDS="20"
+MAX_PENDING_JOBS_GLOBAL="0"
+MAX_PENDING_JOBS_PER_CUSTOMER="0"
 
-PUBLIC_BASE_URL="http://localhost:3000"
+PUBLIC_BASE_URL="https://vidfarm.cloud.zoomgtm.com"
+VIDFARM_ADMIN_EMAILS="admin@example.com"
+VIDFARM_DEVELOPER_EMAILS="developer@example.com"
+TEMPLATE_SOURCE_ROOT="./data/template-sources"
 STORAGE_DRIVER="local"
 AWS_REGION="us-east-1"
 AWS_S3_BUCKET=""
+AWS_S3_PUBLIC_READ="false"
 AWS_S3_ENDPOINT=""
 AWS_ACCESS_KEY_ID=""
 AWS_SECRET_ACCESS_KEY=""
@@ -31,9 +38,9 @@ PERPLEXITY_API_KEY=""
 REMOTION_FUNCTION_NAME=""
 REMOTION_REGION="us-east-1"
 REMOTION_BUCKET_NAME=""
-REMOTION_SITE_NAME="vidfarm-demo-template"
+REMOTION_SITE_NAME="vidfarm-template-0000"
 REMOTION_SERVE_URL=""
-REMOTION_COMPOSITION_ID="demo-template"
+REMOTION_COMPOSITION_ID="template-0000"
 REMOTION_AWS_ACCESS_KEY_ID=""
 REMOTION_AWS_SECRET_ACCESS_KEY=""
 REMOTION_MODE="auto"

package/PLATFORM_SPEC.md

@@ -4,6 +4,8 @@ This document defines the initial platform architecture for Vidfarm running on a
 
 The goal is to let in-house developers build new video production templates quickly while the platform centrally owns auth, billing, job orchestration, customer state, and deployment.
 
+For template code distribution in v1.1, the platform should support GitHub-backed in-house template sources, but production activation must still be manual, admin-approved, and pinned to a specific commit SHA.
+
 ## Goals
 
 1. Support multiple video production patterns behind one consistent API.
@@ -36,6 +38,38 @@ This gives us:
 
 Templates may still opt into isolated execution later when they have special requirements such as native binaries, unusually high memory use, or independent scaling needs.
 
+## Template Source Of Truth
+
+Template code may live in GitHub, but the platform should not auto-pull floating branch heads into production.
+
+Approved v1.1 release model:
+
+1. developer keeps template code in GitHub
+2. the default import branch is `production`
+3. admin manually reviews the repo and chooses when to import
+4. admin publishes the approved Remotion site bundle if the template needs Remotion
+5. platform resolves the chosen branch head to a commit SHA
+6. platform builds and certifies that exact commit
+7. platform activates a pinned release record for that commit
+8. admin rebuilds and redeploys the production Docker image with the approved release set
+
+Live platform state should therefore point to:
+
+- repo URL
+- branch name
+- exact commit SHA
+- certification result
+- active/inactive release state
+
+It should not point only to a floating branch name.
+
+Release authority should be centralized:
+
+- developers can push source code to GitHub
+- developers cannot directly publish to shared Remotion AWS
+- developers cannot directly promote templates into production Docker
+- shared AWS publish, activation, and production deployment are admin-only steps
+
 ## Initial Runtime Choice
 
 ### Production Runtime
@@ -122,8 +156,11 @@ Even if a task could finish quickly, the platform should still prefer job creati
 
 ```txt
 GET    /templates/:templateId
+GET    /templates/:templateId/about/*
+GET    /templates/:templateId/skill
 POST   /templates/:templateId/config
 POST   /templates/:templateId/operations/:operationName
+GET    /templates/:templateId/jobs
 GET    /templates/:templateId/jobs/:jobId
 GET    /templates/:templateId/jobs/:jobId/logs
 POST   /templates/:templateId/jobs/:jobId/cancel
@@ -155,6 +192,62 @@ vidfarm-api-key: string
 }
 ```
 
+### Template Metadata Response Shape
+
+```json
+{
+  "id": "4c7a7e1a-7f35-4f30-9f86-9c8a63c7f2db",
+  "slug_id": "template_0000",
+  "version": "1.0.0",
+  "title": "Template 0000",
+  "description": "Short-form slideshow pipeline",
+  "viral_dna": "Fast TikTok slideshow hooks with mobile-native pacing.",
+  "preview_media": [
+    "https://api.example.com/templates/4c7a7e1a-7f35-4f30-9f86-9c8a63c7f2db/about/preview-01.jpg"
+  ],
+  "link_to_original": "https://www.tiktok.com/@example/video/1234567890",
+  "skill_url": "https://api.example.com/templates/4c7a7e1a-7f35-4f30-9f86-9c8a63c7f2db/skill",
+  "operations": [
+    {
+      "name": "create_slideshow",
+      "description": "Generate slideshow frames.",
+      "providerHint": "openrouter"
+    }
+  ]
+}
+```
+
+`GET /templates/:templateId/about/*` should expose template metadata assets such as preview images or videos. In storage, these assets should use the stable logical prefix `templates/:templateId/about/*`, whether the backing store is local disk or S3.
+
+Template definitions should expose `preview_media` as absolute HTTPS URLs. When the backing store is S3, those entries should point directly at the S3 object for assets stored under `templates/:templateId/about/*`.
+
+### Job List Filtering
+
+`GET /templates/:templateId/jobs` should support:
+
+- `tracer`
+- `start_time`
+- `end_time`
+- `limit`
+
+`GET /templates/:templateId/jobs/:jobId/logs` should use the same time window language:
+
+- `start_time`
+- `end_time`
+- `limit`
+
+`logs_from` is deprecated and should not be used in the standard.
+
+`GET /templates/:templateId` is the template "about" response and must also return:
+
+- `slug_id: string`
+- `title: string`
+- `viral_dna: string`
+- `preview_media: string[]`
+- `link_to_original: string`
+
+In the template definition standard, this response `title` and `description` should be sourced from `template.about.title` and `template.about.description`, not top-level template fields.
+
 ## Template Model
 
 Templates are normal TypeScript packages with unrestricted internal structure and should live under a repo-level `templates/` directory, for example `templates/template_0000/*`.
@@ -178,9 +271,16 @@ Suggested shape:
 
 ```ts
 export const myTemplate = defineTemplate({
-  id: "ugc-voiceover-v1",
+  id: "123e4567-e89b-42d3-a456-426614174000",
+  slugId: "ugc_voiceover_v1",
   version: "1.0.0",
-  description: "Short-form UGC voiceover pipeline",
+  about: {
+    title: "UGC Voiceover V1",
+    description: "Short-form UGC voiceover pipeline",
+    viral_dna: "Fast hooks, native pacing, and repeatable creator-style framing.",
+    preview_media: ["https://cdn.example.com/templates/ugc-voiceover-v1/about/preview-01.mp4"],
+    link_to_original: "https://www.tiktok.com/@example/video/1234567890"
+  },
   configSchema: z.object({
     defaultProvider: z.enum(["openai", "gemini", "openrouter", "perplexity"]).default("openai")
   }),
@@ -337,6 +437,8 @@ This is for:
 2. Template development.
 3. Dry-running internal workflows before deployment.
 
+Template authors should also be able to run the same certification harness locally through the developer CLI before admin review.
+
 ### Production
 
 The platform must use customer-owned provider keys stored in the customer profile when the template requests external AI inference on behalf of that customer.
@@ -349,6 +451,20 @@ Platform-controlled keys may still exist for:
 
 But customer-billed workloads should default to customer-owned keys when available.
 
+Admin-only template source import and activation are allowed in production. Developer template changes should not go live until an admin explicitly imports and activates a pinned release.
+
+## Developer And Admin Auth Model
+
+The platform’s API auth remains customer-style `vidfarm-user-id` plus `vidfarm-api-key`, but the platform must also support an admin allowlist for template-source management endpoints.
+
+Minimum v1.1 rule:
+
+1. all normal template execution uses standard platform auth
+2. template source registration, import, and activation endpoints are admin-only
+3. admin authorization may begin as an allowlist of trusted emails
+
+This keeps v1 simple while still distinguishing runtime customers from internal release operators.
+
 ## Customer Profile Model
 
 Each customer profile should support:
@@ -768,6 +884,16 @@ Artifacts may include:
 
 This prefix is for template-generated outputs and intermediate artifacts. Keep it stable across storage backends so local development mirrors production object layout.
 
+### Template Metadata Convention
+
+Suggested logical prefix:
+
+```txt
+s3://bucket/templates/:templateId/about/...
+```
+
+This prefix is for framework-owned template metadata assets such as preview images, preview videos, and other public "about" media referenced by `GET /templates/:templateId`.
+
 ## Remotion Integration
 
 Remotion should be treated as a specialized downstream execution path, not the center of the platform.
@@ -866,6 +992,8 @@ Responsibilities:
 3. Template code is trusted internal code in v1, not untrusted tenant code.
 4. Webhook signatures must be mandatory.
 5. Customer file access must always be scoped by customer identity.
+6. Template source imports must pin an exact commit SHA before activation.
+7. Templates must include a `SKILL.md` file so customer AI agents have a framework-owned usage contract.
 
 ## Operational Notes for v1
 
@@ -875,6 +1003,18 @@ Responsibilities:
 4. Back up SQLite and treat it as transitional infrastructure.
 5. Store all large artifacts in S3, not on container disk.
 6. Assume template code is centrally reviewed before deployment.
+7. Not every template requires Remotion. Remotion validation should only apply to templates that actually call the Remotion adapter.
+
+## Template Certification Minimum
+
+A template must not be activatable unless all of the following pass:
+
+1. template metadata contract is valid
+2. operation-to-workflow references are valid
+3. a template-local `SKILL.md` file exists
+4. every operation defines a smoke-test payload
+5. the smoke-test harness passes
+6. the release is associated with a pinned commit SHA
 
 ## Known Limits of v1
 

package/README.md

@@ -2,15 +2,20 @@
 
 Vidfarm is a single-container Node.js control plane for async video template pipelines. It follows the architecture in [video-pipeline-architecture-draft.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/video-pipeline-architecture-draft.md) and [PLATFORM_SPEC.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/PLATFORM_SPEC.md): Hono API, SQLite-backed jobs and provider-key leasing, S3-compatible storage, and an in-process worker suitable for Docker on EC2.
 
+Production is now intended to run as one HTTPS host at `https://vidfarm.cloud.zoomgtm.com`, with the HTML console served from `/` and the API served under `/api/v1/...`.
+
 ## What Is Included
 
-- Async template API under `/templates/:templateId/*`
+- Unified frontend and backend on one host
+- Async template API under `/api/v1/templates/:templateId/*`
+- User auth/profile/provider-key routes under `/api/v1/user/*`
 - Email OTP login with API key issuance
+- `is_developer` user flag, assigned from allowlisted emails, required for template source submission
 - Customer-scoped provider key vault with encryption at rest
 - SQLite-backed job queue and provider-key lease coordination
 - Structured job logs, artifacts, billing events, and webhook delivery queue
 - Local storage mode plus S3-compatible mode for AWS
-- Remotion adapter seam and a sample `demo-template` slideshow pipeline
+- Remotion adapter seam and a sample `template_0000` slideshow pipeline
 
 ## Quick Start
 
@@ -28,7 +33,7 @@ For local template development, use the bundled CLI instead of manually wiring a
 
 ```bash
 npm install
-npx @officexapp/vidfarm-devcli dev --port 3310 --reset
+npx @mevdragon/vidfarm-devcli dev --port 3310 --reset
 ```
 
 What it does:
@@ -44,14 +49,26 @@ What it does:
 Useful commands:
 
 ```bash
-npx @officexapp/vidfarm-devcli dev --port 3310 --reset
-npx @officexapp/vidfarm-devcli session
+npx @mevdragon/vidfarm-devcli dev --port 3310 --reset
+npx @mevdragon/vidfarm-devcli session
+npx @mevdragon/vidfarm-devcli validate-template --template-id template_0000
+npx @mevdragon/vidfarm-devcli generate-template --slug-id template_0001 --template-dir ./templates/template_0001
+npx @mevdragon/vidfarm-devcli analyze-viral-dna --template-dir ./templates/template_0001
+npx @mevdragon/vidfarm-devcli analyze-visual-dna --template-dir ./templates/template_0001
 ```
 
 After install, the package also exposes `vidfarm` and `vidfarm-devcli` as local binaries.
 
 `vidfarm session` prints reusable auth headers and a sample `curl` request for the local REST API.
 
+`generate-template` now scaffolds a standard `research/` area for new templates:
+
+- `research/source_notes.md` for original-format notes
+- `research/preview/` for source screenshots or video
+- `src/template-dna.ts` as the checked-in metadata module that feeds `about.viral_dna`, `about.visual_dna`, and `about.link_to_original`
+
+If preview media is already present, or if you pass `--source-preview-dir`, scaffold generation will run both DNA analyzers automatically unless `--skip-dna-analysis` is set.
+
 ## Environment
 
 Primary variables:
@@ -59,15 +76,22 @@ Primary variables:
 - `PORT`
 - `VIDFARM_DB_PATH`
 - `VIDFARM_DATA_DIR`
+- `VIDFARM_ADMIN_EMAILS`
+- `TEMPLATE_SOURCE_ROOT`
 - `ENCRYPTION_SECRET`
 - `API_KEY_SALT`
 - `STORAGE_DRIVER=local|s3`
 - `AWS_REGION`
 - `AWS_S3_BUCKET`
+- `AWS_S3_PUBLIC_READ=true|false`
 - `PUBLIC_BASE_URL`
 - `RESEND_API_KEY`
 - `WEBHOOK_SECRET`
 - `MOCK_PROVIDER_RESPONSES=true|false`
+- `VIDFARM_DEVELOPER_EMAILS`
+- `WORKER_MAX_CONCURRENT_JOBS`
+- `MAX_PENDING_JOBS_GLOBAL`
+- `MAX_PENDING_JOBS_PER_CUSTOMER`
 
 Remotion-specific variables:
 
@@ -82,7 +106,7 @@ Remotion-specific variables:
 
 When `MOCK_PROVIDER_RESPONSES=true`, AI provider calls use the real key-leasing flow but return mocked content. This makes local end-to-end testing possible without burning customer API calls.
 
-When using `npx @officexapp/vidfarm-devcli dev`, the CLI automatically chooses local-friendly defaults:
+When using `npx @mevdragon/vidfarm-devcli dev`, the CLI automatically chooses local-friendly defaults:
 
 - `STORAGE_DRIVER=local`
 - `REMOTION_MODE=local`
@@ -93,6 +117,14 @@ When using `npx @officexapp/vidfarm-devcli dev`, the CLI automatically chooses l
 
 For shared Remotion AWS infra, use [AWS_REMOTION_HANDOFF.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/AWS_REMOTION_HANDOFF.md) as the operating contract.
 
+Shared Remotion AWS publish is an admin-only release step. Template developers may build and test locally, but they should not push directly to the shared production Remotion account. The intended promotion flow is:
+
+1. developer pushes template code to GitHub `production`
+2. admin reviews and pins a commit SHA
+3. admin publishes the approved Remotion site bundle
+4. admin activates the approved template release
+5. admin rebuilds and redeploys the production Docker image
+
 The key points verified on `2026-05-16` are:
 
 - region: `us-east-1`
@@ -122,13 +154,56 @@ If you wire real Lambda rendering into `src/services/remotion.ts`, preserve the
 
 ## API Flow
 
-1. `POST /auth/request-otp`
-2. `POST /auth/verify-otp`
-3. `POST /me/provider-keys`
-4. `POST /templates/demo-template/config`
-5. `POST /templates/demo-template/operations/generate`
-6. Poll `GET /templates/demo-template/jobs/:jobId`
-7. Stream timeline data from `GET /templates/demo-template/jobs/:jobId/logs`
+1. `POST /api/v1/user/request-otp`
+2. `POST /api/v1/user/verify-otp`
+3. `POST /api/v1/user/me/provider-keys`
+4. `GET /api/v1/templates/template_0000`
+5. `GET /api/v1/templates/template_0000/skill`
+6. `POST /api/v1/templates/template_0000/config`
+7. `POST /api/v1/templates/template_0000/operations/create_slideshow`
+8. Poll `GET /api/v1/templates/template_0000/jobs/:jobId`
+9. List or filter jobs with `GET /api/v1/templates/template_0000/jobs?tracer=...&start_time=...&end_time=...`
+10. Stream timeline data from `GET /api/v1/templates/template_0000/jobs/:jobId/logs?start_time=...&end_time=...`
+
+## Template Release Flow
+
+Templates can still live in this repo, but the platform now supports a GitHub-backed release workflow for in-house templates:
+
+1. developer keeps template code in GitHub
+2. the approved branch convention is `production`
+3. admin registers the template source with repo URL, module path, build command, and `SKILL.md` path
+4. admin imports a specific commit, or the current `production` head
+5. platform builds the repo, loads the template module, and runs certification
+6. only certified releases can be activated
+7. activation pins the live template to a specific commit SHA, not a floating branch head
+
+Template management endpoints:
+
+- `GET /api/v1/templates/sources`
+- `POST /api/v1/templates/sources`
+- `GET /api/v1/templates/releases`
+- `POST /api/v1/templates/sources/:sourceId/import`
+- `POST /api/v1/templates/releases/:releaseId/activate`
+
+`POST /api/v1/templates/sources` requires a user with `is_developer=true`. That flag is assigned from `VIDFARM_DEVELOPER_EMAILS` or `VIDFARM_ADMIN_EMAILS` during OTP verification. Registration now requires a self-issued UUIDv4 `template_id` plus a human-readable `slug_id`, and the platform rejects reused `template_id` values with an explicit conflict error. Import and activation remain admin-only via `VIDFARM_ADMIN_EMAILS`.
+
+## Template Certification
+
+Every template must have:
+
+- a `SKILL.md` file for customer AI agents
+- both `about.viral_dna` and `about.visual_dna`
+- at least one operation
+- valid workflow references
+- a `smokeTestPayload` for each operation
+
+Use the CLI locally before import/activation:
+
+```bash
+npx @mevdragon/vidfarm-devcli validate-template --template-id template_0000
+```
+
+Not every template needs Remotion. Certification only requires Remotion behavior if the template itself calls `ctx.remotion`.
 
 ## Docker
 
@@ -136,16 +211,90 @@ If you wire real Lambda rendering into `src/services/remotion.ts`, preserve the
 docker compose up --build
 ```
 
-For AWS/EC2, mount `/app/data` onto durable storage if SQLite is local, or switch artifacts to S3 with:
+For local Docker, `docker-compose.yml` already mounts `./data:/app/data`, which keeps SQLite across container restarts and rebuilds on the same machine.
+
+For production on EC2, the CDK stack mounts a dedicated EBS volume at `/var/lib/vidfarm` on the instance and binds it into the container as `/app/data`. That means the following survive shutdowns, reboots, and container updates:
+
+- `vidfarm.sqlite`
+- SQLite WAL/SHM files
+- imported template source checkouts
+- any local fallback runtime data under `/app/data`
+
+Artifacts and workspace objects should still live in S3 in production:
 
 ```env
 STORAGE_DRIVER=s3
 AWS_REGION=us-east-1
 AWS_S3_BUCKET=your-vidfarm-bucket
-PUBLIC_BASE_URL=https://your-domain.example
+AWS_S3_PUBLIC_READ=false
+PUBLIC_BASE_URL=https://vidfarm.cloud.zoomgtm.com
 ```
 
+Queue protection for small EC2 instances is now SQLite-backed and built into the app:
+
+- `WORKER_MAX_CONCURRENT_JOBS=1` by default
+- `WORKER_BATCH_SIZE=2` by default
+- `MAX_PENDING_JOBS_GLOBAL=0` by default
+- `MAX_PENDING_JOBS_PER_CUSTOMER=0` by default
+
+With the default `0` values, submissions are always accepted and persisted in SQLite. The worker simply drains the queue at the configured concurrency. Set either pending-job value above `0` only if you later decide you want a hard backlog cap.
+
+## AWS CDK Production
+
+The production CDK app lives under `infra/cdk` and provisions:
+
+- one S3 bucket for Vidfarm object storage
+- one public ALB with HTTP to HTTPS redirect
+- one ACM certificate for `vidfarm.cloud.zoomgtm.com`
+- one EC2 instance running the Docker image
+- one dedicated encrypted EBS data volume retained separately from the container
+- Route53 `A` and `AAAA` alias records for the production hostname
+
+Required shell variables before synth or deploy:
+
+```bash
+export CDK_DEFAULT_ACCOUNT=123456789012
+export CDK_DEFAULT_REGION=us-east-1
+export VIDFARM_ZONE_NAME=cloud.zoomgtm.com
+export VIDFARM_ZONE_ID=Z1234567890ABC
+export VIDFARM_RECORD_NAME=vidfarm
+export VIDFARM_PUBLIC_BASE_URL=https://vidfarm.cloud.zoomgtm.com
+export VIDFARM_ADMIN_EMAILS=admin@zoomgtm.com
+export VIDFARM_DEVELOPER_EMAILS=dev1@zoomgtm.com,dev2@zoomgtm.com
+export VIDFARM_ENCRYPTION_SECRET='replace-with-a-long-random-secret'
+export VIDFARM_API_KEY_SALT='replace-with-a-second-long-random-secret'
+export VIDFARM_WEBHOOK_SECRET='replace-with-a-third-long-random-secret'
+```
+
+Optional app env passthroughs are supported too, including:
+
+- `VIDFARM_RESEND_API_KEY`
+- `VIDFARM_RESEND_FROM_EMAIL`
+- `VIDFARM_OPENAI_API_KEY`
+- `VIDFARM_OPENROUTER_API_KEY`
+- `VIDFARM_GEMINI_API_KEY`
+- `VIDFARM_PERPLEXITY_API_KEY`
+- `VIDFARM_REMOTION_*`
+- `VIDFARM_INSTANCE_TYPE`
+- `VIDFARM_DATA_VOLUME_SIZE_GIB`
+- `VIDFARM_WORKER_MAX_CONCURRENT_JOBS`
+- `VIDFARM_MAX_PENDING_JOBS_GLOBAL`
+- `VIDFARM_MAX_PENDING_JOBS_PER_CUSTOMER`
+
+If you do not set `VIDFARM_INSTANCE_TYPE`, the CDK app now defaults to `t3.small`.
+
+Deploy flow:
+
+```bash
+npm install
+npm run cdk:synth
+npx aws-cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGION
+npm run cdk:deploy
+```
+
+The stack builds the Docker image locally, pushes it to an asset ECR repository, installs Docker on the EC2 host, mounts the retained EBS volume, writes the production env file, and runs the container as a systemd service.
+
 ## Notes
 
-- `demo-template` is the canonical sample internal template. Add more templates by registering them in [src/registry.ts](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/src/registry.ts).
+- `template_0000` is the canonical sample internal template slug. Its current sample `template_id` is `4c7a7e1a-7f35-4f30-9f86-9c8a63c7f2db`. Add more templates by registering them in [src/registry.ts](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/src/registry.ts).
 - Remotion is still stubbed behind [src/services/remotion.ts](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/src/services/remotion.ts). When upgrading it to real Lambda submission, follow [AWS_REMOTION_HANDOFF.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/AWS_REMOTION_HANDOFF.md) and use the Remotion-specific AWS credentials already available in local production env config.