@mevdragon/vidfarm-devcli 0.2.1 → 0.2.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/.env.example +6 -39
  2. package/GETTING_STARTED.developers.md +87 -0
  3. package/README.md +94 -238
  4. package/SKILL.developer.md +430 -104
  5. package/dist/src/account-pages.js +1 -1
  6. package/dist/src/app.js +93 -5
  7. package/dist/src/cli.js +456 -8
  8. package/dist/src/config.js +3 -2
  9. package/dist/src/context.js +30 -11
  10. package/dist/src/db.js +2 -57
  11. package/dist/src/dev-app.js +0 -1
  12. package/dist/src/index.js +4 -2
  13. package/dist/src/lib/template-paths.js +21 -0
  14. package/dist/src/runtime.js +3 -1
  15. package/dist/src/services/auth.js +4 -4
  16. package/dist/src/services/job-logs.js +186 -0
  17. package/dist/src/services/jobs.js +3 -2
  18. package/dist/src/services/providers.js +14 -6
  19. package/dist/src/services/storage.js +85 -2
  20. package/dist/src/services/template-sources.js +29 -3
  21. package/dist/templates/template_0000/src/lib/images.js +46 -86
  22. package/dist/templates/template_0000/src/template.js +277 -53
  23. package/package.json +5 -6
  24. package/templates/template_0000/README.md +8 -52
  25. package/templates/template_0000/SKILL.md +35 -3
  26. package/templates/template_0000/package.json +3 -6
  27. package/templates/template_0000/src/lib/images.js +46 -86
  28. package/templates/template_0000/src/lib/images.ts +55 -98
  29. package/templates/template_0000/src/template-dna.js +9 -0
  30. package/templates/template_0000/src/template.js +523 -199
  31. package/templates/template_0000/src/template.ts +356 -61
  32. package/templates/template_0000/template.config.json +7 -12
  33. package/AWS_REMOTION_HANDOFF.md +0 -311
  34. package/PLATFORM_SPEC.md +0 -1039
  35. package/SKILL.director.md +0 -599
  36. package/dist/infra/cdk/bin/vidfarm-prod.js +0 -59
  37. package/dist/infra/cdk/lib/vidfarm-prod-stack.js +0 -212
  38. package/templates/template_0000/package-lock.json +0 -5505
  39. package/templates/template_0000/scripts/create-site.mjs +0 -27
  40. package/templates/template_0000/scripts/render-cloud.mjs +0 -72
package/.env.example CHANGED
@@ -1,48 +1,15 @@
1
1
  NODE_ENV="development"
2
2
  PORT="3000"
3
3
 
4
- VIDFARM_DB_PATH="./data/vidfarm.sqlite"
5
- VIDFARM_DATA_DIR="./data"
6
-
7
- ENCRYPTION_SECRET="replace-with-a-long-random-secret"
8
- API_KEY_SALT="replace-with-a-second-long-random-secret"
9
- WEBHOOK_SECRET="replace-with-a-third-long-random-secret"
10
-
11
- WORKER_POLL_MS="1500"
12
- WORKER_BATCH_SIZE="2"
13
- WORKER_MAX_CONCURRENT_JOBS="1"
14
- DEFAULT_JOB_DELAY_SECONDS="20"
15
- MAX_PENDING_JOBS_GLOBAL="0"
16
- MAX_PENDING_JOBS_PER_CUSTOMER="0"
17
-
18
- PUBLIC_BASE_URL="https://vidfarm.cloud.zoomgtm.com"
19
- VIDFARM_ADMIN_EMAILS="admin@example.com"
20
- VIDFARM_DEVELOPER_EMAILS="developer@example.com,operator@zoom-gtm.com,entropyinternetorg@gmail.com"
21
- TEMPLATE_SOURCE_ROOT="./data/template-sources"
22
- STORAGE_DRIVER="local"
23
- AWS_REGION="us-east-1"
24
- AWS_S3_BUCKET=""
25
- AWS_S3_PUBLIC_READ="false"
26
- AWS_S3_ENDPOINT=""
27
- AWS_ACCESS_KEY_ID=""
28
- AWS_SECRET_ACCESS_KEY=""
29
-
30
- RESEND_API_KEY=""
31
- RESEND_FROM_EMAIL="vidfarm@fwd.zoomgtm.com"
32
-
33
4
  OPENAI_API_KEY=""
34
5
  OPENROUTER_API_KEY=""
35
6
  GEMINI_API_KEY=""
36
7
  PERPLEXITY_API_KEY=""
37
8
 
38
- REMOTION_FUNCTION_NAME=""
39
- REMOTION_REGION="us-east-1"
40
- REMOTION_BUCKET_NAME=""
41
- REMOTION_SITE_NAME="vidfarm-template-0000"
42
- REMOTION_SERVE_URL=""
43
- REMOTION_COMPOSITION_ID="template-0000"
44
- REMOTION_AWS_ACCESS_KEY_ID=""
45
- REMOTION_AWS_SECRET_ACCESS_KEY=""
46
- REMOTION_MODE="auto"
9
+ # Optional when calling a hosted Vidfarm API directly.
10
+ VIDFARM_USER_ID=""
11
+ VIDFARM_API_KEY=""
47
12
 
48
- MOCK_PROVIDER_RESPONSES="true"
13
+ # Optional local overrides. The CLI will choose sensible local defaults.
14
+ PUBLIC_BASE_URL="http://localhost:3310"
15
+ REMOTION_MODE="local"
@@ -0,0 +1,87 @@
1
+ # Vidfarm Developers
2
+
3
+ ## Interview Phase
4
+
5
+ Prepare for your Interview:
6
+
7
+ 1. Browse the Winning Formats Gallery of viral TikTok videos here. These are samples of what you'd be recreating for customers:
8
+ https://winning-formats-gallery-tiktok.cloud.zoomgtm.com/
9
+
10
+ 2. Setup OpenAI Codex on your computer with free tier. This is your primary AI coding agent (really good to know, even for future jobs)
11
+ https://developers.openai.com/codex/cli
12
+ https://www.youtube.com/watch?v=GJsx_FuiwNg
13
+
14
+ 3. Setup Remotion Skill on your computer. Codex will handle all the coding work, you just need to be able to get it to render the output videos/images. no need for suno, just image gen is enough.
15
+ https://www.youtube.com/watch?v=Xdy1vkhSz-M
16
+
17
+ 4. You’ll need to acquire an ai api key that can generate images at least. There’s many free and paid ones out there. Part of the skills test is to be resourceful and find it so you can test with (hint: check huggingface, openrouter, and ask codex to suggest others). You will not be penalized for using a weaker free ai
18
+
19
+ 5. Once you are ready, request the interview with @zoom_gtm and he will give you the skills test, gemini api key, and invite to first week trial run hands on helping customers (you will be paid for your time). You'd be recreating from the winning formats gallery
20
+
21
+ The skills test:
22
+
23
+ try to build your own workflow in VSCode + Codex cli + prompt.md files to generate a winning template. It should be easy to re-run your template to generate infinite content of that type. You may ask codex to explain what I mean
24
+
25
+ ## Pre-Setup for Skills Test
26
+
27
+ Hello Creative Directors! You can
28
+
29
+ 1. Create a Github account and setup SSH key so you can save your work & login on github CLI (ask codex how):
30
+ https://github.com
31
+
32
+ 2. Create an NPM account and login on npm CLI (ask codex how):
33
+ https://www.npmjs.com
34
+
35
+ 3. DM me your github username & npm username. congrats! you are a developer now
36
+
37
+ 4. Make sure you are familiar with codex & api keys prepared and have tried all of the above. I will provide a gemini ai api key for the skills test.
38
+
39
+ MAKE SURE YOU ARE PREPARED. If I have to babysit you during the call, I will immediatlely end the interview early to not waste anyones time.
40
+
41
+ 5. Final step, schedule your 1:1 skills test interview. do not book time until you have finished prior steps.
42
+
43
+ https://calendly.com/zoom-gtm/creative-director-interviews
44
+
45
+ ## Setup for Job
46
+
47
+ 0. Tell your manager your email so he can whitelist you access to Vidfarm. After he confirms you have access, login at https://vidfarm.cloud.zoomgtm.com
48
+
49
+ 1. Open VSCode in a new project called "Vidfarm-Developer"
50
+
51
+ 2. Open terminal and login to github cli and npm cli (ask ai/codex if you need help). ask your manager if they gave you npm access to "vidfarm-devcli" yet.
52
+
53
+ 3. Open Codex cli in terminal, with /permissions set to "Full Access"
54
+
55
+ 4. Install the private skill by copying it from https://vidfarm.cloud.zoomgtm.com/settings in "Developer SKILL.md" button and pasting it into the below prompt for codex.
56
+
57
+ ```txt
58
+ Please take this agent skill and install it locally. Then install the vidfarm-devcli from npm, ask me for the otp if need, I will enter it. Here is the agent skill: [pasted]
59
+ ```
60
+
61
+ 5. Create a .env file at folder root with your AI API KEY. You need at least 1 of 3 and it doesn't matter which. We recommend Gemini. Ask your manager for an AI API key.
62
+
63
+ ```.env
64
+ GEMINI_API_KEY="______"
65
+ OPENAI_API_KEY="______"
66
+ OPENROUTER_API_KEY="______"
67
+ ```
68
+
69
+ 6. Extend that .env file with another secret VIDFARM_API_KEY, you can get it at https://vidfarm.cloud.zoomgtm.com/settings
70
+
71
+ ```.env
72
+ VIDFARM_API_KEY="_______"
73
+ ```
74
+
75
+ 6. Create a new folder called `src/`, this is where we are going to do most of our development work.
76
+
77
+ 7. Tell codex to:
78
+
79
+ ```txt
80
+ use the vidfarm-devcli to copy-reference-template into src/vidfarm_template_0000 so i can view it as example
81
+ ```
82
+
83
+ That will populate with a reference folder of what a template code looks like. We are going to create a new template.
84
+
85
+ 8. Create a new folder beside src called `drafts/` which is where we will put messy files when needed.
86
+
87
+ 9. Now find an easy winning format from https://winning-formats-gallery-tiktok.cloud.zoomgtm.com/ and save the image to your drafts/ folder. then tell Codex to look at that image (right click, copy relative path, paste to codex), then tell codex please make a new template based on the image.
package/README.md CHANGED
@@ -1,300 +1,156 @@
1
- # Vidfarm
1
+ # Vidfarm Dev CLI
2
2
 
3
- Vidfarm is a single-container Node.js control plane for async video template pipelines. It follows the architecture in [video-pipeline-architecture-draft.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/video-pipeline-architecture-draft.md) and [PLATFORM_SPEC.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/PLATFORM_SPEC.md): Hono API, SQLite-backed jobs and provider-key leasing, S3-compatible storage, and an in-process worker suitable for Docker on EC2.
3
+ `@mevdragon/vidfarm-devcli` is the local development toolkit for third-party Vidfarm template authors.
4
4
 
5
- Production is now intended to run as one HTTPS host at `https://vidfarm.cloud.zoomgtm.com`, with the HTML console served from `/` and the API served under `/api/v1/...`.
5
+ This package is intentionally scoped to:
6
6
 
7
- ## What Is Included
7
+ - building templates under `templates/*`
8
+ - running the local Vidfarm API and worker
9
+ - testing template operations as async jobs
10
+ - validating template metadata, skills, and smoke test payloads
11
+ - rendering Remotion locally when a template uses Remotion
8
12
 
9
- - Unified frontend and backend on one host
10
- - Async template API under `/api/v1/templates/:templateId/*`
11
- - User auth/profile/provider-key routes under `/api/v1/user/*`
12
- - Email OTP login with API key issuance
13
- - `is_developer` user flag, assigned from allowlisted emails, required for template source submission
14
- - Customer-scoped provider key vault with encryption at rest
15
- - SQLite-backed job queue and provider-key lease coordination
16
- - Structured job logs, artifacts, billing events, and webhook delivery queue
17
- - Local storage mode plus S3-compatible mode for AWS
18
- - Remotion adapter seam and a sample `template_0000` slideshow pipeline
13
+ It is intentionally not documentation for platform internals, production deployment, admin release flow, or shared cloud infrastructure.
19
14
 
20
15
  ## Quick Start
21
16
 
22
- ```bash
23
- npm install
24
- cp .env.example .env
25
- npm run dev
26
- ```
27
-
28
- The service starts on `http://localhost:3000` by default and creates SQLite plus local artifact storage under `./data`.
29
-
30
- ## Template Developer CLI
31
-
32
- For local template development, use the bundled CLI instead of manually wiring auth, provider keys, SQLite, and Remotion:
17
+ 1. Create a local `.env` from the example file.
18
+ 2. Add at least one AI provider key.
19
+ 3. Start the local CLI runtime.
33
20
 
34
21
  ```bash
35
- npm install
22
+ cp .env.example .env
36
23
  npx @mevdragon/vidfarm-devcli dev --port 3310 --reset
37
24
  ```
38
25
 
39
- What it does:
40
-
41
- - starts the same REST API and in-process worker as production
42
- - provisions a local SQLite database automatically
43
- - uses local filesystem storage as an S3 stand-in
44
- - seeds a dev customer and API key
45
- - seeds provider keys from local `.env` variables like `OPENAI_API_KEY`, `GEMINI_API_KEY`, and `OPENROUTER_API_KEY`
46
- - falls back to mock provider keys plus mocked AI responses when no env keys are present
47
- - renders Remotion compositions locally instead of using Lambda
48
-
49
26
  Useful commands:
50
27
 
51
28
  ```bash
52
29
  npx @mevdragon/vidfarm-devcli dev --port 3310 --reset
53
30
  npx @mevdragon/vidfarm-devcli session
54
31
  npx @mevdragon/vidfarm-devcli validate-template --template-id template_0000
55
- npx @mevdragon/vidfarm-devcli generate-template --slug-id template_0001 --template-dir ./templates/template_0001
56
- npx @mevdragon/vidfarm-devcli analyze-viral-dna --template-dir ./templates/template_0001
57
- npx @mevdragon/vidfarm-devcli analyze-visual-dna --template-dir ./templates/template_0001
32
+ npx @mevdragon/vidfarm-devcli copy-reference-template --template-dir ./src/vidfarm_template_0000
33
+ npx @mevdragon/vidfarm-devcli generate-template --slug-id template_0001 --template-dir ./templates/vidfarm_template_0001
34
+ npx @mevdragon/vidfarm-devcli analyze-viral-dna --template-dir ./templates/vidfarm_template_0001
35
+ npx @mevdragon/vidfarm-devcli analyze-visual-dna --template-dir ./templates/vidfarm_template_0001
36
+ npx @mevdragon/vidfarm-devcli presign-preview-media --file ./drafts/preview/example.png --directory drafts/preview
58
37
  ```
59
38
 
60
- After install, the package also exposes `vidfarm` and `vidfarm-devcli` as local binaries.
61
-
62
- `vidfarm session` prints reusable auth headers and a sample `curl` request for the local REST API.
63
-
64
- `generate-template` now scaffolds a standard `research/` area for new templates:
65
-
66
- - `research/source_notes.md` for original-format notes
67
- - `research/preview/` for source screenshots or video
68
- - `src/template-dna.ts` as the checked-in metadata module that feeds `about.viral_dna`, `about.visual_dna`, and `about.link_to_original`
69
-
70
- If preview media is already present, or if you pass `--source-preview-dir`, scaffold generation will run both DNA analyzers automatically unless `--skip-dna-analysis` is set.
71
-
72
- ## Environment
73
-
74
- Primary variables:
75
-
76
- - `PORT`
77
- - `VIDFARM_DB_PATH`
78
- - `VIDFARM_DATA_DIR`
79
- - `VIDFARM_ADMIN_EMAILS`
80
- - `TEMPLATE_SOURCE_ROOT`
81
- - `ENCRYPTION_SECRET`
82
- - `API_KEY_SALT`
83
- - `STORAGE_DRIVER=local|s3`
84
- - `AWS_REGION`
85
- - `AWS_S3_BUCKET`
86
- - `AWS_S3_PUBLIC_READ=true|false`
87
- - `PUBLIC_BASE_URL`
88
- - `RESEND_API_KEY`
89
- - `WEBHOOK_SECRET`
90
- - `MOCK_PROVIDER_RESPONSES=true|false`
91
- - `VIDFARM_DEVELOPER_EMAILS`
92
- - `WORKER_MAX_CONCURRENT_JOBS`
93
- - `MAX_PENDING_JOBS_GLOBAL`
94
- - `MAX_PENDING_JOBS_PER_CUSTOMER`
95
-
96
- Remotion-specific variables:
97
-
98
- - `REMOTION_REGION`
99
- - `REMOTION_FUNCTION_NAME`
100
- - `REMOTION_BUCKET_NAME`
101
- - `REMOTION_SITE_NAME`
102
- - `REMOTION_SERVE_URL`
103
- - `REMOTION_COMPOSITION_ID`
104
- - `REMOTION_AWS_ACCESS_KEY_ID`
105
- - `REMOTION_AWS_SECRET_ACCESS_KEY`
106
-
107
- When `MOCK_PROVIDER_RESPONSES=true`, AI provider calls use the real key-leasing flow but return mocked content. This makes local end-to-end testing possible without burning customer API calls.
108
-
109
- When using `npx @mevdragon/vidfarm-devcli dev`, the CLI automatically chooses local-friendly defaults:
110
-
111
- - `STORAGE_DRIVER=local`
112
- - `REMOTION_MODE=local`
113
- - a local SQLite path under `.vidfarm/local`
114
- - `MOCK_PROVIDER_RESPONSES=false` if real provider env keys are present, otherwise `true`
115
-
116
- ## Remotion AWS
39
+ After install, the package exposes both `vidfarm` and `vidfarm-devcli` as local binaries.
117
40
 
118
- For shared Remotion AWS infra, use [AWS_REMOTION_HANDOFF.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/AWS_REMOTION_HANDOFF.md) as the operating contract.
41
+ ## What The Local CLI Does
119
42
 
120
- Shared Remotion AWS publish is an admin-only release step. Template developers may build and test locally, but they should not push directly to the shared production Remotion account. The intended promotion flow is:
43
+ The local CLI runs the same template contract used by the hosted platform, but with local-first defaults:
121
44
 
122
- 1. developer pushes template code to GitHub `production`
123
- 2. admin reviews and pins a commit SHA
124
- 3. admin publishes the approved Remotion site bundle
125
- 4. admin activates the approved template release
126
- 5. admin rebuilds and redeploys the production Docker image
45
+ - local SQLite state under `.vidfarm/local`
46
+ - local filesystem storage instead of cloud object storage
47
+ - local Remotion rendering
48
+ - a seeded developer session for local testing
49
+ - provider keys sourced from your local `.env`
50
+ - mocked provider responses only when no real provider keys are present
127
51
 
128
- The key points verified on `2026-05-16` are:
52
+ Use `vidfarm session` to print reusable auth headers and a sample `curl` request for the local REST API.
129
53
 
130
- - region: `us-east-1`
131
- - shared Lambda function: `remotion-render-4-0-355-mem2048mb-disk2048mb-180sec`
132
- - shared bucket: `remotionlambda-useast1-ujg7c0h43q`
133
- - Remotion package version to pin: `4.0.355`
54
+ For hosted preview-media uploads, use `vidfarm-devcli presign-preview-media`. It calls the authenticated Vidfarm API with `VIDFARM_API_KEY` and returns a presigned PUT URL under `developer/<user_id>/preview_media/*` plus the resulting Vidfarm media URL.
134
55
 
135
- This repo also has separate Remotion AWS runner credentials available in local production env config via:
56
+ ## What The Hosted Platform Expects From Templates
136
57
 
137
- - `REMOTION_AWS_ACCESS_KEY_ID`
138
- - `REMOTION_AWS_SECRET_ACCESS_KEY`
58
+ Vidfarm treats templates as async job producers, not as ad hoc endpoints.
139
59
 
140
- Use those for Remotion site publish/render work rather than assuming the generic app AWS credentials are the right identity.
60
+ Each template should provide:
141
61
 
142
- Recommended repo-level config for real Remotion work:
143
-
144
- ```env
145
- REMOTION_REGION=us-east-1
146
- REMOTION_FUNCTION_NAME=remotion-render-4-0-355-mem2048mb-disk2048mb-180sec
147
- REMOTION_BUCKET_NAME=remotionlambda-useast1-ujg7c0h43q
148
- REMOTION_SITE_NAME=your-site-name
149
- REMOTION_SERVE_URL=https://remotionlambda-useast1-ujg7c0h43q.s3.us-east-1.amazonaws.com/sites/your-site-name/index.html
150
- REMOTION_COMPOSITION_ID=YourComposition
151
- ```
152
-
153
- If you wire real Lambda rendering into `src/services/remotion.ts`, preserve the current platform contract and make the service read these env vars rather than hard-coding one-off values.
154
-
155
- ## API Flow
156
-
157
- 1. `POST /api/v1/user/request-otp`
158
- 2. `POST /api/v1/user/verify-otp`
159
- 3. `POST /api/v1/user/me/provider-keys`
160
- 4. `GET /api/v1/templates/template_0000`
161
- 5. `GET /api/v1/templates/template_0000/skill`
162
- 6. `POST /api/v1/templates/template_0000/config`
163
- 7. `POST /api/v1/templates/template_0000/operations/create_slideshow`
164
- 8. Poll `GET /api/v1/templates/template_0000/jobs/:jobId`
165
- 9. List or filter jobs with `GET /api/v1/templates/template_0000/jobs?tracer=...&start_time=...&end_time=...`
166
- 10. Stream timeline data from `GET /api/v1/templates/template_0000/jobs/:jobId/logs?start_time=...&end_time=...`
167
-
168
- ## Template Release Flow
169
-
170
- Templates can still live in this repo, but the platform now supports a GitHub-backed release workflow for in-house templates:
171
-
172
- 1. developer keeps template code in GitHub
173
- 2. the approved branch convention is `production`
174
- 3. admin registers the template source with repo URL, module path, build command, and `SKILL.md` path
175
- 4. admin imports a specific commit, or the current `production` head
176
- 5. platform builds the repo, loads the template module, and runs certification
177
- 6. only certified releases can be activated
178
- 7. activation pins the live template to a specific commit SHA, not a floating branch head
62
+ - stable template metadata
63
+ - a `SKILL.md` file for users and agents
64
+ - a TypeScript entrypoint at `src/template.ts`
65
+ - one or more named operations
66
+ - workflow functions behind those operations
67
+ - a `smokeTestPayload` for each operation
68
+ - `about.viral_dna` and `about.visual_dna`
179
69
 
180
- Template management endpoints:
70
+ At runtime, the platform uses the template wrapper to expose a consistent API shape:
181
71
 
182
- - `GET /api/v1/templates/sources`
183
- - `POST /api/v1/templates/sources`
184
- - `GET /api/v1/templates/releases`
185
- - `POST /api/v1/templates/sources/:sourceId/import`
186
- - `POST /api/v1/templates/releases/:releaseId/activate`
72
+ - `GET /api/v1/templates/:templateId`
73
+ - `GET /api/v1/templates/:templateId/skill`
74
+ - `POST /api/v1/templates/:templateId/config`
75
+ - `POST /api/v1/templates/:templateId/operations/:operationName`
76
+ - `GET /api/v1/templates/:templateId/jobs/:jobId`
77
+ - `GET /api/v1/templates/:templateId/jobs/:jobId/logs`
187
78
 
188
- `POST /api/v1/templates/sources` requires a user with `is_developer=true`. That flag is assigned from `VIDFARM_DEVELOPER_EMAILS` or `VIDFARM_ADMIN_EMAILS` during OTP verification. Registration now requires a self-issued UUIDv4 `template_id` plus a human-readable `slug_id`, and the platform rejects reused `template_id` values with an explicit conflict error. Import and activation remain admin-only via `VIDFARM_ADMIN_EMAILS`.
79
+ Operations should return work through the job system rather than inventing one-off synchronous routes.
189
80
 
190
- ## Template Certification
81
+ ## Approval Flow
191
82
 
192
- Every template must have:
83
+ Third-party developers build and validate templates locally first.
193
84
 
194
- - a `SKILL.md` file for customer AI agents
195
- - both `about.viral_dna` and `about.visual_dna`
196
- - at least one operation
197
- - valid workflow references
198
- - a `smokeTestPayload` for each operation
85
+ After handoff, a Vidfarm admin reviews the template before it is made available on the hosted platform. Template authors should assume there is an approval step between local development and production availability.
199
86
 
200
- Use the CLI locally before import/activation:
87
+ ## Template Deploy Cycle
201
88
 
202
- ```bash
203
- npx @mevdragon/vidfarm-devcli validate-template --template-id template_0000
204
- ```
89
+ When someone says "template deploy cycle" in this repo, they mean this exact sequence:
205
90
 
206
- Not every template needs Remotion. Certification only requires Remotion behavior if the template itself calls `ctx.remotion`.
91
+ 1. commit and `git push` the repo that contains the template folder, such as `templates/vidfarm_template_<slug>/` inside a monorepo or a standalone template repo
92
+ 2. as the platform admin, pull/import that template commit into Vidfarm
93
+ 3. approve and activate the imported template release in the hosted platform
94
+ 4. run the prod deploy script if platform-runtime code changed, or reuse the current prod image if the change was template-only and you just need a formal deploy-cycle restart
207
95
 
208
- ## Docker
96
+ For `template_0000`, that usually means:
209
97
 
210
98
  ```bash
211
- docker compose up --build
99
+ git -C templates/vidfarm_template_0000 push origin production
100
+ # then run the platform operator command
101
+ node dist/src/cli.js deploy-template-cycle \
102
+ --env-file .env.production \
103
+ --stack-name VidfarmProdStack \
104
+ --template-id 4c7a7e1a-7f35-4f30-9f86-9c8a63c7f2db \
105
+ --slug-id template_0000 \
106
+ --repo-url https://github.com/your-org/your-template-repo \
107
+ --branch production \
108
+ --template-module-path templates/vidfarm_template_0000/src/template.ts
212
109
  ```
213
110
 
214
- For local Docker, `docker-compose.yml` already mounts `./data:/app/data`, which keeps SQLite across container restarts and rebuilds on the same machine.
111
+ The important distinction is:
215
112
 
216
- For production on EC2, the CDK stack mounts a dedicated EBS volume at `/var/lib/vidfarm` on the instance and binds it into the container as `/app/data`. That means the following survive shutdowns, reboots, and container updates:
113
+ - template repo changes ship through template source import and activation
114
+ - platform code changes ship through the root prod deploy script
115
+ - multiple templates can live in one git repo as long as each template has its own folder and its `template_module_path` points at that folder's `src/template.ts`
217
116
 
218
- - `vidfarm.sqlite`
219
- - SQLite WAL/SHM files
220
- - imported template source checkouts
221
- - any local fallback runtime data under `/app/data`
117
+ ## Authentication Context
222
118
 
223
- Artifacts and workspace objects should still live in S3 in production:
119
+ When calling a hosted Vidfarm API directly, authenticated requests use:
224
120
 
225
- ```env
226
- STORAGE_DRIVER=s3
227
- AWS_REGION=us-east-1
228
- AWS_S3_BUCKET=your-vidfarm-bucket
229
- AWS_S3_PUBLIC_READ=false
230
- PUBLIC_BASE_URL=https://vidfarm.cloud.zoomgtm.com
231
- ```
121
+ - `vidfarm-user-id`
122
+ - `vidfarm-api-key`
232
123
 
233
- Queue protection for small EC2 instances is now SQLite-backed and built into the app:
124
+ The API key comes from Vidfarm login and settings, not from platform infrastructure secrets.
125
+ If you need to discover the authenticated customer ID first, resolve it from `GET /api/v1/user/me` and then send both headers on subsequent requests.
234
126
 
235
- - `WORKER_MAX_CONCURRENT_JOBS=1` by default
236
- - `WORKER_BATCH_SIZE=2` by default
237
- - `MAX_PENDING_JOBS_GLOBAL=0` by default
238
- - `MAX_PENDING_JOBS_PER_CUSTOMER=0` by default
127
+ For hosted template source registration, treat `template_module_path` as repo-relative and point it at the template's TypeScript entrypoint, for example `templates/vidfarm_template_hooks/src/template.ts`. Each template folder must start with `vidfarm_template_`. The import flow builds the repo and loads the compiled `src/template.js` sibling at activation time.
239
128
 
240
- With the default `0` values, submissions are always accepted and persisted in SQLite. The worker simply drains the queue at the configured concurrency. Set either pending-job value above `0` only if you later decide you want a hard backlog cap.
129
+ For private GitHub repos, export `VIDFARM_GITHUB_TOKEN` (or `GITHUB_TOKEN`) before running `import-source-prod` or `deploy-template-cycle`. The prod operator command injects that token into the live container's git config for the import and keeps the stored `repo_url` clean.
241
130
 
242
- ## AWS CDK Production
131
+ For local CLI usage, the `session` command gives you a local seeded session so you can test without manually wiring auth state.
243
132
 
244
- The production CDK app lives under `infra/cdk` and provisions:
133
+ ## Provider Keys
245
134
 
246
- - one S3 bucket for Vidfarm object storage
247
- - one public ALB with HTTP to HTTPS redirect
248
- - one ACM certificate for `vidfarm.cloud.zoomgtm.com`
249
- - one EC2 instance running the Docker image
250
- - one dedicated encrypted EBS data volume retained separately from the container
251
- - Route53 `A` and `AAAA` alias records for the production hostname
135
+ Templates should work with the developer's own upstream AI keys.
252
136
 
253
- Required shell variables before synth or deploy:
137
+ Typical local `.env`:
254
138
 
255
- ```bash
256
- export CDK_DEFAULT_ACCOUNT=123456789012
257
- export CDK_DEFAULT_REGION=us-east-1
258
- export VIDFARM_ZONE_NAME=cloud.zoomgtm.com
259
- export VIDFARM_ZONE_ID=Z1234567890ABC
260
- export VIDFARM_RECORD_NAME=vidfarm
261
- export VIDFARM_PUBLIC_BASE_URL=https://vidfarm.cloud.zoomgtm.com
262
- export VIDFARM_ADMIN_EMAILS=admin@zoomgtm.com
263
- export VIDFARM_DEVELOPER_EMAILS=dev1@zoomgtm.com,dev2@zoomgtm.com
264
- export VIDFARM_ENCRYPTION_SECRET='replace-with-a-long-random-secret'
265
- export VIDFARM_API_KEY_SALT='replace-with-a-second-long-random-secret'
266
- export VIDFARM_WEBHOOK_SECRET='replace-with-a-third-long-random-secret'
139
+ ```env
140
+ GEMINI_API_KEY=""
141
+ OPENAI_API_KEY=""
142
+ OPENROUTER_API_KEY=""
143
+ PERPLEXITY_API_KEY=""
267
144
  ```
268
145
 
269
- Optional app env passthroughs are supported too, including:
270
-
271
- - `VIDFARM_RESEND_API_KEY`
272
- - `VIDFARM_RESEND_FROM_EMAIL`
273
- - `VIDFARM_OPENAI_API_KEY`
274
- - `VIDFARM_OPENROUTER_API_KEY`
275
- - `VIDFARM_GEMINI_API_KEY`
276
- - `VIDFARM_PERPLEXITY_API_KEY`
277
- - `VIDFARM_REMOTION_*`
278
- - `VIDFARM_INSTANCE_TYPE`
279
- - `VIDFARM_DATA_VOLUME_SIZE_GIB`
280
- - `VIDFARM_WORKER_MAX_CONCURRENT_JOBS`
281
- - `VIDFARM_MAX_PENDING_JOBS_GLOBAL`
282
- - `VIDFARM_MAX_PENDING_JOBS_PER_CUSTOMER`
283
-
284
- If you do not set `VIDFARM_INSTANCE_TYPE`, the CDK app now defaults to `t3.small`.
146
+ You usually only need one provider key to develop and test a template locally.
285
147
 
286
- Deploy flow:
148
+ ## Remotion
287
149
 
288
- ```bash
289
- npm install
290
- npm run cdk:synth
291
- npx aws-cdk bootstrap aws://$CDK_DEFAULT_ACCOUNT/$CDK_DEFAULT_REGION
292
- npm run cdk:deploy
293
- ```
150
+ For third-party template authors, Remotion is local.
294
151
 
295
- The stack builds the Docker image locally, pushes it to an asset ECR repository, installs Docker on the EC2 host, mounts the retained EBS volume, writes the production env file, and runs the container as a systemd service.
152
+ If your template uses `ctx.remotion`, develop and validate it locally through the CLI. This package does not document or require shared cloud Remotion infrastructure.
296
153
 
297
- ## Notes
154
+ ## Next Read
298
155
 
299
- - `template_0000` is the canonical sample internal template slug. Its current sample `template_id` is `4c7a7e1a-7f35-4f30-9f86-9c8a63c7f2db`. Add more templates by registering them in [src/registry.ts](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/src/registry.ts).
300
- - Remotion is still stubbed behind [src/services/remotion.ts](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/src/services/remotion.ts). When upgrading it to real Lambda submission, follow [AWS_REMOTION_HANDOFF.md](/Users/localghost/Projects/OfficeX/OfficeX/ZoomGTM/vidfarm/AWS_REMOTION_HANDOFF.md) and use the Remotion-specific AWS credentials already available in local production env config.
156
+ For the external template-author workflow, continue with [GETTING_STARTED.developers.md](./GETTING_STARTED.developers.md) and the example template in [`templates/template_0000`](./templates/template_0000).