@makefully/dutifully 1.22.1 → 1.23.1

package/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## Unreleased
 
+## 1.23.1
+
+### Added
+
+- Shared `auth` on jobs and `env-cheerfully.json` `auth.<service>` map (job wins; env fills defaults). Legacy `elevenLabsApiKey` and top-level ElevenLabs `apiKey` still map into `auth.apiKey`.
+
+## 1.23.0
+
+- Provider umbrellas: `translate` (`amazon` default → `amazontranslate`, or `deepl` / `google` / `microsoft`), `lipsync` (`rhubarb` default / `allosaurus`), and `voiceover` (`elevenlabs` default / `polly`) resolve to Cheerfully concrete services before Yap submit.
+- Concrete MT parsers: `amazontranslate`, `deepl`, `googletranslate`, `microsofttranslate`.
+
 ## 1.22.1
 
 - Allow translate `format: "catalog"` (Cheerfully flat `{lang}.json` app catalogs) alongside existing loc formats.

package/README.md

@@ -6,17 +6,28 @@ Client API for Cheerfully services. Cheerfully makes voice-over easy.
 npm install @makefully/dutifully
 ```
 
-To use the API, you will need to specify a root `./env-cheerfully.json` file with the following format:
+To use the API, you will need a root `./env-cheerfully.json` with Showfully Yap credentials and optional per-service auth defaults:
 
 ```javascript
 {
     "server": "http://server:port",
-    "accessToken": "BearerTokenForShowfullyYap"
+    "accessToken": "BearerTokenForShowfullyYap",
+    "auth": {
+        "elevenlabs": { "apiKey": "…" },
+        "deepl": { "apiKey": "…", "serverUrl": "https://api-free.deepl.com" },
+        "googletranslate": { "apiKey": "…" },
+        "microsofttranslate": { "key": "…", "region": "eastus" },
+        "amazontranslate": { "config": { } },
+        "polly": { "config": { }, "bucket": "…", "root": "…" },
+        "transcription": { "config": { }, "bucket": "…", "root": "…" }
+    }
 }
 ```
 
 `accessToken` is sent as `Authorization: Bearer …` on yap submit/status/download (Showfully 1.4+). For local Showfully, use the same value as `SHOWFULLY_DEBUG_TOKEN` with `SHOWFULLY_ALLOW_DEBUG_TOKEN=1`.
 
+Job listings may include their own `auth` object (same shapes). Dutifully merges `auth.<service>` from env into each job; **job `auth` wins**. Legacy `elevenLabsApiKey` still maps to `auth.elevenlabs.apiKey`.
+
 ## Services
 
 All services can be specified in `cheerfully.json` as described in the following sections. Services can be run as scripts specified in `package.json` as:
@@ -100,7 +111,7 @@ await cheerfully.say('The build is complete');
 ```
 
 
-Voice settings come from the first `elevenlabs` entry in `cheerfully.json` (same as other ElevenLabs chores). Server credentials and `elevenLabsApiKey` come from `env-cheerfully.json`. Cached files are keyed by voice + text, so repeats play immediately without regenerating.
+Voice settings come from the first `elevenlabs` entry in `cheerfully.json` (same as other ElevenLabs chores). Server credentials and optional `auth.elevenlabs` (or legacy `elevenLabsApiKey`) come from `env-cheerfully.json`. Cached files are keyed by voice + text, so repeats play immediately without regenerating.
 
 ### ElevenLabs
 
@@ -124,14 +135,20 @@ Specifying the `model` is optional: if unspecified, Cheerfully tries to look up
 
 Specifying `updateAllMetaData` is `false` by default. If it's set to `true`, album, title, and unsynchronized lyrics are appended to any MP3's in the source directory that were not generated.
 
-You may include your ElevenLabs API Key directly in the ElevenLabs spec above as `apiKey`, or set your API Key separately in `env-cheerfully.json`:
+You may include credentials on the job as `auth.apiKey`, or set defaults in `env-cheerfully.json`:
 
 ```javascript
 {
-    "elevenLabsApiKey": "elevenlabsapikeyforyouraccount"
+    "auth": {
+        "elevenlabs": {
+            "apiKey": "elevenlabsapikeyforyouraccount"
+        }
+    }
 }
 ```
 
+Legacy `elevenLabsApiKey` at the top level of `env-cheerfully.json` is still accepted.
+
 ### FFMPEG
 
 This audio file process accepts ffmpeg commandline arguments to perform operations on an audio file.
@@ -432,13 +449,22 @@ Script lines may also include an optional `events` array for one-shot timed mark
 
 Cheerfully resolves these against caption timings and exports them in parallel using the same caption `format` (`events.json`, `{id}.events.vtt` / `.srt` / `.sami` / `.smi` / `.lrc`, or a separate MP3 SYLT frame with content type `EVENTS` labeled `"events"`).
 
-### Translate
+### Provider umbrellas (`translate`, `lipsync`, `voiceover`)
+
+Dutifully logical services pick a Cheerfully processor via per-job `provider` (stripped before upload):
+
+| Dutifully `-service` | `provider` | Default | Cheerfully target |
+| --- | --- | --- | --- |
+| `translate` | `amazon`, `deepl`, `google`, `microsoft` | `amazon` | `amazontranslate`, `deepl`, `googletranslate`, `microsofttranslate` |
+| `lipsync` | `rhubarb`, `allosaurus` | `rhubarb` | same as provider |
+| `voiceover` | `polly`, `elevenlabs` | `elevenlabs` | same as provider |
 
-This text-to-text service translates Cheerfully scripts or language-matrix CSV/TSV via Amazon Translate. Set jobs in `cheerfully.json`:
+Concrete keys (`deepl`, `polly`, `rhubarb`, …) still work without `provider`.
 
 ```javascript
 {
     "translate": [{
+        "provider": "amazon",
         "script": "./example/script.json",
         "language": "en-US",
         "languages": ["es-ES", "fr-FR"],
@@ -449,15 +475,29 @@ This text-to-text service translates Cheerfully scripts or language-matrix CSV/T
         "language": "en-US",
         "languages": ["es-ES", "fr-FR"],
         "output": "./example/scripts/"
+    }],
+    "voiceover": [{
+        "provider": "elevenlabs",
+        "script": "./example/script.json",
+        "output": "./example/vo/en/",
+        "voice": "21m00Tcm4TlvDq8ikWAM"
+    }],
+    "lipsync": [{
+        "provider": "rhubarb",
+        "src": "./example/vo/en/",
+        "output": "./example/lipsync/en/"
     }]
 }
 ```
 
+Translate job fields:
+
 * `script` — path to a Cheerfully `.json` script or a language-matrix `.csv` / `.tsv` (first column = ids, first row = language tags).
 * `language` — source locale (CSV: source column). Matrix jobs require this; JSON may use `auto`.
 * `languages` — target locales to produce or fill.
 * `format` / `exports` — optional. Defaults to the source extension (`json`, `csv`, or `tsv`).
 * `ignore` — phrases left untranslated.
+* `provider` — MT engine (see table); default `amazon`.
 
 JSON sources write `{lang}/{basename}.{ext}` under `output` (`catalog` → `{lang}.json`). CSV/TSV sources write an updated matrix file and leave existing non-empty cells unchanged.
 

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@makefully/dutifully",
-    "version": "1.22.1",
+    "version": "1.23.1",
     "description": "Client tools for Cheerfully service.",
     "main": "./src/index.js",
     "bin": {
@@ -52,7 +52,10 @@
         "atlas",
         "spritesheet",
         "classfully",
-        "translate"
+        "translate",
+        "deepl",
+        "voiceover",
+        "lipsync"
     ],
     "author": "Derek Detweiler <derek@makefullystudios.com>",
     "contributors": [

package/src/classes/Cheer.js

@@ -6,6 +6,7 @@ const
     https = require('https'),
     {PassThrough} = require('stream'),
     unzipper = require('unzip-stream'),
+    {applyServiceAuth} = require('../helpers/applyServiceAuth'),
     cleanPath = (path) => path[path.length - 1] === '/' ? path.substring(0, path.length - 1) : path,
     isZipMagic = (buf) => buf && buf.length >= 4 && buf[0] === 0x50 && buf[1] === 0x4b &&
         ((buf[2] === 0x03 && buf[3] === 0x04) || (buf[2] === 0x05 && buf[3] === 0x06) || (buf[2] === 0x07 && buf[3] === 0x08)),
@@ -264,9 +265,9 @@ const
     },
     Cheer = class {
         constructor ({config, contents}) {
-            this.config = config;
             this.contents = contents;
             this.service = contents.service;
+            this.config = applyServiceAuth(config, contents, contents.service);
         }
 
         prepare ({difference = true, extract = true}) {

package/src/classes/ElevenLabs.js

@@ -4,14 +4,11 @@ const
 module.exports = class ElevenLabs extends TextToSpeech {
     constructor (data) {
         const
-            {id, elevenLabsApiKey} = data.contents;
+            {id} = data.contents;
 
         data.album = id;
         data.encodedBy = 'ElevenLabs';
-        if (elevenLabsApiKey) {
-            data.config.apiKey = elevenLabsApiKey;
-        }
-
+        // Auth (env auth.elevenlabs / elevenLabsApiKey / job.auth) is applied in Cheer.
         super(data);
     }
-};
+};

package/src/helpers/applyServiceAuth.js

@@ -0,0 +1,89 @@
+/**
+ * Merge Dutifully env-cheerfully.json `auth.<service>` into a job's `auth`.
+ * Job-level auth wins; env fills defaults. Legacy `elevenLabsApiKey` maps to elevenlabs.
+ */
+
+const isPlainObject = (value) => (
+    value != null && typeof value === 'object' && !Array.isArray(value)
+);
+
+/**
+ * Normalize legacy env aliases into `contents.auth`.
+ * @param {Record<string, unknown>} contents
+ * @returns {Record<string, unknown>}
+ */
+const normalizeContentsAuth = (contents = {}) => {
+    const
+        auth = isPlainObject(contents.auth) ? {...contents.auth} : {};
+
+    if (contents.elevenLabsApiKey) {
+        auth.elevenlabs = {
+            ...(isPlainObject(auth.elevenlabs) ? auth.elevenlabs : {}),
+            apiKey: contents.elevenLabsApiKey
+        };
+    }
+
+    return auth;
+};
+
+/**
+ * Apply service auth defaults onto a job config (mutates config).
+ * @param {Record<string, unknown>} config job listing from cheerfully.json
+ * @param {Record<string, unknown>} contents merged cheer + env + CLI
+ * @param {string} [service] Cheerfully service id (defaults to contents.service)
+ * @returns {Record<string, unknown>} config
+ */
+const applyServiceAuth = (config, contents = {}, service = contents.service) => {
+    if (!config || typeof config !== 'object') {
+        return config;
+    }
+
+    const
+        authMap = normalizeContentsAuth(contents),
+        fromEnv = service && isPlainObject(authMap[service]) ? {...authMap[service]} : {},
+        fromJob = isPlainObject(config.auth) ? {...config.auth} : {},
+        // Legacy ElevenLabs top-level apiKey on the job
+        legacyJobKey = service === 'elevenlabs'
+            && typeof config.apiKey === 'string'
+            && config.apiKey
+            && fromJob.apiKey == null
+            ? {apiKey: config.apiKey}
+            : {};
+
+    const
+        merged = {
+            ...fromEnv,
+            ...legacyJobKey,
+            ...fromJob
+        };
+
+    if (isPlainObject(fromEnv.config) || isPlainObject(fromJob.config) || isPlainObject(legacyJobKey.config)) {
+        merged.config = {
+            ...(isPlainObject(fromEnv.config) ? fromEnv.config : {}),
+            ...(isPlainObject(fromJob.config) ? fromJob.config : {})
+        };
+
+        if (isPlainObject(fromEnv.config?.credentials) || isPlainObject(fromJob.config?.credentials)) {
+            merged.config.credentials = {
+                ...(isPlainObject(fromEnv.config?.credentials) ? fromEnv.config.credentials : {}),
+                ...(isPlainObject(fromJob.config?.credentials) ? fromJob.config.credentials : {})
+            };
+        }
+    }
+
+    if (Object.keys(merged).length) {
+        config.auth = merged;
+    }
+
+    // Prefer auth container; drop legacy top-level apiKey once folded in.
+    if (service === 'elevenlabs' && config.apiKey != null && config.auth?.apiKey) {
+        delete config.apiKey;
+    }
+
+    return config;
+};
+
+module.exports = {
+    normalizeContentsAuth,
+    applyServiceAuth
+};

package/src/helpers/jobExports.js

@@ -28,6 +28,22 @@ const EXPORT_CATALOG = {
         keys: ['format', 'exports'],
         formats: TRANSLATE_FORMATS
     },
+    amazontranslate: {
+        keys: ['format', 'exports'],
+        formats: TRANSLATE_FORMATS
+    },
+    deepl: {
+        keys: ['format', 'exports'],
+        formats: TRANSLATE_FORMATS
+    },
+    googletranslate: {
+        keys: ['format', 'exports'],
+        formats: TRANSLATE_FORMATS
+    },
+    microsofttranslate: {
+        keys: ['format', 'exports'],
+        formats: TRANSLATE_FORMATS
+    },
     packfully: {
         keys: ['format', 'exports', 'exportFormat'],
         formats: PACKFULLY_FORMATS

package/src/helpers/resolveProviderService.js

@@ -0,0 +1,86 @@
+/**
+ * Map Dutifully umbrella services + provider → Cheerfully concrete service ids.
+ */
+
+const
+    UMBRELLAS = {
+        translate: {
+            defaultProvider: 'amazon',
+            providers: {
+                amazon: 'amazontranslate',
+                amazontranslate: 'amazontranslate',
+                deepl: 'deepl',
+                google: 'googletranslate',
+                googletranslate: 'googletranslate',
+                microsoft: 'microsofttranslate',
+                microsofttranslate: 'microsofttranslate'
+            }
+        },
+        lipsync: {
+            defaultProvider: 'rhubarb',
+            providers: {
+                rhubarb: 'rhubarb',
+                allosaurus: 'allosaurus'
+            }
+        },
+        voiceover: {
+            defaultProvider: 'elevenlabs',
+            providers: {
+                polly: 'polly',
+                elevenlabs: 'elevenlabs'
+            }
+        }
+    },
+    isUmbrella = (service) => Boolean(UMBRELLAS[service]),
+    /**
+     * @returns {{ cheerfullyService: string, provider: string, job: object }}
+     */
+    resolveProviderService = (umbrella, job = {}) => {
+        const
+            table = UMBRELLAS[umbrella];
+
+        if (!table) {
+            const
+                cleaned = {...job};
+
+            delete cleaned.provider;
+
+            return {
+                cheerfullyService: umbrella,
+                provider: null,
+                job: cleaned
+            };
+        }
+
+        const
+            raw = job.provider == null || job.provider === ''
+                ? table.defaultProvider
+                : String(job.provider).toLowerCase().trim(),
+            cheerfullyService = table.providers[raw];
+
+        if (!cheerfullyService) {
+            const
+                allowed = [...new Set(Object.keys(table.providers))].sort().join(', ');
+
+            throw new Error(
+                `Unknown provider "${job.provider}" for "${umbrella}". Use one of: ${allowed}.`
+            );
+        }
+
+        const
+            cleaned = {...job};
+
+        delete cleaned.provider;
+
+        return {
+            cheerfullyService,
+            provider: raw,
+            job: cleaned
+        };
+    };
+
+module.exports = {
+    UMBRELLAS,
+    isUmbrella,
+    resolveProviderService
+};

package/src/index.js

@@ -2,11 +2,16 @@
 /* global console, process, require */
 const
     LipSync = require('./classes/LipSync'),
+    Translate = require('./classes/Translate'),
     parsers = {
         allosaurus: LipSync,
+        amazontranslate: Translate,
         classfully: require('./classes/Classfully'),
+        deepl: Translate,
         elevenlabs: require('./classes/ElevenLabs'),
         ffmpeg: require('./classes/FFMPEG'),
+        googletranslate: Translate,
+        microsofttranslate: Translate,
         packfully: require('./classes/Packfully'),
         polly: require('./classes/Polly'),
         rasterize: require('./classes/Rasterize'),
@@ -14,10 +19,12 @@ const
         sharp: require('./classes/Sharp'),
         stackfully: require('./classes/Stackfully'),
         transcription: require('./classes/Transcription'),
-        translate: require('./classes/Translate')
+        // Concrete only — umbrellas (translate / lipsync / voiceover) resolve first.
+        translate: Translate
     },
     getJSON = require('./helpers/getJSON'),
     {normalizeOnly, filterJobsById} = require('./helpers/filterJobs'),
+    {isUmbrella, resolveProviderService} = require('./helpers/resolveProviderService'),
     send = async function (contents) {
         const
             {id, service} = contents,
@@ -36,11 +43,40 @@ const
                 continue;
             }
 
+            let
+                cheerfullyService = service,
+                jobConfig = config;
+
+            if (isUmbrella(service)) {
+                try {
+                    const
+                        resolved = resolveProviderService(service, config);
+
+                    cheerfullyService = resolved.cheerfullyService;
+                    jobConfig = resolved.job;
+                } catch (e) {
+                    console.warn(e.message);
+                    continue;
+                }
+            }
+
+            const
+                Parser = parsers[cheerfullyService];
+
+            if (!Parser) {
+                console.warn(`No Dutifully parser for Cheerfully service "${cheerfullyService}".`);
+                continue;
+            }
+
             const
-                serviceHandler = new parsers[service]({config, contents});
+                contentsForJob = {
+                    ...contents,
+                    service: cheerfullyService
+                },
+                serviceHandler = new Parser({config: jobConfig, contents: contentsForJob});
 
             try {
-                await serviceHandler.prepare({...contents, ...config});
+                await serviceHandler.prepare({...contentsForJob, ...jobConfig});
             } catch (e) {
                 console.warn(e.message);
                 continue;
@@ -48,10 +84,10 @@ const
 
             try {
                 await serviceHandler.send({
-                    instanceId: `${id}-${service}${length > 1 ? `-${index}` : ''}`
+                    instanceId: `${id}-${cheerfullyService}${length > 1 ? `-${index}` : ''}`
                 });
             } catch (e) {
-                console.warn(`Error running "${service}" (${index}): ${e.message || e}`);
+                console.warn(`Error running "${cheerfullyService}" (${index}): ${e.message || e}`);
             }
         }
     };