ugcinc 4.21.0 → 4.23.0

package/README.md

@@ -362,3 +362,39 @@ than caching by URL. Results include source SHA-256, encoded/display dimensions,
 rotation, time base, source starting PTS, decoded frame count and an owned manifest.
 Video URLs must be public HTTPS sources. Limits: 100 MB, 30 minutes, 4096 pixels per
 encoded dimension, 120,000 decoded frames. Unsupported/malformed media fails explicitly.
+
+### Image and video OCR
+
+Quick reference: `client.media.readImageText({url})`, `client.media.readVideoText({url, mode, range?, intervalMs?})`; agent tools `read_image_text` and `read_video_text`. Both use the authenticated analysis job/status flow above.
+
+```ts
+const imageText = await client.media.readImageText({url: imageUrl});
+const sampledText = await client.media.readVideoText({url: videoUrl, mode: 'fast', intervalMs: 1000});
+const everyFrame = await client.media.readVideoText({url: videoUrl, mode: 'frame_scan', range: {startMs: 0, endMs: 5000}});
+```
+
+`POST /media/analysis/read-image-text` accepts an HTTPS image URL. Results contain `regions` (raw recognized text, four-point polygon, enclosing box, model confidence, readable/unreadable status), inferred `lines` with region IDs and reading order, source dimensions/hash, model details, `previewUrl` and `manifestUrl`. Coordinates are pixels after EXIF orientation; transparent pixels are composited on white. English recognition only; no guaranteed reading order for columns or inverted text. Low-confidence candidates retain `rawText` but return `text: null`. Undetected text cannot be marked unreadable, and empty output does not prove absence. Images must be static, at most 4096 pixels per axis /16 megapixels, and at most 100 MiB.
+
+`POST /media/analysis/read-video-text` accepts a video URL, `mode` (`fast` default or `frame_scan`), optional `{startMs,endMs}` range and optional `intervalMs` (fast only, default1000, minimum100). Range end is exclusive; range defaults to the whole clip. A request may cover at most120 seconds and inspect at most240 frames; split longer frame scans into ranges. Invalid/excessive requests fail explicitly, never silently truncate. Times are relative to the first displayed frame; original `sourcePts` and time base are retained. The frame displayed at a partial range's start is included, so its source timestamp can precede `startMs`.
+
+Results include per-frame regions/lines, up to eight annotated frame previews, heuristic text `tracks` with observed boxes, and `coverage.inspected` plus `coverage.uninspected`. `coverage.complete` means all displayed frames within the range were inspected, not that OCR found every word. A track's `firstObservedMs` / `lastObservedEndMs` bounds observations; it does not prove uninterrupted visibility between sampled frames. Fast mode can miss brief text entirely. Fully scanned recognized single-frame text retains its decoded frame interval. Recognition scores are not calibrated probabilities. Track identity, scene-change rejection and line grouping are estimates; use individual observations and polygons for evidence.
+
+Both methods return `ApiResponse<MediaAnalysisJob>`, waiting through up to180 two-second status polls. A pending/processing response retains `jobId`; resume with `getMediaAnalysis({jobId})`. Job results include model/runtime versions and SHA-256 checksums. These CPU jobs share the existing bounded analysis queue. A failed job does not return partial OCR as complete.
+
+### Local speech transcription and audio levels
+
+Quick reference: `client.media.transcribeAudio({url})`, `client.media.measureAudioLevels({url})`; agent tools `transcribe_audio`, `measure_audio_levels`. Both use the authenticated media-analysis job/status flow.
+
+```ts
+const transcript = await client.media.transcribeAudio({url: videoUrl, language: 'en', timing: 'word'});
+const levels = await client.media.measureAudioLevels({url: videoUrl, windowMs: 100, silenceThresholdDbfs: -50});
+const excerpt = await client.media.transcribeAudio({url: videoUrl, range: {startMs: 5000, endMs: 15000}, channel: 0});
+```
+
+`POST /media/analysis/transcribe-audio` runs an owned CPU int8 Whisper Small model through faster-whisper. There is no hosted transcription call. Inputs: HTTPS audio/video URL; optional range, `timing: 'word' | 'segment'` (default word), language (auto default; en/es/fr/de/it/pt/ja/ko/zh/ar/hi/ru), and channel0 or1. Without a channel, native channels are arithmetically averaged; opposite-phase stereo can cancel. Results contain raw recognized text, segments/words with source-relative millisecond timestamps, model scores, detected language, pinned model revision/hashes and a mono16kHz WAV preview. Segment and word timings are explicitly **estimated**, not forced alignment. Segment mode leaves words empty and word timing unavailable. Speaker labels are unavailable; overlapping voices are not separated. Empty output means no speech recognized, not proof of no speech. Silence and files without audio return explicit successful statuses.
+
+`POST /media/analysis/measure-audio-levels` uses native channels/sample rate. Inputs: URL; optional range, `windowMs`20–5000 (default100), `silenceThresholdDbfs`−100 to−10 (default−50). Results contain per-channel RMS/sample peak levels in dBFS, near-full-scale sample counts, estimated true peaks in dBTP, integrated LUFS, a100ms momentary/short-term LUFS timeline, measured silence ranges and a waveform preview. Silence means all channels' window RMS is below the threshold; boundaries have window resolution. Near-full-scale samples (absolute amplitude>=1−1/32768) indicate possible clipping, not proven distortion. `intersampleOvershoot` distinguishes a true-peak estimate above0dBTP while sample peaks remain below0dBFS. Zero levels, gated/too-short loudness and unavailable metrics return null, never infinity. Momentary loudness needs400ms of history; short-term needs3seconds. FFmpeg summary LUFS/true peak values have0.1dB display precision.
+
+Both operations accept mono/stereo8–192kHz, source files<=100MiB and source duration<=30minutes. Each request covers at most120seconds; larger sources require an explicit smaller range. Range end is exclusive. Times are relative to the first displayed video frame, or the first decoded audio frame for audio-only files. Original audio PTS/timebase, audio offset, native channel layout and source hash are retained. Coverage distinguishes intervals with audio from video intervals without an audio stream; missing audio is not measured silence. Boundaries are rounded inward to decoded samples with the tolerance reported. Unsupported discontinuous audio timestamps fail explicitly. No source gain normalization is applied to level measurement.
+
+Both return `ApiResponse<MediaAnalysisJob>`, with the existing polling behavior and `getMediaAnalysis({jobId})` continuation. Owned JSON manifests retain results. Failed jobs never return incomplete analysis as complete. Recognition can miss words under music/overlap; timing precision and recognition accuracy must be evaluated separately.

package/dist/automations/nodes/index.d.ts

@@ -340,7 +340,7 @@ export declare const nodeDefinitions: {
         selectionMode: null;
         outputType: "video" | "image";
         inputs: {
-            type: "number" | "boolean" | "object" | "video" | "image" | "audio" | "text";
+            type: "number" | "boolean" | "object" | "video" | "image" | "text" | "audio";
             id: string;
             isArray: boolean;
             required: boolean;

package/dist/automations/nodes/remotion-composition.d.ts

@@ -11,7 +11,7 @@ declare const defaults: {
     selectionMode: null;
     outputType: "video" | "image";
     inputs: {
-        type: "number" | "boolean" | "object" | "video" | "image" | "audio" | "text";
+        type: "number" | "boolean" | "object" | "video" | "image" | "text" | "audio";
         id: string;
         isArray: boolean;
         required: boolean;
@@ -28,7 +28,7 @@ declare const _default: import("./types").NodeDefinition<"remotion-composition",
     selectionMode: null;
     outputType: "video" | "image";
     inputs: {
-        type: "number" | "boolean" | "object" | "video" | "image" | "audio" | "text";
+        type: "number" | "boolean" | "object" | "video" | "image" | "text" | "audio";
         id: string;
         isArray: boolean;
         required: boolean;

package/dist/index.d.ts

@@ -86,5 +86,5 @@ export type { SuccessResponse, ErrorResponse, ApiResponse, } from './types';
 export type { CropBoundary, CropAxisConfig, DynamicCropConfig, BorderRadiusConfig, VerticalAnchor, HorizontalAnchor, HorizontalSelfAnchor, VerticalSelfAnchor, RelativePositionConfigX, RelativePositionConfigY, } from './render/types';
 export type { PublicPostFailureCode, PublicPostRecovery, PublicPostNextAction, PublicPostFailure } from './post-failure';
 export type { ListProfilePostsParams, ListProfilePostsResult, DownloadSocialPostParams, DownloadSocialPostResult, SocialReferencePost, SocialReferenceAsset } from './social-acquisition';
-export type { ExtractFramesParams, DetectScenesParams, FramesResult, ScenesResult, MediaAnalysisResult, MediaAnalysisJob } from "./media-analysis";
-export { extractFramesSchema, detectScenesSchema, mediaAnalysisStatusSchema, mediaAnalysisResultSchema } from "./media-analysis";
+export type { TranscribeAudioParams, MeasureAudioLevelsParams, AudioTranscriptResult, AudioLevelsResult, ReadImageTextParams, ReadVideoTextParams, ImageTextResult, VideoTextResult, ExtractFramesParams, DetectScenesParams, FramesResult, ScenesResult, MediaAnalysisResult, MediaAnalysisJob } from "./media-analysis";
+export { transcribeAudioSchema, measureAudioLevelsSchema, audioTranscriptResultSchema, audioLevelsResultSchema, readImageTextSchema, readVideoTextSchema, imageTextResultSchema, videoTextResultSchema, extractFramesSchema, detectScenesSchema, mediaAnalysisStatusSchema, mediaAnalysisResultSchema } from "./media-analysis";

package/dist/index.js

@@ -7,7 +7,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.hasMissingTriggerError = exports.getPortErrorsForNode = exports.getErrorNodeIds = exports.validateWorkflow = exports.getCapturedNodes = exports.getNodesDownstreamOfSet = exports.getDownstreamNodes = exports.checkCrossContextViolation = exports.getForEachContext = exports.getConnectedSource = exports.cleanupStaleConnections = exports.removeNodeConnections = exports.removeConnection = exports.addConnection = exports.deriveConnections = exports.createNode = exports.getOutputSchema = exports.getNodeByType = exports.getAllNodes = exports.getInputPreviewValue = exports.computeAllNodePortsWithPreviews = exports.computeAllNodePorts = exports.computePortsWithPreviews = exports.computePortsForNode = exports.areTypesCompatible = exports.parseRemotionPropsJson = exports.REMOTION_COMPOSITION_IMPORTS = exports.remotionCompositionPortSchema = exports.remotionCompositionNodeSchema = exports.remotionCompositionSchema = exports.submitIMessageDmRenderJob = exports.submitInstagramDmRenderJob = exports.submitAutoCaptionRenderJob = exports.submitRemotionCompositionRenderJob = exports.submitScreenshotAnimationRenderJob = exports.submitSceneSplitJob = exports.submitDeduplicationJob = exports.getRenderJobStatus = exports.submitVideoRenderJob = exports.submitImageRenderJob = exports.BillingClient = exports.CommentsClient = exports.MediaClient = exports.AutomationsClient = exports.OrganizationClient = exports.StatsClient = exports.PostsClient = exports.TasksClient = exports.AccountsClient = exports.UGCClient = void 0;
 exports.parseOpenAPIOutputPath = exports.parseOpenAPIInputs = exports.mapOpenAPIType = exports.nodeConfigToCaptionStyle = exports.HttpMethods = exports.prepareImageComposerInput = exports.substituteVariables = exports.processTemplate = exports.extractTemplateVariables = exports.PortIdSchema = exports.normalizeToPortId = exports.portIdToTitle = exports.isValidPortId = exports.portId = exports.selectFromPool = exports.getModelDurations = exports.getModelAspectRatios = exports.ALL_VIDEO_MODELS = exports.getAvailableInputModes = exports.hasReferenceVariant = exports.getReferenceModelId = exports.hasFirstLastFrameVariant = exports.getFirstLastFrameModelId = exports.hasImageToVideoVariant = exports.getImageToVideoModelId = exports.isImageToVideoModel = exports.IMAGE_ASPECT_RATIOS = exports.ALL_IMAGE_MODELS = exports.getEditModelId = exports.isEditModel = exports.getCapturedSourceNodeTypes = exports.getFlowControlNodeTypes = exports.getAllNodeDefinitions = exports.getNodeDefinition = exports.isPortType = exports.formatPortType = exports.isAsyncExecutor = exports.internalNodeTypes = exports.nodeDefinitions = exports.extractWorkflowConfig = exports.generateNodeName = exports.shuffleNodePreview = exports.getPreviewValue = exports.removePreviewForConnection = exports.updatePreviewMapForConnection = exports.computePreviewMap = exports.computeAllNodePreviews = exports.resolveNodePreview = exports.getWorkflowOutputSchema = exports.hasMissingTerminalError = void 0;
-exports.mediaAnalysisResultSchema = exports.mediaAnalysisStatusSchema = exports.detectScenesSchema = exports.extractFramesSchema = exports.prepareVideoComposerInput = exports.formatDateTag = exports.outputFieldsToZod = exports.outputFieldToZod = exports.LLMProviders = exports.applyLogicOperator = exports.IfLogicOperators = exports.resolvePath = exports.indexExpressionToString = exports.indexExpressionToIndexes = exports.resolveUnknownPath = exports.getOutputTypeFromCategory = void 0;
+exports.mediaAnalysisResultSchema = exports.mediaAnalysisStatusSchema = exports.detectScenesSchema = exports.extractFramesSchema = exports.videoTextResultSchema = exports.imageTextResultSchema = exports.readVideoTextSchema = exports.readImageTextSchema = exports.audioLevelsResultSchema = exports.audioTranscriptResultSchema = exports.measureAudioLevelsSchema = exports.transcribeAudioSchema = exports.prepareVideoComposerInput = exports.formatDateTag = exports.outputFieldsToZod = exports.outputFieldToZod = exports.LLMProviders = exports.applyLogicOperator = exports.IfLogicOperators = exports.resolvePath = exports.indexExpressionToString = exports.indexExpressionToIndexes = exports.resolveUnknownPath = exports.getOutputTypeFromCategory = void 0;
 // =============================================================================
 // Client Exports
 // =============================================================================
@@ -175,6 +175,14 @@ Object.defineProperty(exports, "formatDateTag", { enumerable: true, get: functio
 var video_composer_1 = require("./automations/nodes/video-composer");
 Object.defineProperty(exports, "prepareVideoComposerInput", { enumerable: true, get: function () { return video_composer_1.prepareVideoComposerInput; } });
 var media_analysis_1 = require("./media-analysis");
+Object.defineProperty(exports, "transcribeAudioSchema", { enumerable: true, get: function () { return media_analysis_1.transcribeAudioSchema; } });
+Object.defineProperty(exports, "measureAudioLevelsSchema", { enumerable: true, get: function () { return media_analysis_1.measureAudioLevelsSchema; } });
+Object.defineProperty(exports, "audioTranscriptResultSchema", { enumerable: true, get: function () { return media_analysis_1.audioTranscriptResultSchema; } });
+Object.defineProperty(exports, "audioLevelsResultSchema", { enumerable: true, get: function () { return media_analysis_1.audioLevelsResultSchema; } });
+Object.defineProperty(exports, "readImageTextSchema", { enumerable: true, get: function () { return media_analysis_1.readImageTextSchema; } });
+Object.defineProperty(exports, "readVideoTextSchema", { enumerable: true, get: function () { return media_analysis_1.readVideoTextSchema; } });
+Object.defineProperty(exports, "imageTextResultSchema", { enumerable: true, get: function () { return media_analysis_1.imageTextResultSchema; } });
+Object.defineProperty(exports, "videoTextResultSchema", { enumerable: true, get: function () { return media_analysis_1.videoTextResultSchema; } });
 Object.defineProperty(exports, "extractFramesSchema", { enumerable: true, get: function () { return media_analysis_1.extractFramesSchema; } });
 Object.defineProperty(exports, "detectScenesSchema", { enumerable: true, get: function () { return media_analysis_1.detectScenesSchema; } });
 Object.defineProperty(exports, "mediaAnalysisStatusSchema", { enumerable: true, get: function () { return media_analysis_1.mediaAnalysisStatusSchema; } });