@imgly/plugin-ai-generation-web 0.2.17 → 1.68.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,1620 +1,9 @@
1
- # IMG.LY AI Generation Utilities
1
+ # @imgly/plugin-ai-generation-web
2
2
 
3
- A powerful toolkit for implementing AI generation providers in CreativeEditor SDK.
3
+ AI generation plugin for the CE.SDK editor
4
4
 
5
- ## Overview
5
+ For documentation, visit: https://img.ly/docs/cesdk
6
6
 
7
- **Note**: This package is only relevant if you need to create new AI providers or extend existing functionality. For simple integration of AI features, use the [@imgly/plugin-ai-apps-web](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-apps-web) package instead.
7
+ ## License
8
8
 
9
- This package provides the foundation for creating AI generation plugins for CreativeEditor SDK. It offers a standardized interface for implementing AI generation providers that can create images, videos, audio, or text assets. The package includes utilities for handling:
10
-
11
- - Provider registration and initialization
12
- - User interface generation
13
- - Global action registry for quick actions and plugin actions
14
- - Type-safe quick action definitions
15
- - Cross-plugin action support
16
-
17
- ## Getting Started
18
-
19
- ### Installation
20
-
21
- ```bash
22
- npm install @imgly/plugin-ai-generation-web
23
- ```
24
-
25
- ### Creating a Custom Provider
26
-
27
- The core of this package is the `Provider` interface which defines the contract for AI generation providers. Here's how to implement a basic provider:
28
-
29
- ```typescript
30
- import {
31
- Provider,
32
- ImageOutput,
33
- initializeProvider,
34
- loggingMiddleware,
35
- CommonProviderConfiguration
36
- } from '@imgly/plugin-ai-generation-web';
37
-
38
- // Define your provider configuration interface
39
- interface MyProviderConfiguration
40
- extends CommonProviderConfiguration<MyInputType, ImageOutput> {
41
- // Add any provider-specific configuration here
42
- baseURL?: string;
43
- }
44
-
45
- // Create a provider factory function
46
- function createMyImageProvider(config: MyProviderConfiguration): Provider<'image', MyInputType, ImageOutput> {
47
- return {
48
- // Unique identifier for this provider
49
- id: 'my-image-provider',
50
-
51
- // Define output asset type, other options are 'video', 'audio', 'text'
52
- kind: 'image',
53
-
54
- // Initialize the provider
55
- initialize: async ({ engine, cesdk }) => {
56
- // Setup APIs, register further components, etc.
57
- myAIApi.configure({
58
- apiKey: 'YOUR_API_KEY',
59
- headers: config.headers // Use custom headers if provided
60
- });
61
- },
62
-
63
- // Define input panel and UI components
64
- input: {
65
- // Define how the input panel is rendered
66
- panel: {
67
- // Option 1: Schema-based UI (using OpenAPI)
68
- type: 'schema',
69
- document: myApiSchema,
70
- inputReference: '#/components/schemas/GenerationInput',
71
- getBlockInput: async (input) => ({
72
- image: { width: 1024, height: 1024 }
73
- })
74
- },
75
-
76
- // Quick actions supported by this provider
77
- quickActions: {
78
- supported: {
79
- 'ly.img.editImage': {
80
- mapInput: (input) => ({
81
- prompt: input.prompt,
82
- image_url: input.uri
83
- })
84
- },
85
- 'ly.img.styleTransfer': {
86
- mapInput: (input) => ({
87
- prompt: input.style,
88
- image_url: input.uri
89
- })
90
- }
91
- }
92
- }
93
- },
94
-
95
- // Define output generation behavior
96
- output: {
97
- // Allow cancellation
98
- abortable: true,
99
-
100
- // Store generated assets, options are:
101
- // - false: No history
102
- // - '@imgly/local': In-memory storage (lost on refresh)
103
- // - '@imgly/indexedDB': Browser IndexedDB storage
104
- // - any other string: Handled as a custom asset source ID
105
- history: '@imgly/indexedDB',
106
-
107
- // Add middleware for pre/post-processing of the generation
108
- middleware: [loggingMiddleware()],
109
-
110
- // Configure success/error notifications
111
- notification: {
112
- success: {
113
- show: true,
114
- message: 'Generation successful!'
115
- }
116
- },
117
-
118
- // Core generation function
119
- generate: async (input, { abortSignal, engine }) => {
120
- // Call your AI API and return result
121
- const response = await myAIApi.generateImage(input, {
122
- headers: config.headers // Pass custom headers to API
123
- });
124
-
125
- return {
126
- kind: 'image',
127
- url: response.imageUrl
128
- };
129
- }
130
- }
131
- };
132
- }
133
-
134
- // Usage example
135
- const myImageProvider = createMyImageProvider({
136
- proxyUrl: 'http://your-proxy-server.com/api/proxy',
137
- headers: {
138
- 'x-client-version': '1.0.0',
139
- 'x-request-source': 'cesdk-plugin'
140
- },
141
- debug: false,
142
- middleware: [loggingMiddleware()],
143
- baseURL: 'https://assets.example.com'
144
- });
145
- ```
146
-
147
- ## Action Registry
148
-
149
- The package includes a global `ActionRegistry` for managing quick actions and plugin actions. To register a new action:
150
-
151
- ```typescript
152
- import { ActionRegistry } from '@imgly/plugin-ai-generation-web';
153
-
154
- // Get the global registry instance
155
- const registry = ActionRegistry.get();
156
-
157
- // Register a quick action
158
- const unregister = registry.register({
159
- id: 'my-quick-action',
160
- type: 'quick',
161
- kind: 'image',
162
- label: 'My Quick Action',
163
- enable: true,
164
- render: (context) => {
165
- // Render the quick action UI
166
- context.builder.Button('my-button', {
167
- label: 'Generate',
168
- onClick: async () => {
169
- await context.generate({ prompt: 'Hello world' });
170
- }
171
- });
172
- }
173
- });
174
- ```
175
-
176
- ## Provider Interface
177
-
178
- The Provider interface is generic and type-safe, supporting four output kinds:
179
-
180
- ```typescript
181
- // K: Output kind ('image', 'video', 'audio', 'text')
182
- // I: Input type specific to your provider, i.e. what does the generate function need
183
- // O: Output type (ImageOutput, VideoOutput, AudioOutput, TextOutput)
184
- // C: Chunk type for streaming (optional, defaults to O)
185
- interface Provider<K extends OutputKind, I, O extends Output, C = O> { ... }
186
- ```
187
-
188
- ## Common Provider Configuration
189
-
190
- All providers should extend the `CommonProviderConfiguration` interface, which provides standardized configuration options:
191
-
192
- ```typescript
193
- interface CommonProviderConfiguration<I, O extends Output> {
194
- // The proxy URL to use for the provider
195
- proxyUrl: string;
196
-
197
- // Enable debug mode for additional logging
198
- debug?: boolean;
199
-
200
- // Middleware for request/response processing
201
- middleware?: Middleware<I, O>[];
202
-
203
- // Custom headers to include in all API requests
204
- headers?: Record<string, string>;
205
-
206
- // Override provider's default history asset source
207
- history?: false | '@imgly/local' | '@imgly/indexedDB' | (string & {});
208
-
209
- // Configure supported quick actions
210
- supportedQuickActions?: {
211
- [quickActionId: string]: Partial<QuickActionSupport<I>> | false | null;
212
- };
213
-
214
- // Configure default property values
215
- properties?: PropertiesConfiguration;
216
- }
217
- ```
218
-
219
- ### Extended Configuration Options
220
-
221
- #### History Configuration
222
-
223
- The `history` field allows you to override the provider's default history storage behavior:
224
-
225
- ```typescript
226
- const provider = createMyImageProvider({
227
- proxyUrl: 'https://proxy.example.com',
228
-
229
- // Disable history storage entirely
230
- history: false,
231
-
232
- // Or use temporary local storage (not persistent)
233
- // history: '@imgly/local',
234
-
235
- // Or use persistent browser storage (default for most providers)
236
- // history: '@imgly/indexedDB',
237
-
238
- // Or use a custom asset source ID
239
- // history: 'my-custom-asset-source'
240
- });
241
- ```
242
-
243
- **Available Options:**
244
- - `false`: Disable history storage entirely
245
- - `'@imgly/local'`: Use temporary local storage (not persistent across sessions)
246
- - `'@imgly/indexedDB'`: Use browser IndexedDB storage (persistent across sessions)
247
- - `string`: Use your own custom asset source ID
248
-
249
- #### Quick Actions Configuration
250
-
251
- The `supportedQuickActions` field allows you to customize which quick actions are supported and how they behave:
252
-
253
- ```typescript
254
- const provider = createMyImageProvider({
255
- proxyUrl: 'https://proxy.example.com',
256
-
257
- // Configure quick actions
258
- supportedQuickActions: {
259
- // Remove an unwanted quick action
260
- 'ly.img.editImage': false,
261
-
262
- // Override with custom mapping
263
- 'ly.img.swapBackground': {
264
- mapInput: (input) => ({
265
- prompt: input.prompt,
266
- image_url: input.uri,
267
- strength: 0.9, // Custom strength
268
- style: 'REALISTIC' // Force realistic style
269
- })
270
- },
271
-
272
- // Add support for new quick action
273
- 'ly.img.customAction': {
274
- mapInput: (input) => ({
275
- prompt: `Custom prefix: ${input.text}`,
276
- image_url: input.imageUrl
277
- })
278
- }
279
- }
280
- });
281
- ```
282
-
283
- **Configuration Values:**
284
- - `false` or `null`: Remove the quick action entirely
285
- - `true`: Keep the provider's default implementation
286
- - Object with `mapInput`: Override the quick action with custom input mapping
287
- - Object with other properties: Override with custom configuration
288
-
289
- #### Property Configuration
290
-
291
- The `properties` field allows you to define default values for any provider property. These defaults can be static values or dynamic functions that receive context:
292
-
293
- ```typescript
294
- const provider = createMyImageProvider({
295
- proxyUrl: 'https://proxy.example.com',
296
-
297
- // Configure default property values
298
- properties: {
299
- // Static default value
300
- image_size: 'square_hd',
301
-
302
- // Dynamic default based on context
303
- style: (context) => {
304
- // Context includes: engine, cesdk, locale
305
- const locale = context.locale;
306
-
307
- // Return different defaults for different locales
308
- if (locale === 'de') {
309
- return 'realistic';
310
- }
311
- return 'digital_illustration';
312
- },
313
-
314
- // Dynamic default based on design state
315
- resolution: (context) => {
316
- const engine = context.engine;
317
- const scene = engine.scene.get();
318
- const width = engine.block.getFloat(scene, 'scene/frame/width');
319
-
320
- // Choose resolution based on canvas size
321
- if (width > 1920) {
322
- return '1080p';
323
- }
324
- return '720p';
325
- }
326
- }
327
- });
328
- ```
329
-
330
- **Property Configuration Features:**
331
- - **Static Defaults**: Simple values that apply to all users
332
- - **Dynamic Defaults**: Functions that return values based on context (engine state, locale, etc.)
333
- - **Context Available**: `engine`, `cesdk`, `locale` for making informed decisions
334
- - **Fallback Chain**: Properties use configured value → schema default → undefined
335
-
336
- ### Headers Configuration
337
-
338
- The `headers` property allows you to include custom HTTP headers in all API requests made by your provider. This is useful for:
339
- - Adding custom client identification headers
340
- - Including version information
341
- - Passing through metadata required by your API
342
- - Adding correlation IDs for request tracing
343
-
344
- **Implementation Note:** When implementing your provider's `generate` function, ensure you merge the custom headers with any required headers for your API:
345
-
346
- ```typescript
347
- // In your generate function
348
- const response = await fetch(apiUrl, {
349
- method: 'POST',
350
- headers: {
351
- 'Content-Type': 'application/json',
352
- 'Authorization': `Bearer ${apiKey}`,
353
- ...config.headers // Spread custom headers
354
- },
355
- body: JSON.stringify(requestData)
356
- });
357
- ```
358
-
359
- ### Key Provider Options
360
-
361
- - **id**: Unique identifier for your provider
362
- - **kind**: Type of asset generated ('image', 'video', 'audio', 'text')
363
- - **name**: Optional human-readable name
364
- - **initialize**: Setup function called when the provider is loaded
365
- - **input**: Configuration for input UI and parameters
366
- - **output**: Configuration for generation and result handling
367
-
368
- #### Provider Output Options
369
-
370
- The `output` property has several important options:
371
-
372
- - **generate**: Main function that performs the actual generation
373
- - **history**: Asset storage strategy ('false', '@imgly/local', '@imgly/indexedDB', or custom ID)
374
- - **abortable**: Whether generation can be cancelled by the user
375
- - **middleware**: Array of middleware functions for pre/post-processing
376
- - **notification**: Success and error notification configuration
377
- - **generationHintText**: Text to display below the generation button
378
-
379
- ##### Notification Configuration
380
-
381
- The notification system allows fine-grained control over success and error messages:
382
-
383
- ```typescript
384
- notification: {
385
- success: {
386
- // Control whether to show notifications (can be dynamic)
387
- show: true, // or (context) => shouldShow(context)
388
-
389
- // Message text or i18n key (can be dynamic)
390
- message: 'Generation successful!', // or (context) => getMessage(context)
391
-
392
- // Optional action button
393
- action: {
394
- label: 'View', // or (context) => getLabel(context)
395
- onClick: (context) => { /* handle click */ }
396
- },
397
-
398
- // How long to show the notification
399
- duration: 'short' // or 'medium', 'long', 'infinite'
400
- },
401
-
402
- error: {
403
- // Similar options for error notifications
404
- show: true,
405
- message: 'Generation failed', // or (context) => getErrorMessage(context)
406
- // ...
407
- }
408
- }
409
- ```
410
-
411
- ##### Streaming Generation
412
-
413
- The `generate` function can return a simple output object or an AsyncGenerator for streaming results:
414
-
415
- ```typescript
416
- // Simple response
417
- generate: async (input, options) => {
418
- const result = await api.generateImage(input);
419
- return { kind: 'image', url: result.url };
420
- }
421
-
422
- // Streaming response (currently only supported for text)
423
- generate: async function* (input, options) {
424
- const stream = api.streamGenerationResult(input);
425
-
426
- let inferredText: string = '';
427
- // Yield interim results
428
- for await (const chunk of stream) {
429
- inferredText += chunk;
430
- yield { kind: 'text', text: inferredText };
431
- }
432
-
433
- // Return final result
434
- return { kind: 'text', text: inferredText };
435
- }
436
- ```
437
-
438
- ##### Generation Hint Text
439
-
440
- The `generationHintText` property allows providers to display helpful information below the generation button:
441
-
442
- ```typescript
443
- generationHintText: "Generation may take up to a minute. You can close this panel and will be notified when ready."
444
- ```
445
-
446
- ## Input Panel Types
447
-
448
- The package supports two approaches for creating input panels:
449
-
450
- ### 1. Schema-based Input Panels
451
-
452
- The `schema` type uses OpenAPI specification to declaratively define your input form.
453
-
454
- ```typescript
455
- input: {
456
- panel: {
457
- type: 'schema',
458
- // Complete OpenAPI v3 document describing your inputs
459
- document: myOpenAPISchema,
460
- // JSON pointer to your input schema within the document
461
- inputReference: '#/components/schemas/GenerationInput',
462
- // Optional property to control display order
463
- orderExtensionKeyword: 'x-order-properties',
464
- // Function that converts input to block parameters
465
- getBlockInput: async (input) => ({
466
- image: { width: 1024, height: 1024 }
467
- }),
468
- // Optional custom renderers for specific properties found in the schema
469
- renderCustomProperty: {
470
- // This is a custom renderer for a fictional `imageUrl` property
471
- imageUrl: (context, property) => {
472
- const valueState = context.state('imageUrl', '');
473
- context.builder.TextInput('imageUrl', {
474
- inputLabel: 'Image URL',
475
- ...valueState
476
- });
477
-
478
- // Return a function that returns the value for this property
479
- return () => { id: property.id, type: 'string', value: valueState.value };
480
- }
481
- }
482
- }
483
- }
484
- ```
485
-
486
- #### OpenAPI Schema Example
487
-
488
- ```json
489
- {
490
- "openapi": "3.0.0",
491
- "components": {
492
- "schemas": {
493
- "GenerationInput": {
494
- "type": "object",
495
- "required": ["prompt"],
496
- "properties": {
497
- "prompt": {
498
- "type": "string",
499
- "title": "Prompt",
500
- "description": "Describe what you want to generate",
501
- "x-imgly-builder": {
502
- "component": "TextArea"
503
- }
504
- },
505
- "width": {
506
- "type": "integer",
507
- "title": "Width",
508
- "default": 1024,
509
- "enum": [512, 1024, 2048],
510
- "x-imgly-builder": {
511
- "component": "Select"
512
- }
513
- }
514
- },
515
- "x-order-properties": ["prompt", "width"]
516
- }
517
- }
518
- }
519
- }
520
- ```
521
-
522
- #### Benefits of Schema-based Input
523
-
524
- - Built-in validation based on schema constraints
525
- - AI providers like fal.ai provide schemas for their models
526
- - Automatic UI component generation based on property types
527
- - Extensions like `x-imgly-builder` to specify component types
528
- - Property ordering via `orderExtensionKeyword`
529
- - Customizable property rendering with `renderCustomProperty`
530
-
531
- ### 2. Custom Input Panels
532
-
533
- The `custom` type gives you complete control over UI components. For more details on how to build custom panels and see all available builder components, refer to the [Create a Custom Panel](https://img.ly/docs/cesdk/js/user-interface/ui-extensions/create-custom-panel-d87b83/) guide.
534
-
535
- ```typescript
536
- input: {
537
- panel: {
538
- type: 'custom',
539
- render: (context, options) => {
540
- // Use the builder pattern to create UI components
541
- const promptState = context.state('prompt', '');
542
- context.builder.TextArea('prompt', {
543
- inputLabel: 'Prompt',
544
- ...promptState
545
- });
546
-
547
- // Set up width selection
548
- const widthState = context.state('width', 1024);
549
- context.builder.Select('width', {
550
- inputLabel: 'Width',
551
- options: [
552
- { value: 512, label: '512px' },
553
- { value: 1024, label: '1024px' },
554
- { value: 2048, label: '2048px' }
555
- ],
556
- ...widthState
557
- });
558
-
559
- // Return functions to get input values and block parameters
560
- return {
561
- // The input for the generate function
562
- getInput: () => ({
563
- prompt: promptState.value,
564
- width: widthState.value
565
- }),
566
- // The input for the block creation
567
- getBlockInput: () => ({
568
- image: {
569
- width: widthState.value,
570
- height: widthState.value,
571
- label: `AI Image: ${promptState.value.substring(0, 20)}...`
572
- }
573
- })
574
- };
575
- }
576
- }
577
- }
578
- ```
579
-
580
- #### Benefits of Custom Input Panels
581
-
582
- - Complete control over UI components and layout
583
- - Complex logic between fields (dependencies, conditionals)
584
- - Dynamic UI that changes based on user interactions
585
-
586
- #### Panel User Flow Options
587
-
588
- Both panel types accept additional configuration:
589
-
590
- ```typescript
591
- panel: {
592
- type: 'schema', // or 'custom'
593
- // ...panel type specific options
594
-
595
- // Control the generation flow
596
- userFlow: 'placeholder', // or 'generation-only' (default)
597
-
598
- // Include/exclude history library from panel
599
- includeHistoryLibrary: true // (default)
600
- }
601
- ```
602
-
603
- - **userFlow**:
604
- - `placeholder`: Creates a block as a placeholder with loading state when generation starts
605
- - `generation-only`: Only triggers generation without creating a placeholder
606
-
607
- - **includeHistoryLibrary**: Controls whether the history library is shown in the panel
608
-
609
- ## The `getBlockInput` Function
610
-
611
- The `getBlockInput` function is crucial for both panel types. It converts your input into the parameters needed to create a block in CreativeEditor SDK.
612
-
613
- ### What It Does
614
-
615
- - Defines dimensions, duration, and appearance of asset blocks
616
- - Creates placeholders before generation completes
617
- - Maps your AI provider's inputs to standardized block parameters
618
-
619
- ### Required Return Values by Output Kind
620
-
621
- Each output kind requires specific parameters:
622
-
623
- #### For Images
624
-
625
- ```typescript
626
- getBlockInput: async (input) => ({
627
- image: {
628
- width: 1024, // Required - Width in pixels
629
- height: 1024, // Required - Height in pixels
630
- label: 'My Image' // Optional - Display name
631
- }
632
- });
633
- ```
634
-
635
- #### For Videos
636
-
637
- ```typescript
638
- getBlockInput: async (input) => ({
639
- video: {
640
- width: 1280, // Required - Width in pixels
641
- height: 720, // Required - Height in pixels
642
- duration: 10, // Required - Duration in seconds
643
- label: 'My Video' // Optional - Display name
644
- }
645
- });
646
- ```
647
-
648
- #### For Audio
649
-
650
- ```typescript
651
- getBlockInput: async (input) => ({
652
- audio: {
653
- duration: 30, // Optional - Duration in seconds
654
- thumbnailUrl: 'path/to/img.jpg', // Optional - URL for thumbnail
655
- label: 'My Audio' // Optional - Display name
656
- }
657
- });
658
- ```
659
-
660
- #### For Text
661
-
662
- ```typescript
663
- getBlockInput: async (input) => ({
664
- text: {
665
- length: 250, // Required - Approximate character length
666
- label: 'My Text' // Optional - Display name
667
- }
668
- });
669
- ```
670
-
671
- ## Quick Actions
672
-
673
- Quick Actions provide context-aware AI generation capabilities directly in CreativeEditor SDK's canvas menu. Unlike panels (which appear in the side panel), quick actions appear when users select elements on the canvas.
674
-
675
- ### Available Quick Action IDs
676
-
677
- Here are all the quick action IDs that can be used in the `supported` field of your provider configuration:
678
-
679
- #### Image Quick Actions
680
-
681
- - **`ly.img.artistTransfer`**: Transform image in the style of famous artists
682
- - Input: `{ artist: string, uri: string }`
683
-
684
- - **`ly.img.combineImages`**: Combine multiple images with instructions
685
- - Input: `{ prompt: string, uris: string[], exportFromBlockIds: number[] }`
686
-
687
- - **`ly.img.createVariant`**: Create a variation of the image
688
- - Input: `{ prompt: string, uri: string }`
689
-
690
- - **`ly.img.editImage`**: Change image based on description
691
- - Input: `{ prompt: string, uri: string }`
692
-
693
- - **`ly.img.remixPage`**: Convert the page into a single image
694
- - Input: `{ prompt: string, uri: string }`
695
-
696
- - **`ly.img.remixPageWithPrompt`**: Remix the page with custom instructions
697
- - Input: `{ prompt: string, uri: string }`
698
-
699
- - **`ly.img.styleTransfer`**: Transform image into different art styles
700
- - Input: `{ style: string, uri: string }`
701
-
702
- - **`ly.img.swapBackground`**: Change the background of the image
703
- - Input: `{ prompt: string, uri: string }`
704
-
705
- - **`ly.img.gpt-image-1.changeStyleLibrary`**: Apply different art styles (GPT-specific)
706
- - Input: `{ prompt: string, uri: string }`
707
-
708
- #### Text Quick Actions
709
-
710
- - **`ly.img.changeTextTo`**: Change text to a different format or style
711
- - Input: `{ prompt: string, customPrompt: string }`
712
-
713
- - **`ly.img.changeTone`**: Change the tone of the text
714
- - Input: `{ prompt: string, type: string }`
715
-
716
- - **`ly.img.fix`**: Fix spelling and grammar
717
- - Input: `{ prompt: string }`
718
-
719
- - **`ly.img.improve`**: Improve writing quality
720
- - Input: `{ prompt: string }`
721
-
722
- - **`ly.img.longer`**: Make text longer
723
- - Input: `{ prompt: string }`
724
-
725
- - **`ly.img.shorter`**: Make text shorter
726
- - Input: `{ prompt: string }`
727
-
728
- - **`ly.img.translate`**: Translate text to different languages
729
- - Input: `{ prompt: string, language: string }`
730
-
731
- #### Video Quick Actions
732
-
733
- - **`ly.img.createVideo`**: Opens the image2video generation panel with the current image
734
- - Input: `{ uri: string }`
735
-
736
- ### Provider Quick Action Support
737
-
738
- Providers declare which quick actions they support and how to map quick action inputs to provider inputs:
739
-
740
- ```typescript
741
- const myProvider = {
742
- // ... other provider config
743
- input: {
744
- // ... panel config
745
- quickActions: {
746
- supported: {
747
- 'ly.img.editImage': {
748
- mapInput: (quickActionInput) => ({
749
- prompt: quickActionInput.prompt,
750
- image_url: quickActionInput.uri
751
- })
752
- },
753
- 'ly.img.styleTransfer': {
754
- mapInput: (quickActionInput) => ({
755
- style: quickActionInput.style,
756
- image_url: quickActionInput.uri
757
- })
758
- }
759
- }
760
- }
761
- }
762
- };
763
- ```
764
-
765
- ### Quick Action Expanded View
766
-
767
- Quick actions can have two rendering modes:
768
-
769
- 1. **Collapsed View**: Shows as a simple button in the quick action menu alongside other actions
770
- 2. **Expanded View**: Takes over the entire menu space, hiding other actions while the user interacts with this specific action
771
-
772
- The expanded view is useful for quick actions that need user input (like text prompts). When a quick action is expanded, the complete menu is replaced with the expanded interface, and other menu items are not shown until the user either completes the action or cancels back to the collapsed view.
773
-
774
- ```typescript
775
- render: ({ builder, isExpanded, toggleExpand }) => {
776
- if (isExpanded) {
777
- // Expanded view - takes over the entire menu
778
- builder.TextArea('prompt', { /* input fields */ });
779
- builder.ButtonRow('actions', { /* confirm/cancel buttons */ });
780
- } else {
781
- // Collapsed view - simple button alongside other actions
782
- builder.Button('expand', {
783
- label: 'Edit Image...',
784
- onClick: toggleExpand
785
- });
786
- }
787
- }
788
- ```
789
-
790
- ## Customizing Labels and Text
791
-
792
- You can customize all labels and text in the AI generation interface using the translation system. This allows you to provide better labels for your users in any language.
793
-
794
- ### Translation Priority
795
-
796
- The system checks for translations in this order (highest to lowest priority):
797
-
798
- 1. **Provider & Kind-specific**: `ly.img.plugin-ai-${kind}-generation-web.${provider}.property.${field}` - Override labels for a specific AI provider and generation type
799
- 2. **Generic**: `ly.img.plugin-ai-generation-web.property.${field}` - Override labels for all AI plugins
800
-
801
- Where `${kind}` can be:
802
- - `image` for image generation plugins
803
- - `video` for video generation plugins
804
- - `audio` for audio generation plugins
805
- - `text` for text generation plugins
806
-
807
- ### Basic Example
808
-
809
- ```typescript
810
- // Customize labels for your AI generation interface
811
- cesdk.i18n.setTranslations({
812
- en: {
813
- // Generic labels (applies to ALL AI plugins)
814
- 'ly.img.plugin-ai-generation-web.property.prompt': 'Describe what you want to create',
815
- 'ly.img.plugin-ai-generation-web.property.image_size': 'Image Dimensions',
816
- 'ly.img.plugin-ai-generation-web.property.duration': 'Video Length',
817
-
818
- // Provider-specific for images (highest priority)
819
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.prompt': 'Describe your Recraft image',
820
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.image_size': 'Canvas Size',
821
-
822
- // Provider-specific for videos (highest priority)
823
- 'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.prompt': 'Describe your video scene',
824
- 'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.duration': 'Video Duration'
825
- }
826
- });
827
- ```
828
-
829
- ### Customizing Input Placeholders
830
-
831
- You can customize placeholder text for input fields (like the prompt textarea) using the same translation system. Placeholders help guide users on what to enter:
832
-
833
- ```typescript
834
- cesdk.i18n.setTranslations({
835
- en: {
836
- // Generic placeholder (applies to ALL AI plugins)
837
- 'ly.img.plugin-ai-generation-web.property.prompt.placeholder':
838
- 'e.g., A serene mountain landscape at sunset...',
839
-
840
- // Provider-specific placeholders (highest priority)
841
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.prompt.placeholder':
842
- 'Describe your image in detail...',
843
-
844
- 'ly.img.plugin-ai-image-generation-web.fal-ai/gemini-25-flash-image/edit.property.prompt.placeholder':
845
- 'Describe the changes you want to make...',
846
-
847
- 'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.prompt.placeholder':
848
- 'Describe the video scene, camera movements, and style...'
849
- }
850
- });
851
- ```
852
-
853
- **Translation Priority for Placeholders:**
854
-
855
- Placeholders follow the same priority chain as labels:
856
- 1. `ly.img.plugin-ai-{kind}-generation-web.{provider.id}.property.{field}.placeholder` (highest)
857
- 2. `ly.img.plugin-ai-generation-web.property.{field}.placeholder`
858
- 3. `ly.img.plugin-ai-{kind}-generation-web.{provider.id}.defaults.property.{field}.placeholder`
859
- 4. `ly.img.plugin-ai-generation-web.defaults.property.{field}.placeholder` (lowest)
860
-
861
- ### Dropdown Options
862
-
863
- For dropdown menus, add the option value to the translation key:
864
-
865
- ```typescript
866
- cesdk.i18n.setTranslations({
867
- en: {
868
- // Image generation dropdown options
869
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.image_size.square_hd': 'Square HD (1024×1024)',
870
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.image_size.portrait_4_3': 'Portrait 4:3 (768×1024)',
871
-
872
- // Video generation dropdown options
873
- 'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.duration.5': '5 seconds',
874
- 'ly.img.plugin-ai-video-generation-web.fal-ai/veo3.property.duration.10': '10 seconds'
875
- }
876
- });
877
- ```
878
-
879
- ### QuickAction Translations
880
-
881
- QuickActions use their own translation keys with provider-specific overrides:
882
-
883
- ```typescript
884
- cesdk.i18n.setTranslations({
885
- en: {
886
- // Provider-specific translations (highest priority)
887
- 'ly.img.plugin-ai-image-generation-web.fal-ai/gemini-flash-edit.quickAction.editImage': 'Edit with Gemini',
888
- 'ly.img.plugin-ai-text-generation-web.anthropic.quickAction.improve': 'Improve with Claude',
889
-
890
- // Generic plugin translations
891
- 'ly.img.plugin-ai-image-generation-web.quickAction.editImage': 'Edit Image...',
892
- 'ly.img.plugin-ai-image-generation-web.quickAction.editImage.prompt': 'Edit Image...',
893
- 'ly.img.plugin-ai-image-generation-web.quickAction.editImage.apply': 'Change',
894
- 'ly.img.plugin-ai-text-generation-web.quickAction.improve': 'Improve Text',
895
- 'ly.img.plugin-ai-text-generation-web.quickAction.translate': 'Translate Text',
896
- 'ly.img.plugin-ai-video-generation-web.quickAction.createVideo': 'Create Video'
897
- }
898
- });
899
- ```
900
-
901
- **QuickAction Translation Priority:**
902
- 1. Provider-specific: `ly.img.plugin-ai-${kind}-generation-web.${provider}.quickAction.${action}.${field}`
903
- 2. Generic plugin: `ly.img.plugin-ai-${kind}-generation-web.quickAction.${action}.${field}`
904
-
905
- **Translation Structure:**
906
- - Base key (e.g., `.quickAction.editImage`): Button text when QuickAction is collapsed
907
- - `.prompt`: Label for input field when expanded
908
- - `.prompt.placeholder`: Placeholder text for input field
909
- - `.apply`: Text for action/submit button
910
-
911
- ## Using Your Provider
912
-
913
- Once you've created your provider, you need to initialize it with CreativeEditor SDK and integrate it into the UI.
914
-
915
- ### Initializing Your Provider
916
-
917
- Use the `initializeProvider` function to register your provider:
918
-
919
- ```typescript
920
- import { initializeProvider } from '@imgly/plugin-ai-generation-web';
921
-
922
- // Create your provider
923
- const myProvider = createMyProvider({
924
- proxyUrl: 'http://your-proxy-server.com/api/proxy',
925
- headers: {
926
- 'x-custom-header': 'value',
927
- 'x-client-version': '1.0.0'
928
- }
929
- });
930
-
931
- // Initialize the provider
932
- function setupMyProvider(cesdk) {
933
- const result = initializeProvider(
934
- myProvider,
935
- {
936
- engine: cesdk.engine,
937
- cesdk
938
- },
939
- {
940
- debug: false,
941
- dryRun: false
942
- }
943
- );
944
-
945
- return result;
946
- }
947
- ```
948
-
949
- ### Panel IDs and Registration
950
-
951
- When a provider is initialized, it automatically registers panels with specific IDs:
952
-
953
- ```
954
- ly.img.plugin-ai-{kind}-generation-web.{provider-id}
955
- ```
956
-
957
- For example:
958
- - A provider with ID `my-image-provider` for images registers a panel with ID `ly.img.plugin-ai-image-generation-web.my-image-provider`
959
- - A provider with ID `fal-ai/recraft-v3` for images registers a panel with ID `ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3`
960
-
961
- You can programmatically get a panel ID using the `getPanelId` function:
962
-
963
- ```typescript
964
- import { getPanelId } from '@imgly/plugin-ai-generation-web';
965
-
966
- // Get panel ID for a provider
967
- const panelId = getPanelId('my-image-provider');
968
-
969
- // Open the panel
970
- cesdk.ui.openPanel(panelId);
971
- ```
972
-
973
- ### Quick Action Menu Registration
974
-
975
- Quick actions are automatically registered in canvas menus with these IDs:
976
-
977
- ```
978
- ly.img.plugin-ai-{kind}-generation-web.canvasMenu
979
- ```
980
-
981
- For example:
982
- - Image quick actions: `ly.img.plugin-ai-image-generation-web.canvasMenu`
983
- - Video quick actions: `ly.img.plugin-ai-video-generation-web.canvasMenu`
984
- - Audio quick actions: `ly.img.plugin-ai-audio-generation-web.canvasMenu`
985
- - Text quick actions: `ly.img.plugin-ai-text-generation-web.canvasMenu`
986
-
987
- ### Customizing Quick Action Availability
988
-
989
- You can control which quick actions appear in your application using the Feature API. This is useful when you want to:
990
- - Show only specific AI capabilities to certain user groups
991
- - Simplify the UI by hiding advanced features
992
- - Create different feature sets for different subscription tiers
993
- - Disable actions that aren't relevant to your use case
994
-
995
- #### Disabling Specific Quick Actions
996
-
997
- All quick actions are enabled by default. To hide specific quick actions from the UI:
998
-
999
- ```typescript
1000
- // Hide the "Edit Image" quick action from users
1001
- cesdk.feature.enable(
1002
- 'ly.img.plugin-ai-image-generation-web.quickAction.editImage',
1003
- false
1004
- );
1005
-
1006
- // Hide multiple text quick actions for a simpler experience
1007
- cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.changeTone', false);
1008
- cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.translate', false);
1009
- ```
1010
-
1011
- #### Dynamic Feature Control
1012
-
1013
- You can also pass a function to dynamically control feature availability based on context. This is powerful for implementing complex business logic, time-based features, or context-sensitive UI. See the [CE.SDK Feature API documentation](https://img.ly/docs/cesdk/js/user-interface/customization/disable-or-enable-f058e2/) for more details.
1014
-
1015
- ```typescript
1016
- // Show advanced AI features only during business hours
1017
- cesdk.feature.enable(
1018
- 'ly.img.plugin-ai-image-generation-web.quickAction.artistTransfer',
1019
- ({ isPreviousEnable }) => {
1020
- const hour = new Date().getHours();
1021
- const isBusinessHours = hour >= 9 && hour < 18;
1022
- return isBusinessHours && isPreviousEnable();
1023
- }
1024
- );
1025
-
1026
- // Disable certain quick actions based on selected content
1027
- cesdk.feature.enable(
1028
- 'ly.img.plugin-ai-text-generation-web.quickAction.translate',
1029
- ({ engine, isPreviousEnable }) => {
1030
- const selectedBlocks = engine.block.findAllSelected();
1031
- if (selectedBlocks.length === 0) return false;
1032
-
1033
- const blockId = selectedBlocks[0];
1034
- const textContent = engine.block.getString(blockId, 'text/text');
1035
-
1036
- // Only show translate if text is long enough
1037
- const hasEnoughText = textContent && textContent.length > 20;
1038
- return hasEnoughText && isPreviousEnable();
1039
- }
1040
- );
1041
- ```
1042
-
1043
- #### Creating Feature Sets for Different User Tiers
1044
-
1045
- ```typescript
1046
- // Configure features based on user subscription
1047
- function configureAIFeatures(cesdk, userTier) {
1048
- if (userTier === 'basic') {
1049
- // Basic users only get simple text improvements
1050
- cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.improve', true);
1051
- cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.fix', true);
1052
- cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.translate', false);
1053
- cesdk.feature.enable('ly.img.plugin-ai-text-generation-web.quickAction.changeTone', false);
1054
-
1055
- // Disable advanced image features
1056
- cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction.artistTransfer', false);
1057
- cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction.styleTransfer', false);
1058
- } else if (userTier === 'premium') {
1059
- // Premium users get all features (default behavior)
1060
- // All quick actions are enabled by default
1061
- }
1062
- }
1063
- ```
1064
-
1065
- #### Available Feature Flags
1066
-
1067
- ##### Core Plugin Features
1068
-
1069
- These feature flags control the main functionality of each AI plugin:
1070
-
1071
- **General Features:**
1072
- - `ly.img.plugin-ai-{kind}-generation-web.providerSelect` - Enable/disable provider selection dropdown in panels
1073
- - `ly.img.plugin-ai-{kind}-generation-web.quickAction` - Enable/disable all quick actions for a plugin
1074
- - `ly.img.plugin-ai-{kind}-generation-web.quickAction.providerSelect` - Enable/disable provider selection dropdown in quick actions
1075
-
1076
- **Input Type Features (Image & Video only):**
1077
- - `ly.img.plugin-ai-image-generation-web.fromText` - Enable/disable text-to-image generation
1078
- - `ly.img.plugin-ai-image-generation-web.fromImage` - Enable/disable image-to-image generation
1079
- - `ly.img.plugin-ai-video-generation-web.fromText` - Enable/disable text-to-video generation
1080
- - `ly.img.plugin-ai-video-generation-web.fromImage` - Enable/disable image-to-video generation
1081
-
1082
- **Usage Examples:**
1083
-
1084
- ```typescript
1085
- // Hide provider selection dropdown in video generation panel
1086
- cesdk.feature.enable('ly.img.plugin-ai-video-generation-web.providerSelect', false);
1087
-
1088
- // Only allow text-to-image, disable image-to-image editing
1089
- cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.fromImage', false);
1090
- cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.fromText', true);
1091
-
1092
- // Hide provider selection dropdown in quick actions (use default provider only)
1093
- cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction.providerSelect', false);
1094
-
1095
- // Disable all quick actions but keep panel generation available
1096
- cesdk.feature.enable('ly.img.plugin-ai-image-generation-web.quickAction', false);
1097
- ```
1098
-
1099
- ##### Quick Action Features
1100
-
1101
- The quick action feature flags follow this pattern: `ly.img.plugin-ai-{kind}-generation-web.quickAction.{actionName}`
1102
-
1103
- **Image Quick Actions:**
1104
- - `ly.img.plugin-ai-image-generation-web.quickAction.editImage`
1105
- - `ly.img.plugin-ai-image-generation-web.quickAction.swapBackground`
1106
- - `ly.img.plugin-ai-image-generation-web.quickAction.styleTransfer`
1107
- - `ly.img.plugin-ai-image-generation-web.quickAction.artistTransfer`
1108
- - `ly.img.plugin-ai-image-generation-web.quickAction.createVariant`
1109
- - `ly.img.plugin-ai-image-generation-web.quickAction.combineImages`
1110
- - `ly.img.plugin-ai-image-generation-web.quickAction.remixPage`
1111
- - `ly.img.plugin-ai-image-generation-web.quickAction.remixPageWithPrompt`
1112
-
1113
- **Text Quick Actions:**
1114
- - `ly.img.plugin-ai-text-generation-web.quickAction.improve`
1115
- - `ly.img.plugin-ai-text-generation-web.quickAction.fix`
1116
- - `ly.img.plugin-ai-text-generation-web.quickAction.shorter`
1117
- - `ly.img.plugin-ai-text-generation-web.quickAction.longer`
1118
- - `ly.img.plugin-ai-text-generation-web.quickAction.changeTone`
1119
- - `ly.img.plugin-ai-text-generation-web.quickAction.translate`
1120
- - `ly.img.plugin-ai-text-generation-web.quickAction.changeTextTo`
1121
-
1122
- **Video Quick Actions:**
1123
- - `ly.img.plugin-ai-video-generation-web.quickAction.createVideo`
1124
-
1125
- **Note:** Quick actions are automatically enabled when their plugin is loaded. Each quick action manages its own feature flag internally, ensuring proper initialization and registration.
1126
-
1127
- ##### Provider-Specific Style Features
1128
-
1129
- Some providers (like RecraftV3 and Recraft20b) support style groups that can be controlled independently:
1130
-
1131
- **RecraftV3 Style Groups:**
1132
- - `ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.style.image` - Enable/disable image styles (realistic, digital illustration)
1133
- - `ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.style.vector` - Enable/disable vector styles (vector illustration and variants)
1134
-
1135
- **Recraft20b Style Groups:**
1136
- - `ly.img.plugin-ai-image-generation-web.fal-ai/recraft/v2/text-to-image.style.image` - Enable/disable image styles
1137
- - `ly.img.plugin-ai-image-generation-web.fal-ai/recraft/v2/text-to-image.style.vector` - Enable/disable vector styles
1138
- - `ly.img.plugin-ai-image-generation-web.fal-ai/recraft/v2/text-to-image.style.icon` - Enable/disable icon styles
1139
-
1140
- When all style groups are disabled for a provider, it automatically falls back to the 'any' style. For more details on style control, see the [@imgly/plugin-ai-image-generation-web documentation](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-image-generation-web).
1141
-
1142
- ### Using with Existing AI Generation Plugins
1143
-
1144
- IMG.LY offers several pre-built AI generation packages that work with this base plugin:
1145
-
1146
- ```typescript
1147
- import CreativeEditorSDK from '@cesdk/cesdk-js';
1148
-
1149
- // Import plugin packages
1150
- import ImageGeneration from '@imgly/plugin-ai-image-generation-web';
1151
- import FalAiImage from '@imgly/plugin-ai-image-generation-web/fal-ai';
1152
- import VideoGeneration from '@imgly/plugin-ai-video-generation-web';
1153
- import FalAiVideo from '@imgly/plugin-ai-video-generation-web/fal-ai';
1154
-
1155
- // Initialize CreativeEditor SDK
1156
- CreativeEditorSDK.create(domElement, {
1157
- license: 'your-license-key'
1158
- }).then(async (cesdk) => {
1159
- // Add default asset sources
1160
- await cesdk.addDefaultAssetSources();
1161
-
1162
- // Image generation with Fal.ai models
1163
- cesdk.addPlugin(
1164
- ImageGeneration({
1165
- text2image: FalAiImage.RecraftV3({
1166
- proxyUrl: 'http://your-proxy-server.com/api/proxy'
1167
- }),
1168
- // Alternative: FalAiImage.Recraft20b({ proxyUrl: 'http://your-proxy-server.com/api/proxy' }),
1169
- image2image: FalAiImage.GeminiFlashEdit({
1170
- proxyUrl: 'http://your-proxy-server.com/api/proxy'
1171
- })
1172
- })
1173
- );
1174
-
1175
- // Video generation
1176
- cesdk.addPlugin(
1177
- VideoGeneration({
1178
- text2video: FalAiVideo.MinimaxVideo01Live({
1179
- proxyUrl: 'http://your-proxy-server.com/api/proxy'
1180
- })
1181
- })
1182
- );
1183
-
1184
- // Add quick action menus to canvas
1185
- cesdk.ui.setCanvasMenuOrder([
1186
- 'ly.img.plugin-ai-image-generation-web.canvasMenu',
1187
- 'ly.img.plugin-ai-video-generation-web.canvasMenu',
1188
- ...cesdk.ui.getCanvasMenuOrder()
1189
- ]);
1190
- });
1191
- ```
1192
-
1193
- ## Advanced Features
1194
-
1195
- ### Middleware
1196
-
1197
- The package includes a middleware system to augment the generation flow:
1198
-
1199
- #### Rate Limiting Middleware
1200
-
1201
- ```typescript
1202
- import { rateLimitMiddleware } from '@imgly/plugin-ai-generation-web';
1203
-
1204
- // Create a rate limiting middleware
1205
- const rateLimit = rateLimitMiddleware({
1206
- maxRequests: 10,
1207
- timeWindowMs: 60000, // 1 minute
1208
- onRateLimitExceeded: (input, options, info) => {
1209
- console.log(
1210
- `Rate limit exceeded: ${info.currentCount}/${info.maxRequests}`
1211
- );
1212
- return false; // Reject request
1213
- }
1214
- });
1215
-
1216
- // Apply middleware to your provider
1217
- const provider = {
1218
- // ...provider config
1219
- output: {
1220
- middleware: [rateLimit]
1221
- // ...other output config
1222
- }
1223
- };
1224
- ```
1225
-
1226
- **Note**: This middleware provides client-side rate limiting for UI purposes only. Always implement proper server-side rate limiting and authentication for production APIs.
1227
-
1228
- #### Upload Middleware
1229
-
1230
- The `uploadMiddleware` allows you to upload generated content to your own servers:
1231
-
1232
- ```typescript
1233
- import { uploadMiddleware } from '@imgly/plugin-ai-generation-web';
1234
-
1235
- // Create an upload middleware
1236
- const upload = uploadMiddleware(async (output) => {
1237
- // Upload the output to your server/storage
1238
- const response = await fetch('https://your-api.example.com/upload', {
1239
- method: 'POST',
1240
- headers: { 'Content-Type': 'application/json' },
1241
- body: JSON.stringify(output)
1242
- });
1243
-
1244
- const result = await response.json();
1245
-
1246
- // Return the output with the updated URL
1247
- return {
1248
- ...output,
1249
- url: result.url
1250
- };
1251
- });
1252
-
1253
- // Apply middleware to your provider
1254
- const provider = {
1255
- // ...provider config
1256
- output: {
1257
- middleware: [upload]
1258
- // ...other output config
1259
- }
1260
- };
1261
- ```
1262
-
1263
- #### Preventing Default Feedback
1264
-
1265
- Middleware can suppress default UI feedback behaviors (notifications, block states, console logging) using `options.preventDefault()`. This is useful when you want to handle success or error feedback yourself:
1266
-
1267
- ```typescript
1268
- const customErrorMiddleware: Middleware<any, any> = async (input, options, next) => {
1269
- try {
1270
- return await next(input, options);
1271
- } catch (error) {
1272
- // Prevent default error notification and block error state
1273
- options.preventDefault();
1274
-
1275
- // When preventDefault() is called, you need to handle the block state yourself.
1276
- // Here we set the error state to mimic the default behavior:
1277
- options.blockIds?.forEach(blockId => {
1278
- if (options.engine.block.isValid(blockId)) {
1279
- options.engine.block.setState(blockId, { type: 'Error', error: 'Unknown' });
1280
- }
1281
- });
1282
- // Alternative: Delete the placeholder block instead
1283
- // options.blockIds?.forEach(blockId => {
1284
- // if (options.engine.block.isValid(blockId)) {
1285
- // options.engine.block.destroy(blockId);
1286
- // }
1287
- // });
1288
-
1289
- // Show custom notification
1290
- options.cesdk?.ui.showNotification({
1291
- type: 'error',
1292
- message: `Custom error: ${error.message}`,
1293
- duration: 5000,
1294
- action: {
1295
- label: 'Contact Support',
1296
- onClick: () => window.open('mailto:support@example.com')
1297
- }
1298
- });
1299
-
1300
- throw error;
1301
- }
1302
- };
1303
- ```
1304
-
1305
- **What gets prevented:**
1306
- - Error/success notifications (toast messages)
1307
- - Block error state (error icon)
1308
- - Console error logging
1309
-
1310
- **What is NOT prevented:**
1311
- - Pending → Ready transition (loading spinner always stops)
1312
-
1313
- **Common use cases:**
1314
-
1315
- **1. Custom Error Notifications:**
1316
- ```typescript
1317
- const middleware = async (input, options, next) => {
1318
- try {
1319
- return await next(input, options);
1320
- } catch (error) {
1321
- options.preventDefault();
1322
-
1323
- // When preventDefault() is called, you need to handle the block state yourself.
1324
- // Here we set the error state to mimic the default behavior:
1325
- options.blockIds?.forEach(blockId => {
1326
- if (options.engine.block.isValid(blockId)) {
1327
- options.engine.block.setState(blockId, { type: 'Error', error: 'Unknown' });
1328
- }
1329
- });
1330
-
1331
- options.cesdk?.ui.showNotification({
1332
- type: 'error',
1333
- message: `Generation failed: ${error.message}`,
1334
- action: { label: 'Retry', onClick: () => retry() }
1335
- });
1336
- throw error;
1337
- }
1338
- };
1339
- ```
1340
-
1341
- **2. Silent Failures with External Logging:**
1342
- ```typescript
1343
- const middleware = async (input, options, next) => {
1344
- try {
1345
- return await next(input, options);
1346
- } catch (error) {
1347
- options.preventDefault();
1348
- errorTracker.capture({ error, context: { input, blockIds: options.blockIds } });
1349
- throw error;
1350
- }
1351
- };
1352
- ```
1353
-
1354
- **3. Retry Logic:**
1355
- ```typescript
1356
- const retryMiddleware = async (input, options, next) => {
1357
- const maxRetries = 3;
1358
- let attempt = 0;
1359
-
1360
- while (attempt < maxRetries) {
1361
- try {
1362
- return await next(input, options);
1363
- } catch (error) {
1364
- attempt++;
1365
- if (attempt < maxRetries) {
1366
- options.preventDefault();
1367
- options.cesdk?.ui.showNotification({
1368
- type: 'info',
1369
- message: `Retrying... (${attempt}/${maxRetries})`
1370
- });
1371
- await new Promise(resolve => setTimeout(resolve, 1000));
1372
- } else {
1373
- options.preventDefault();
1374
-
1375
- // When preventDefault() is called, you need to handle the block state yourself.
1376
- // After final retry failure, you might want to delete the placeholder:
1377
- options.blockIds?.forEach(blockId => {
1378
- if (options.engine.block.isValid(blockId)) {
1379
- options.engine.block.destroy(blockId); // Remove failed placeholder
1380
- }
1381
- });
1382
-
1383
- options.cesdk?.ui.showNotification({
1384
- type: 'error',
1385
- message: `Failed after ${maxRetries} attempts`
1386
- });
1387
- throw error;
1388
- }
1389
- }
1390
- }
1391
- };
1392
- ```
1393
-
1394
- ### Provider Registry
1395
-
1396
- The `ProviderRegistry` is a global singleton that manages all registered providers:
1397
-
1398
- ```typescript
1399
- import { ProviderRegistry } from '@imgly/plugin-ai-generation-web';
1400
-
1401
- // Get the global registry
1402
- const registry = ProviderRegistry.get();
1403
-
1404
- // Get all registered providers
1405
- const allProviders = registry.getAll();
1406
-
1407
- // Get providers by kind
1408
- const imageProviders = registry.getByKind('image');
1409
-
1410
- // Find a specific provider
1411
- const myProvider = registry.getById('my-provider-id');
1412
- ```
1413
-
1414
- ## TypeScript Support
1415
-
1416
- This package is fully typed with TypeScript, providing excellent IntelliSense support during development:
1417
-
1418
- - **Generic Provider Types**: Strongly typed providers with input/output validation
1419
- - **Quick Action Types**: Type-safe quick action definitions with proper input mapping
1420
- - **Registry Types**: Fully typed action and provider registries
1421
- - **Middleware Types**: Typed middleware functions for better composition
1422
-
1423
- ## API Reference
1424
-
1425
- ### Core Exports
1426
-
1427
- ```typescript
1428
- // Provider types and interfaces
1429
- export { Provider, ImageOutput, VideoOutput, AudioOutput, TextOutput } from './core/provider';
1430
-
1431
- // Action registry
1432
- export { ActionRegistry, QuickActionDefinition, PluginActionDefinition } from './core/ActionRegistry';
1433
-
1434
- // Provider registry
1435
- export { ProviderRegistry } from './core/ProviderRegistry';
1436
-
1437
- // Initialization functions
1438
- export { initializeProvider, initializeProviders } from './providers/';
1439
-
1440
- // Middleware
1441
- export { loggingMiddleware, rateLimitMiddleware, uploadMiddleware } from './middleware/';
1442
-
1443
- // Utilities
1444
- export { getPanelId, enableQuickActionForImageFill, mergeQuickActionsConfig } from './utils/';
1445
- ```
1446
-
1447
- ### Initialization Functions
1448
-
1449
- #### initializeProviders
1450
-
1451
- The `initializeProviders` function is used to initialize multiple providers at once. It creates a composite history asset source and library entry for all providers of the same kind.
1452
-
1453
- ```typescript
1454
- const result = await initializeProviders(
1455
- providers,
1456
- { engine, cesdk },
1457
- config
1458
- );
1459
-
1460
- // Return value structure:
1461
- {
1462
- panel: {
1463
- builderRenderFunction: Function // UI builder function for provider selection
1464
- },
1465
- history: {
1466
- assetSourceId: string, // ID of the composite history asset source
1467
- assetLibraryEntryId: string // ID of the automatically created asset library entry
1468
- },
1469
- providerInitializationResults: Array<{
1470
- provider: Provider,
1471
- result: ProviderInitializationResult
1472
- }>
1473
- }
1474
- ```
1475
-
1476
- **Key Points:**
1477
- - Creates a composite history asset source with ID format: `ly.img.plugin-ai-{kind}-generation-web.history`
1478
- - Automatically creates an asset library entry with the same ID as the asset source
1479
- - The library entry is configured with appropriate settings (sortBy: insertedAt descending, canRemove: true, etc.)
1480
- - Returns both the asset source ID and library entry ID for reference
1481
-
1482
- ### Common Types
1483
-
1484
- ```typescript
1485
- // Provider configuration
1486
- interface CommonProviderConfiguration<I, O extends Output> {
1487
- proxyUrl: string;
1488
- debug?: boolean;
1489
- middleware?: Middleware<I, O>[];
1490
- headers?: Record<string, string>;
1491
- history?: false | '@imgly/local' | '@imgly/indexedDB' | (string & {});
1492
- supportedQuickActions?: {
1493
- [quickActionId: string]: Partial<QuickActionSupport<I>> | false | null;
1494
- };
1495
- }
1496
-
1497
- // Quick action definition
1498
- interface QuickActionDefinition<Q extends Record<string, any>> {
1499
- id: string;
1500
- type: 'quick';
1501
- kind: OutputKind;
1502
- label?: string;
1503
- enable: boolean | ((context: { engine: CreativeEngine }) => boolean);
1504
- render: (context: QuickActionRenderContext<Q>) => void;
1505
- }
1506
- ```
1507
-
1508
- ## Internationalization (i18n)
1509
-
1510
- The AI plugins support full internationalization, allowing integrators to customize all user-facing text. The translation system is designed to let integrators override default translations **before** plugins are loaded.
1511
-
1512
- ### Custom Translations
1513
-
1514
- To customize translations, call `cesdk.i18n.setTranslations()` **before** adding the AI plugins:
1515
-
1516
- ```typescript
1517
- import CreativeEditorSDK from '@cesdk/cesdk-js';
1518
- import AiApps from '@imgly/plugin-ai-apps-web';
1519
-
1520
- CreativeEditorSDK.create(domElement, {
1521
- license: 'your-license-key',
1522
- locale: 'de' // Set your desired locale
1523
- }).then(async (cesdk) => {
1524
- // Set custom translations BEFORE adding plugins
1525
- cesdk.i18n.setTranslations({
1526
- en: {
1527
- // Override AI Apps labels
1528
- '@imgly/plugin-ai-image-generation-web.action.label': 'Create Image',
1529
- '@imgly/plugin-ai-video-generation-web.action.label': 'Create Video',
1530
-
1531
- // Override provider-specific labels
1532
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.style': 'Art Style',
1533
- 'ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.style.realistic_image': 'Photo Realistic'
1534
- },
1535
- de: {
1536
- // German translations
1537
- '@imgly/plugin-ai-image-generation-web.action.label': 'Bild erstellen',
1538
- '@imgly/plugin-ai-video-generation-web.action.label': 'Video erstellen',
1539
- 'common.generate': 'Generieren',
1540
- 'panel.ly.img.ai.apps': 'KI'
1541
- }
1542
- });
1543
-
1544
- // Now add the plugins - they won't override your custom translations
1545
- await cesdk.addPlugin(AiApps({ providers: { /* ... */ } }));
1546
- });
1547
- ```
1548
-
1549
- ### How It Works
1550
-
1551
- The AI plugins use `setDefaultTranslations()` internally, which only sets translation keys that don't already exist. This means:
1552
-
1553
- 1. **Your translations take priority**: Any translations you set before loading plugins are preserved
1554
- 2. **Fallback to defaults**: Keys you don't customize will use the plugin's default English translations
1555
- 3. **Locale support**: Set translations for any locale supported by CE.SDK
1556
-
1557
- ### Translation Key Structure
1558
-
1559
- Translation keys follow a consistent naming pattern:
1560
-
1561
- ```
1562
- ly.img.plugin-ai-{type}-generation-web.{provider-id}.property.{property-name}.{value}
1563
- ```
1564
-
1565
- For example:
1566
- - `ly.img.plugin-ai-image-generation-web.fal-ai/recraft-v3.property.style.realistic_image`
1567
- - `ly.img.plugin-ai-video-generation-web.fal-ai/veo-3.property.duration.5s`
1568
-
1569
- ### Action Labels
1570
-
1571
- Each AI plugin registers an action with a translatable label. The action uses both a static `label` (for backwards compatibility) and a `labelKey` for dynamic i18n resolution:
1572
-
1573
- | Plugin | Label Key |
1574
- |--------|-----------|
1575
- | Image Generation | `@imgly/plugin-ai-image-generation-web.action.label` |
1576
- | Video Generation | `@imgly/plugin-ai-video-generation-web.action.label` |
1577
- | Audio Generation (Sound) | `@imgly/plugin-ai-audio-generation-web.sound.action.label` |
1578
- | Audio Generation (Speech) | `@imgly/plugin-ai-audio-generation-web.speech.action.label` |
1579
- | Sticker Generation | `@imgly/plugin-ai-sticker-generation-web.action.label` |
1580
-
1581
- ### Translation Files
1582
-
1583
- Each AI plugin includes a `translations.json` file with all available translation keys:
1584
-
1585
- - [Base translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-generation-web/translations.json) - Core translation keys
1586
- - [AI Apps translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-apps-web/translations.json) - AI Apps panel labels
1587
- - [Image generation translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-image-generation-web/translations.json) - Image generation interfaces
1588
- - [Video generation translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-video-generation-web/translations.json) - Video generation interfaces
1589
- - [Text generation translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-text-generation-web/translations.json) - Text generation interfaces
1590
- - [Audio generation translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-audio-generation-web/translations.json) - Audio generation interfaces
1591
- - [Sticker generation translations](https://github.com/imgly/plugins/tree/main/packages/plugin-ai-sticker-generation-web/translations.json) - Sticker generation interfaces
1592
-
1593
- ### Exported Utilities
1594
-
1595
- The package exports translation utilities for use in custom providers:
1596
-
1597
- ```typescript
1598
- import {
1599
- setDefaultTranslations,
1600
- createTranslationCallback,
1601
- buildTranslationKeys
1602
- } from '@imgly/plugin-ai-generation-web';
1603
-
1604
- // Set translations that won't override existing keys
1605
- setDefaultTranslations(cesdk, {
1606
- en: { 'my.key': 'My Value' }
1607
- });
1608
-
1609
- // Create a translation callback for asset sources
1610
- const translateLabel = createTranslationCallback(cesdk, 'my-provider', 'style', 'image');
1611
-
1612
- // Build translation keys with fallback order
1613
- const keys = buildTranslationKeys('my-provider', 'style', 'realistic', 'image');
1614
- // Returns: [
1615
- // 'ly.img.plugin-ai-image-generation-web.my-provider.property.style.realistic',
1616
- // 'ly.img.plugin-ai-generation-web.property.style.realistic',
1617
- // 'ly.img.plugin-ai-image-generation-web.my-provider.defaults.property.style.realistic',
1618
- // 'ly.img.plugin-ai-generation-web.defaults.property.style.realistic'
1619
- // ]
1620
- ```
9
+ This plugin is part of the IMG.LY plugin ecosystem for CreativeEditor SDK.