massgen 0.1.3__py3-none-any.whl → 0.1.4__py3-none-any.whl

massgen/tool/docs/multimodal_tools.md

@@ -769,6 +769,594 @@ Error: File does not appear to be a video file
 
 ---
 
+---
+
+## Image Generation Tools
+
+### text_to_image_generation
+
+**What it does**: Generates images from text descriptions using OpenAI's GPT-4.1 API **WITHOUT ANY INPUT IMAGES**. Creates new images from scratch based solely on text prompts.
+
+**Why use it**: Allows agents to create original visual content from descriptions. Perfect for generating illustrations, concept art, product visualizations, or any creative visual content.
+
+**Location**: `massgen.tool._multimodal_tools.text_to_image_generation`
+
+#### Parameters
+
+- `prompt` (required): Text description of the image to generate
+  - Be specific and detailed for better results
+  - Include style, composition, lighting, and mood details
+- `model` (optional): Model to use (default: "gpt-4.1")
+  - Options: "gpt-4.1"
+- `storage_path` (optional): Directory path where to save the image
+  - **IMPORTANT**: Must be a DIRECTORY path only, NOT a file path
+  - Example: "images/generated" NOT "images/cat.png"
+  - Filename is automatically generated from prompt and timestamp
+  - Relative path: Resolved relative to agent's workspace
+  - Absolute path: Must be within allowed directories
+  - None/empty: Saves to agent's workspace root
+- `allowed_paths` (optional): List of allowed base paths for validation
+
+#### Returns
+
+ExecutionResult containing:
+- `success`: Whether operation succeeded
+- `operation`: "generate_and_store_image_no_input_images"
+- `note`: Note about operation
+- `images`: List of generated images with file paths and metadata
+- `model`: Model used for generation
+- `prompt`: The prompt used for generation
+- `total_images`: Total number of images generated and saved
+
+#### Security Features
+
+- Requires valid OpenAI API key
+- Files are saved to specified path within workspace
+- Path must be within allowed directories
+- Automatic timestamp-based filename generation
+
+#### Examples
+
+**Basic Image Generation**:
+
+```python
+from massgen.tool._multimodal_tools import text_to_image_generation
+
+# Generate an image from a text description
+result = await text_to_image_generation(prompt="a cat in space")
+
+# Output includes file path and metadata
+print(result.output_blocks[0].data)
+# {
+#   "success": true,
+#   "operation": "generate_and_store_image_no_input_images",
+#   "images": [{
+#     "file_path": "/workspace/20240115_143022_a_cat_in_space.png",
+#     "filename": "20240115_143022_a_cat_in_space.png",
+#     "size": 125340
+#   }],
+#   "total_images": 1
+# }
+```
+
+**Custom Storage Path**:
+
+```python
+# Generate with custom storage location
+result = await text_to_image_generation(
+    prompt="sunset over mountains",
+    storage_path="art/landscapes"
+)
+```
+
+**Configuration Example**:
+
+```yaml
+# massgen/configs/tools/custom_tools/multimodal_tools/text_to_image_generation_single.yaml
+agents:
+  - id: "image_generator"
+    backend:
+      type: "openai"
+      model: "gpt-4o"
+      cwd: "workspace1"
+      enable_image_generation: true
+      custom_tools:
+        - name: ["text_to_image_generation"]
+          category: "multimodal"
+          path: "massgen/tool/_multimodal_tools/text_to_image_generation.py"
+          function: ["text_to_image_generation"]
+```
+
+**CLI Usage**:
+
+```bash
+massgen --config massgen/configs/tools/custom_tools/multimodal_tools/text_to_image_generation_single.yaml "Generate an image of a futuristic city at night"
+```
+
+---
+
+### image_to_image_generation
+
+**What it does**: Creates variations based on multiple input images using OpenAI's GPT-4.1 API. Generates new images inspired by existing ones.
+
+**Why use it**: Allows agents to create variations, mashups, or transformations of existing images. Perfect for style transfer, image editing, or creating variations of designs.
+
+**Location**: `massgen.tool._multimodal_tools.image_to_image_generation`
+
+#### Parameters
+
+- `base_image_paths` (required): List of paths to base images
+  - Supported formats: PNG, JPEG (less than 4MB each)
+  - Relative paths: Resolved relative to workspace
+  - Absolute paths: Must be within allowed directories
+- `prompt` (optional): Text description for the variation (default: "Create a variation of the provided images")
+- `model` (optional): Model to use (default: "gpt-4.1")
+- `storage_path` (optional): Directory path where to save variations
+  - **IMPORTANT**: Must be a DIRECTORY path only
+  - Filename is automatically generated
+- `allowed_paths` (optional): List of allowed base paths for validation
+
+#### Returns
+
+ExecutionResult containing:
+- `success`: Whether operation succeeded
+- `operation`: "generate_and_store_image_with_input_images"
+- `note`: Note about usage
+- `images`: List of generated images with file paths and metadata
+- `model`: Model used for generation
+- `prompt`: The prompt used
+- `total_images`: Total number of images generated
+
+#### Security Features
+
+- Requires valid OpenAI API key
+- Input images must be valid image files less than 4MB
+- Files are saved to specified path within workspace
+- Path validation for security
+
+#### Examples
+
+**Create Image Variation**:
+
+```python
+from massgen.tool._multimodal_tools import image_to_image_generation
+
+# Generate variation from a single image
+result = await image_to_image_generation(
+    base_image_paths=["logo.png"],
+    prompt="Create a modern variation of this logo"
+)
+```
+
+**Combine Multiple Images**:
+
+```python
+# Generate variation combining multiple images
+result = await image_to_image_generation(
+    base_image_paths=["cat.png", "dog.png"],
+    prompt="Combine these animals into a single creature"
+)
+```
+
+---
+
+## Video Generation Tools
+
+### text_to_video_generation
+
+**What it does**: Generates videos from text descriptions using OpenAI's Sora-2 API. Creates high-quality video content from detailed scene descriptions.
+
+**Why use it**: Allows agents to create video content from descriptions. Perfect for marketing content, concept visualization, educational videos, or social media content.
+
+**Location**: `massgen.tool._multimodal_tools.text_to_video_generation`
+
+#### Parameters
+
+- `prompt` (required): Text description for the video to generate
+  - Include scene details, camera movements, lighting, atmosphere
+  - Be specific about actions, objects, and environment
+- `model` (optional): Model to use (default: "sora-2")
+- `seconds` (optional): Video duration in seconds (default: 4)
+  - Supported range: 4-20 seconds
+- `storage_path` (optional): Directory path where to save the video
+  - **IMPORTANT**: Must be a DIRECTORY path only
+  - Filename is automatically generated from prompt and timestamp
+  - Relative path: Resolved relative to workspace
+  - Absolute path: Must be within allowed directories
+- `allowed_paths` (optional): List of allowed base paths for validation
+
+#### Returns
+
+ExecutionResult containing:
+- `success`: Whether operation succeeded
+- `operation`: "generate_and_store_video_no_input_images"
+- `video_path`: Path to the saved video file
+- `filename`: Name of the generated file
+- `size`: File size in bytes
+- `model`: Model used for generation
+- `prompt`: The prompt used
+- `duration`: Time taken for generation in seconds
+
+#### Security Features
+
+- Requires valid OpenAI API key with Sora-2 access
+- Files are saved to specified path within workspace
+- Automatic video download and storage
+
+#### Examples
+
+**Basic Video Generation**:
+
+```python
+from massgen.tool._multimodal_tools import text_to_video_generation
+
+# Generate a 4-second video
+result = await text_to_video_generation(
+    prompt="A cool cat on a motorcycle in the night"
+)
+
+# Output includes video path
+print(result.output_blocks[0].data)
+# {
+#   "success": true,
+#   "operation": "generate_and_store_video_no_input_images",
+#   "video_path": "/workspace/20240115_143022_a_cool_cat_on_motorcycle.mp4",
+#   "size": 5242880,
+#   "duration": 45.2
+# }
+```
+
+**Detailed Scene with Duration**:
+
+```python
+# Generate with detailed prompt and custom duration
+result = await text_to_video_generation(
+    prompt="Neon-lit alley at night, light rain, slow push-in, cinematic lighting",
+    seconds=10,
+    storage_path="videos/cinematic"
+)
+```
+
+**Configuration Example**:
+
+```yaml
+# massgen/configs/tools/custom_tools/multimodal_tools/text_to_video_generation_single.yaml
+agents:
+  - id: "video_generator"
+    backend:
+      type: "openai"
+      model: "gpt-4o"
+      cwd: "workspace1"
+      enable_video_generation: true
+      custom_tools:
+        - name: ["text_to_video_generation"]
+          category: "multimodal"
+          path: "massgen/tool/_multimodal_tools/text_to_video_generation.py"
+          function: ["text_to_video_generation"]
+```
+
+**CLI Usage**:
+
+```bash
+massgen --config massgen/configs/tools/custom_tools/multimodal_tools/text_to_video_generation_single.yaml "Generate a 4 seconds video with neon-lit alley at night, light rain, slow push-in, cinematic."
+```
+
+---
+
+## Audio/Speech Generation Tools
+
+### text_to_speech_continue_generation
+
+**What it does**: Generates expressive speech from text using OpenAI's GPT-4o Audio Preview model. Creates natural-sounding speech with emotional expression and context awareness.
+
+**Why use it**: Allows agents to generate expressive speech with emotional tone. Perfect for creating voice-overs, narrations, audiobooks, or any content requiring natural, emotional speech.
+
+**Location**: `massgen.tool._multimodal_tools.text_to_speech_continue_generation`
+
+#### Parameters
+
+- `prompt` (required): Text content to convert to audio speech
+- `model` (optional): Model to use (default: "gpt-4o-audio-preview")
+- `voice` (optional): Voice to use (default: "alloy")
+  - Options: "alloy", "echo", "fable", "onyx", "nova", "shimmer"
+- `audio_format` (optional): Audio format for output (default: "wav")
+  - Options: "wav", "mp3", "opus", "aac", "flac"
+- `storage_path` (optional): Directory path where to save the audio
+  - **IMPORTANT**: Must be a DIRECTORY path only
+  - Filename is automatically generated from prompt and timestamp
+- `allowed_paths` (optional): List of allowed base paths for validation
+
+#### Returns
+
+ExecutionResult containing:
+- `success`: Whether operation succeeded
+- `operation`: "generate_and_store_audio_no_input_audios"
+- `audio_file`: Generated audio file with path and metadata
+- `model`: Model used for generation
+- `prompt`: The prompt used for generation
+- `voice`: Voice used for generation
+- `format`: Audio format used
+
+#### Examples
+
+**Expressive Speech**:
+
+```python
+from massgen.tool._multimodal_tools import text_to_speech_continue_generation
+
+# Generate expressive speech
+result = await text_to_speech_continue_generation(
+    prompt="I want you to tell me a very short introduction about Sherlock Holmes in one sentence, and I want you to use emotion voice to read it out loud."
+)
+```
+
+**Custom Voice and Format**:
+
+```python
+# Generate with specific voice and format
+result = await text_to_speech_continue_generation(
+    prompt="Hello world",
+    voice="nova",
+    audio_format="mp3",
+    storage_path="audio/generated"
+)
+```
+
+---
+
+### text_to_speech_transcription_generation
+
+**What it does**: Converts text directly to speech using OpenAI's TTS API with streaming response. Provides fast, cost-effective text-to-speech conversion.
+
+**Why use it**: Allows agents to quickly convert text to speech. Perfect for transcription conversion, simple voice-overs, or when expressive emotion is not required.
+
+**Location**: `massgen.tool._multimodal_tools.text_to_speech_transcription_generation`
+
+#### Parameters
+
+- `input_text` (required): The text content to convert to speech
+- `model` (optional): TTS model to use (default: "gpt-4o-mini-tts")
+  - Options: "gpt-4o-mini-tts", "tts-1", "tts-1-hd"
+- `voice` (optional): Voice to use (default: "alloy")
+  - Options: "alloy", "echo", "fable", "onyx", "nova", "shimmer", "coral", "sage"
+- `instructions` (optional): Optional speaking instructions for tone and style
+  - Example: "Speak in a cheerful tone"
+- `storage_path` (optional): Directory path where to save the audio file
+  - **IMPORTANT**: Must be a DIRECTORY path only
+- `audio_format` (optional): Output audio format (default: "mp3")
+  - Options: "mp3", "opus", "aac", "flac", "wav", "pcm"
+- `allowed_paths` (optional): List of allowed base paths for validation
+
+#### Returns
+
+ExecutionResult containing:
+- `success`: Whether operation succeeded
+- `operation`: "convert_text_to_speech"
+- `audio_file`: Generated audio file with path and metadata
+- `model`: TTS model used
+- `voice`: Voice used
+- `format`: Audio format used
+- `text_length`: Length of input text
+- `instructions`: Speaking instructions if provided
+
+#### Examples
+
+**Simple Text-to-Speech**:
+
+```python
+from massgen.tool._multimodal_tools import text_to_speech_transcription_generation
+
+# Convert text to speech
+result = await text_to_speech_transcription_generation(
+    input_text="Hello world, this is a test."
+)
+```
+
+**With Instructions**:
+
+```python
+# Convert with specific voice and instructions
+result = await text_to_speech_transcription_generation(
+    input_text="Today is a wonderful day to build something people love!",
+    voice="coral",
+    instructions="Speak in a cheerful and positive tone."
+)
+```
+
+---
+
+## File Generation Tools
+
+### text_to_file_generation
+
+**What it does**: Generates text content using OpenAI API and saves it as various file formats (TXT, MD, PDF, PPTX). Creates professional documents from text prompts.
+
+**Why use it**: Allows agents to create formatted documents automatically. Perfect for generating reports, documentation, presentations, or any structured text content.
+
+**Location**: `massgen.tool._multimodal_tools.text_to_file_generation`
+
+#### Parameters
+
+- `prompt` (required): Description of the content to generate
+  - Be specific about structure, sections, and formatting
+  - Example: "Write a technical report about AI"
+- `file_format` (optional): Output file format (default: "txt")
+  - Options: "txt", "md", "pdf", "pptx"
+- `filename` (optional): Custom filename without extension
+  - If not provided, generates from prompt and timestamp
+- `model` (optional): OpenAI model to use (default: "gpt-4o")
+  - Options: "gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-3.5-turbo"
+- `storage_path` (optional): Directory path where to save the file
+  - **IMPORTANT**: Must be a DIRECTORY path only
+  - Filename is automatically generated from prompt or custom filename
+  - Relative path: Resolved relative to workspace
+  - Absolute path: Must be within allowed directories
+- `allowed_paths` (optional): List of allowed base paths for validation
+
+#### Returns
+
+ExecutionResult containing:
+- `success`: Whether operation succeeded
+- `operation`: "generate_and_store_file"
+- `file_path`: Path to the generated file
+- `filename`: Name of the generated file
+- `file_format`: Format of the generated file
+- `content_preview`: First 500 characters of generated content
+- `file_size`: Size of the generated file in bytes
+- `model`: Model used for generation
+- `prompt`: The prompt used
+
+#### Security Features
+
+- Requires valid OpenAI API key
+- Files are saved to specified path within workspace
+- Path must be within allowed directories
+
+#### Dependencies
+
+- PDF generation requires either `reportlab` or `fpdf2` library
+- PPTX generation requires `python-pptx` library
+
+```bash
+pip install reportlab  # For PDF
+pip install python-pptx  # For PPTX
+```
+
+#### Examples
+
+**Generate Markdown Document**:
+
+```python
+from massgen.tool._multimodal_tools import text_to_file_generation
+
+# Generate a markdown file
+result = await text_to_file_generation(
+    prompt="Write a blog post about Python",
+    file_format="md"
+)
+```
+
+**Generate PDF Report**:
+
+```python
+# Generate a PDF with custom filename
+result = await text_to_file_generation(
+    prompt="Create a technical report on machine learning",
+    file_format="pdf",
+    filename="ml_report",
+    storage_path="documents/reports"
+)
+```
+
+**Generate PowerPoint Presentation**:
+
+```python
+# Generate PPTX - structure prompt with slide titles (# or ##) and bullet points (-)
+result = await text_to_file_generation(
+    prompt="""Create a presentation about AI trends:
+
+    # Introduction
+    - Overview of AI landscape
+    - Key developments in 2024
+
+    # Current Trends
+    - Large Language Models
+    - Multimodal AI
+    - AI Safety
+
+    # Future Outlook
+    - Predictions for 2025
+    - Emerging technologies
+    """,
+    file_format="pptx",
+    filename="ai_trends_presentation"
+)
+```
+
+**Configuration Example**:
+
+```yaml
+# massgen/configs/tools/custom_tools/multimodal_tools/text_to_file_generation_single.yaml
+agents:
+  - id: "document_generator"
+    backend:
+      type: "openai"
+      model: "gpt-4o"
+      cwd: "workspace1"
+      enable_file_generation: true
+      custom_tools:
+        - name: ["text_to_file_generation"]
+          category: "multimodal"
+          path: "massgen/tool/_multimodal_tools/text_to_file_generation.py"
+          function: ["text_to_file_generation"]
+```
+
+**CLI Usage**:
+
+```bash
+massgen --config massgen/configs/tools/custom_tools/multimodal_tools/text_to_file_generation_single.yaml "Generate a technical report about LLMs and save as PDF"
+```
+
+#### Note
+
+- For PPTX format, structure your prompt to include slide titles (using # or ##) and bullet points (using -)
+- The quality and format of generated content depends on the prompt
+- Longer content may consume more tokens
+
+---
+
+## Best Practices for Generation Tools
+
+### Image Generation
+
+1. **Prompt Quality**:
+   - Be specific about style, composition, lighting, and mood
+   - Include details about colors, perspective, and atmosphere
+   - Use artistic terminology for better results
+
+2. **Cost Management**:
+   - Image generation (GPT-4.1) is more expensive than standard API calls
+   - Test prompts with understanding tools first
+   - Use multi-agent workflows to refine prompts before generation
+
+### Video Generation
+
+1. **Prompt Structure**:
+   - Include: setting, lighting, camera movements, atmosphere
+   - Specify duration based on content complexity (4-20 seconds)
+   - Use cinematic terminology (push-in, pull-out, pan, etc.)
+
+2. **Quality Verification**:
+   - Combine with `understand_video` tool for quality checks
+   - Use multi-agent workflows for iterative refinement
+
+### Audio/Speech Generation
+
+1. **Voice Selection**:
+   - Choose appropriate voice for content type
+   - Use expressive model (gpt-4o-audio-preview) for emotional content
+   - Use TTS model (gpt-4o-mini-tts) for simple conversions
+
+2. **Format Selection**:
+   - WAV for highest quality
+   - MP3 for balanced quality/size
+   - OPUS for web streaming
+
+### Document Generation
+
+1. **Format Selection**:
+   - TXT for simple content
+   - MD for formatted documentation
+   - PDF for professional documents
+   - PPTX for presentations
+
+2. **Prompt Structure**:
+   - Outline structure clearly
+   - Specify sections and formatting
+   - For PPTX, use markdown-style headers and bullets
+
+---
+
 ## Additional Resources
 
 - [OpenAI API Documentation](https://platform.openai.com/docs)
@@ -777,3 +1365,4 @@ Error: File does not appear to be a video file
 - [python-docx Documentation](https://python-docx.readthedocs.io/)
 - [openpyxl Documentation](https://openpyxl.readthedocs.io/)
 - [python-pptx Documentation](https://python-pptx.readthedocs.io/)
+- [reportlab Documentation](https://www.reportlab.com/documentation/)