@botpress/adk-cli 1.19.0-beta.4 → 1.19.0-beta.5

package/dist/files-dzt0wxn3.md

@@ -0,0 +1,918 @@
+# Files API
+
+The Files API provides structured file storage with automatic indexing, semantic search capabilities, and CDN-backed content delivery.
+
+## Accessing the Files API
+
+All file operations use the Botpress client. Get the client using the Context API:
+
+```typescript
+import { context } from "@botpress/runtime";
+
+const client = context.get("client");
+```
+
+This pattern is used in all examples below. See **[Context API](./context-api.md)** for more details.
+
+## Basic Concepts
+
+### What is the Files API?
+- **Persistent storage**: Upload and manage files for your bot
+- **Semantic search**: Optional vector indexing for searchable content
+- **CDN delivery**: Fast file access through CloudFront
+- **Access control**: Fine-grained permissions and policies
+- **Metadata & tags**: Organize files with custom attributes
+
+### File Properties
+
+Files have the following core properties:
+- **id**: Unique file identifier
+- **key**: User-defined unique key per bot
+- **url**: CDN URL for file content access
+- **size**: File size in bytes
+- **contentType**: MIME type (e.g., "application/pdf")
+- **tags**: Key-value pairs for organization
+- **metadata**: Custom JSON metadata
+- **accessPolicies**: Array of access control policies
+- **status**: Upload/indexing status
+- **createdAt/updatedAt**: Timestamps
+
+## Creating and Uploading Files
+
+### Basic File Upload
+
+```typescript
+const { file } = await client.uploadFile({
+  key: "documents/report-2024.pdf",           // Unique key
+  content: fileBuffer,                        // File content (Buffer/ArrayBuffer/Blob/string)
+  contentType: "application/pdf",             // MIME type
+  tags: {
+    category: "reports",
+    year: "2024"
+  },
+  index: true,                                 // Enable semantic search
+  accessPolicies: ["public_content"]          // Make publicly accessible
+});
+
+console.log("File ID:", file.id);
+console.log("File URL:", file.url);
+```
+
+### Upload Options
+
+```typescript
+await client.uploadFile({
+  key: "unique-file-key",                     // Required: Unique identifier
+  content: fileBuffer,                        // Required: File content
+  contentType: "text/plain",                  // Optional: MIME type
+
+  // Indexing options
+  index: true,                                // Enable semantic search
+  indexing: {
+    configuration: {
+      vision: {
+        transcribePages: true,                // OCR for images/PDFs
+        indexPages: "all"                     // Index all pages
+      }
+    }
+  },
+
+  // Access control
+  accessPolicies: [
+    "public_content",                         // Public URL access
+    "integrations"                            // Integration access
+  ],
+
+  // Organization
+  tags: {
+    category: "documents",
+    department: "sales"
+  },
+
+  // Custom metadata
+  metadata: {
+    author: "John Doe",
+    version: "1.0"
+  },
+
+  // Expiry (max 90 days)
+  expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000).toISOString(),
+
+  // Immediate public access (requires public_content policy)
+  publicContentImmediatelyAccessible: true
+});
+```
+
+### Uploading Text Content
+
+For simple text/JSON content:
+
+```typescript
+const { file } = await client.uploadFile({
+  key: "config/settings.json",
+  content: jsonContent,                       // String content
+  contentType: "application/json",
+  index: false
+});
+```
+
+### Uploading from URL
+
+To upload from an external URL:
+
+```typescript
+const { file } = await client.uploadFile({
+  key: "imported/image.jpg",
+  url: "https://example.com/image.jpg",       // Fetch from URL
+  contentType: "image/jpeg",
+  index: false
+});
+```
+
+## Retrieving Files
+
+### Get File Metadata
+
+```typescript
+// Get by ID
+const { file } = await client.getFile({
+  id: "file_abc123"
+});
+
+// Get by key
+const { file } = await client.getFile({
+  id: "documents/report-2024.pdf"  // Can use key as ID
+});
+
+console.log("File URL:", file.url);
+console.log("File size:", file.size);
+console.log("Status:", file.status);
+```
+
+### Access File Content
+
+Files are accessed via URL. The URL type depends on access policies:
+
+```typescript
+const { file } = await client.getFile({ id: fileId });
+
+// For public files: permanent CDN URL
+// For private files: temporary presigned S3 URL
+
+const response = await fetch(file.url);
+const content = await response.text();
+
+// For JSON files
+const data = await response.json();
+
+// For binary files
+const buffer = await response.arrayBuffer();
+```
+
+### File Status States
+
+```typescript
+type FileStatus =
+  | "upload_pending"      // Awaiting content upload
+  | "upload_completed"    // Upload successful
+  | "upload_failed"       // Upload failed
+  | "indexing_pending"    // Queued for indexing
+  | "indexing_completed"  // Indexed and searchable
+  | "indexing_failed"     // Indexing failed
+
+// Check file status
+const { file } = await client.getFile({ id: fileId });
+if (file.status === "indexing_completed") {
+  console.log("File is indexed and searchable");
+}
+```
+
+## Listing Files
+
+### Basic Listing
+
+```typescript
+const { files, meta } = await client.listFiles({
+  limit: 20,
+  offset: 0
+});
+
+files.forEach(file => {
+  console.log(file.key, file.size, file.status);
+});
+
+// Pagination
+if (meta.nextToken) {
+  const nextPage = await client.listFiles({
+    nextToken: meta.nextToken
+  });
+}
+```
+
+### Filtering by Tags
+
+```typescript
+// Single tag filter
+const { files } = await client.listFiles({
+  tags: { category: "reports" }
+});
+
+// Multiple values (OR condition)
+const { files } = await client.listFiles({
+  tags: {
+    category: ["reports", "documents"]
+  }
+});
+
+// Exclude values
+const { files } = await client.listFiles({
+  tags: {
+    category: {
+      not: "archived"
+    }
+  }
+});
+```
+
+### Filtering by IDs
+
+```typescript
+const { files } = await client.listFiles({
+  ids: ["file_123", "file_456", "file_789"]
+});
+```
+
+### Sorting
+
+```typescript
+const { files } = await client.listFiles({
+  sortBy: "createdAt",
+  sortOrder: "desc",
+  limit: 10
+});
+
+// Available sort fields:
+// - key
+// - size
+// - createdAt
+// - updatedAt
+// - status
+```
+
+## Semantic Search
+
+### Searching File Content
+
+```typescript
+const { passages } = await client.searchFiles({
+  query: "machine learning algorithms",
+  limit: 5
+});
+
+passages.forEach(passage => {
+  console.log("File:", passage.file.key);
+  console.log("Content:", passage.content);
+  console.log("Similarity:", passage.score);
+  console.log("Page:", passage.meta.pageNumber);
+});
+```
+
+### Search Options
+
+```typescript
+const { passages } = await client.searchFiles({
+  query: "customer feedback",
+
+  // Filtering
+  tags: { category: "support" },
+
+  // Context
+  contextDepth: 2,              // Include 2 surrounding passages
+  includeBreadcrumb: true,      // Include document structure
+  withContext: true,            // Return detailed context
+
+  // Consolidation
+  consolidate: true,            // Group by file, sort by position
+
+  // Pagination
+  limit: 10
+});
+```
+
+### Search Result Structure
+
+```typescript
+{
+  passages: [
+    {
+      content: "The matched text passage...",
+      score: 0.89,                           // Similarity score (0-1)
+      meta: {
+        type: "paragraph",                   // passage | title | subtitle | etc.
+        subtype: "text",
+        pageNumber: 5,
+        position: 12,
+        sourceUrl: "https://..."
+      },
+      file: {
+        id: "file_abc123",
+        key: "documents/report.pdf",
+        contentType: "application/pdf",
+        tags: { category: "reports" },
+        createdAt: "2024-01-15T10:30:00Z",
+        updatedAt: "2024-01-15T10:30:00Z"
+      },
+      context: [                            // Only if withContext: true
+        {
+          type: "preceding",
+          content: "Previous passage..."
+        },
+        {
+          type: "current",
+          content: "The matched text passage..."
+        },
+        {
+          type: "subsequent",
+          content: "Next passage..."
+        }
+      ]
+    }
+  ]
+}
+```
+
+## Updating Files
+
+### Update File Metadata
+
+```typescript
+// Update without changing content
+const { file } = await client.updateFileMetadata({
+  id: fileId,
+
+  metadata: {
+    version: "2.0",
+    updatedBy: "Jane Doe"
+  },
+
+  tags: {
+    status: "reviewed",
+    newTag: "value"
+  },
+
+  accessPolicies: ["public_content"],
+
+  expiresAt: new Date("2024-12-31").toISOString()
+});
+```
+
+### Metadata Update Behavior
+
+```typescript
+// Partial updates - existing keys preserved
+await client.updateFileMetadata({
+  id: fileId,
+  metadata: {
+    newKey: "newValue"
+    // Other metadata keys remain unchanged
+  }
+});
+
+// Delete metadata key
+await client.updateFileMetadata({
+  id: fileId,
+  metadata: {
+    oldKey: null  // Removes this key
+  }
+});
+
+// Delete tag
+await client.updateFileMetadata({
+  id: fileId,
+  tags: {
+    oldTag: null  // Removes this tag
+  }
+});
+```
+
+### Replace File Content
+
+```typescript
+// Upload new content with same key (replaces old content)
+const { file } = await client.uploadFile({
+  key: existingFileKey,
+  content: newContent,
+  contentType: "text/plain"
+});
+```
+
+## Managing File Passages
+
+### List File Passages
+
+```typescript
+const { passages, meta } = await client.listFilePassages({
+  id: fileId,
+  limit: 20
+});
+
+passages.forEach(passage => {
+  console.log("Passage ID:", passage.id);
+  console.log("Content:", passage.content);
+  console.log("Type:", passage.meta.type);
+  console.log("Page:", passage.meta.pageNumber);
+});
+```
+
+### Set Custom Passages
+
+```typescript
+// Replace all passages with custom content
+await client.setFilePassages({
+  id: fileId,
+  passages: [
+    {
+      content: "# Introduction\n\nThis is the first section...",
+      type: "title"
+    },
+    {
+      content: "Main content paragraph here.",
+      type: "paragraph",
+      pageNumber: 1
+    },
+    {
+      content: "## Subsection\n\nMore details...",
+      type: "subtitle"
+    }
+  ]
+});
+
+// File status becomes "indexing_pending"
+// Check status later to confirm completion
+```
+
+## Deleting Files
+
+### Delete Single File
+
+```typescript
+// Delete by ID
+await client.deleteFile({ id: "file_abc123" });
+
+// Delete by key
+await client.deleteFile({ id: "documents/report.pdf" });
+```
+
+### Cleanup
+
+When a file is deleted:
+- File metadata removed from database
+- File content removed from S3
+- Vector embeddings removed (if indexed)
+- All passages deleted
+
+## Knowledge Bases
+
+For high-level RAG implementation with automatic indexing and search integration, see **[Knowledge Bases](./knowledge-bases.md)**.
+
+The Files API provides low-level file management. Knowledge Bases use the Files API internally but provide a simpler interface for document indexing and semantic search in AI responses.
+
+## Copying Files
+
+### Copy Within Bot
+
+```typescript
+const { file } = await client.copyFile({
+  idOrKey: sourceFileId,
+  destinationKey: "backups/report-copy.pdf",
+  overwrite: false  // Fail if destination exists
+});
+```
+
+### Copy Across Bots
+
+```typescript
+const { file } = await client.copyFile({
+  idOrKey: sourceFileId,
+  destinationKey: "imported/report.pdf",
+  destinationBotId: targetBotId,
+  overwrite: true
+});
+```
+
+## Supported File Types
+
+### Indexable Formats
+
+These file types support semantic search indexing:
+
+```typescript
+const INDEXABLE_TYPES = {
+  // Documents
+  pdf: "application/pdf",
+  docx: "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+
+  // Text
+  txt: "text/plain",
+  md: "text/markdown",
+  html: "text/html",
+
+  // Images (with vision processing)
+  jpg: "image/jpeg",
+  jpeg: "image/jpeg",
+  png: "image/png"
+};
+```
+
+### Vision Processing
+
+For images and PDFs with vision enabled:
+
+```typescript
+await client.uploadFile({
+  key: "documents/invoice.pdf",
+  content: pdfBuffer,
+  contentType: "application/pdf",
+  index: true,
+  indexing: {
+    configuration: {
+      vision: {
+        transcribePages: true,    // OCR text extraction
+        indexPages: "all"         // or [1, 2, 3] for specific pages
+      }
+    }
+  }
+});
+```
+
+## Usage in Actions/Workflows
+
+### In Actions
+
+```typescript
+import { Action, z, context } from "@botpress/runtime";
+
+export const uploadDocument = new Action({
+  name: "uploadDocument",
+  input: z.object({
+    content: z.string(),
+    filename: z.string()
+  }),
+  output: z.object({
+    fileId: z.string(),
+    fileUrl: z.string()
+  }),
+
+  async handler({ input }) {
+    const client = context.get("client");
+
+    // Upload file
+    const { file } = await client.uploadFile({
+      key: `uploads/${input.filename}`,
+      content: input.content,
+      contentType: "text/plain",
+      index: true,
+      tags: {
+        source: "action",
+        type: "user-upload"
+      }
+    });
+
+    return {
+      fileId: file.id,
+      fileUrl: file.url
+    };
+  }
+});
+```
+
+### In Workflows
+
+```typescript
+import { Workflow, z, context } from "@botpress/runtime";
+
+export const ProcessDocumentWorkflow = new Workflow({
+  name: "processDocument",
+  input: z.object({ documentKey: z.string() }),
+
+  async handler({ input, step }) {
+    const client = context.get("client");
+
+    // Get file
+    const file = await step("get-file", async () => {
+      const { file } = await client.getFile({
+        id: input.documentKey
+      });
+      return file;
+    });
+
+    // Wait for indexing
+    await step("wait-indexing", async () => {
+      let status = file.status;
+      while (status === "INDEXING_PENDING") {
+        await new Promise(resolve => setTimeout(resolve, 1000));
+        const { file: updated } = await client.getFile({
+          id: file.id
+        });
+        status = updated.status;
+      }
+      return status;
+    });
+
+    // Search content
+    const results = await step("search", async () => {
+      const { passages } = await client.searchFiles({
+        query: "important findings",
+        tags: { key: input.documentKey }
+      });
+      return passages;
+    });
+
+    return {
+      processed: true,
+      findings: results.length
+    };
+  }
+});
+```
+
+## Best Practices
+
+### 1. Use Meaningful Keys
+
+```typescript
+// ✅ Good - Clear, hierarchical structure
+await client.uploadFile({
+  key: "documents/2024/reports/q1-sales.pdf"
+});
+
+// ❌ Bad - Unclear, no structure
+await client.uploadFile({
+  key: "file123.pdf"
+});
+```
+
+### 2. Tag Appropriately
+
+```typescript
+// ✅ Good - Useful, queryable tags
+tags: {
+  category: "reports",
+  year: "2024",
+  quarter: "Q1",
+  department: "sales"
+}
+
+// ❌ Bad - Too specific or redundant
+tags: {
+  filename: "q1-sales.pdf",  // Already in key
+  timestamp: "1234567890"     // Use createdAt instead
+}
+```
+
+### 3. Handle File Status
+
+```typescript
+// ✅ Good - Check and handle status
+const { file } = await client.getFile({ id: fileId });
+
+switch (file.status) {
+  case "UPLOAD_PENDING":
+    console.log("Waiting for upload...");
+    break;
+  case "INDEXING_PENDING":
+    console.log("Indexing in progress...");
+    break;
+  case "INDEXING_COMPLETED":
+    // File is searchable
+    break;
+  case "INDEXING_FAILED":
+    console.error("Indexing failed:", file.failedStatusReason);
+    break;
+}
+```
+
+### 4. Set Appropriate Expiry
+
+```typescript
+// ✅ Good - Reasonable expiry for temporary files
+await client.uploadFile({
+  key: "temp/cache-data.json",
+  expiresAt: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString() // 1 day
+});
+
+// For permanent files, omit expiresAt
+await client.uploadFile({
+  key: "documents/permanent-record.pdf"
+  // No expiresAt = never expires
+});
+```
+
+### 5. Optimize Indexing
+
+```typescript
+// ✅ Good - Index only searchable content
+await client.uploadFile({
+  key: "documents/report.pdf",
+  content: pdfBuffer,
+  index: true  // Searchable document
+});
+
+await client.uploadFile({
+  key: "images/logo.png",
+  content: imageBuffer,
+  index: false  // Not searchable, save quota
+});
+
+// ✅ Good - Use vision only when needed
+await client.uploadFile({
+  key: "documents/scanned-invoice.pdf",
+  content: pdfBuffer,
+  index: true,
+  indexing: {
+    configuration: {
+      vision: {
+        transcribePages: true  // OCR needed
+      }
+    }
+  }
+});
+```
+
+## Common Patterns
+
+### File Upload with Validation
+
+```typescript
+export const uploadWithValidation = async (
+  fileContent: Buffer,
+  filename: string
+) => {
+  const client = context.get("client");
+
+  // Validate file size (max 100MB)
+  const MAX_SIZE = 100 * 1024 * 1024;
+  if (fileContent.length > MAX_SIZE) {
+    throw new Error("File too large");
+  }
+
+  // Validate file type
+  const contentType = getContentType(filename);
+  if (!ALLOWED_TYPES.includes(contentType)) {
+    throw new Error("File type not allowed");
+  }
+
+  // Upload
+  const { file } = await client.uploadFile({
+    key: `uploads/${Date.now()}-${filename}`,
+    content: fileContent,
+    contentType,
+    index: true,
+    tags: {
+      originalName: filename,
+      uploadedAt: new Date().toISOString()
+    }
+  });
+
+  return file;
+};
+```
+
+### Search with Context
+
+```typescript
+export const searchWithContext = async (
+  query: string,
+  category?: string
+) => {
+  const client = context.get("client");
+
+  const { passages } = await client.searchFiles({
+    query,
+    tags: category ? { category } : undefined,
+    contextDepth: 2,
+    includeBreadcrumb: true,
+    limit: 5
+  });
+
+  return passages.map(p => ({
+    content: p.content,
+    source: p.file.key,
+    relevance: p.score,
+    location: `Page ${p.meta.pageNumber || "N/A"}`
+  }));
+};
+```
+
+### Bulk File Operations
+
+```typescript
+export const bulkUpload = async (files: FileData[]) => {
+  const client = context.get("client");
+
+  const results = await Promise.allSettled(
+    files.map(async (fileData) => {
+      const { file } = await client.uploadFile({
+        key: fileData.key,
+        content: fileData.content,
+        contentType: fileData.contentType,
+        index: true
+      });
+
+      return file;
+    })
+  );
+
+  const successful = results
+    .filter(r => r.status === "fulfilled")
+    .map(r => r.value);
+
+  const failed = results
+    .filter(r => r.status === "rejected")
+    .map(r => r.reason);
+
+  return { successful, failed };
+};
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Upload fails with 403 error**
+   - Ensure you have proper access policies set
+   - Check that the bot token has file upload permissions
+   - Verify the file size is within limits
+
+2. **File not searchable**
+   - Check `file.status` is `INDEXING_COMPLETED`
+   - Verify `index: true` was set during upload
+   - Ensure file type is in supported indexable formats
+   - Check billing plan supports indexing features
+
+3. **Search returns no results**
+   - Verify files are indexed (`status: "INDEXING_COMPLETED"`)
+   - Check tag filters aren't too restrictive
+   - Try broader search queries
+   - Ensure files have searchable content
+
+4. **File access denied**
+   - Verify access policies are correctly set
+   - Check workspace member permissions
+   - For integration files, ensure proper namespace
+
+5. **Quota exceeded**
+   - Check storage usage in workspace settings
+   - Delete unused files or increase quota
+   - Consider file expiry policies for temporary files
+
+## Limits and Quotas
+
+- **Maximum file size**: 100 MB
+- **Maximum expiry**: 90 days from upload
+- **Maximum tags per file**: 50
+- **Maximum indexable file size**: varies by file type
+- **Search result limit**: Configurable, recommended max 100
+
+## Migration Notes
+
+### Current API (ADK 1.13+)
+
+The current Files API uses direct content upload:
+
+```typescript
+// ✅ CURRENT - Direct content parameter
+await client.uploadFile({
+  key: "unique-key",
+  content: fileBuffer,           // Direct upload
+  contentType: "application/pdf",
+  tags: { category: "docs" }
+});
+
+// Alternative: Upload from URL
+await client.uploadFile({
+  key: "unique-key",
+  url: "https://example.com/file.pdf",  // Fetch from URL
+  tags: { category: "docs" }
+});
+```
+
+### Retrieving File Content
+
+Files are accessed via their URL:
+
+```typescript
+const { file } = await client.getFile({ id: fileId });
+
+// Fetch file content
+const response = await fetch(file.url);
+const content = await response.arrayBuffer();
+
+// For text/JSON
+const text = await response.text();
+const json = await response.json();
+```
+
+**URL Types:**
+- **Public files** (`public_content` policy): Permanent CDN URL
+- **Private files**: Temporary presigned URL (use shortly after retrieval)