insforge 0.3.3 → 1.2.10

package/docs/core-concepts/functions/architecture.mdx

@@ -0,0 +1,105 @@
+---
+title: Architecture
+description: Serverless JavaScript functions running in isolated Deno runtime
+---
+
+## Overview
+
+InsForge Functions provide a secure, scalable serverless compute platform that runs JavaScript/TypeScript code in isolated Deno workers with full access to the InsForge SDK.
+
+## Technology Stack
+
+```mermaid
+graph TB
+    Client[Client Application] --> Backend[Backend API :7130]
+    Backend --> Proxy[Proxy Layer]
+    Proxy --> Deno[Deno Runtime :7133]
+    Deno --> Worker[Web Worker]
+    Worker --> Sandbox[Isolated Sandbox]
+
+    Backend --> DB[(PostgreSQL)]
+    Deno --> DB
+    Worker --> SDK[InsForge SDK]
+    SDK --> Backend
+
+    style Client fill:#1e293b,stroke:#475569,color:#e2e8f0
+    style Backend fill:#166534,stroke:#22c55e,color:#dcfce7
+    style Proxy fill:#1e40af,stroke:#3b82f6,color:#dbeafe
+    style Deno fill:#c2410c,stroke:#fb923c,color:#fed7aa
+    style Worker fill:#4c1d95,stroke:#8b5cf6,color:#ede9fe
+    style Sandbox fill:#7c2d12,stroke:#f97316,color:#fed7aa
+    style DB fill:#0e7490,stroke:#06b6d4,color:#cffafe
+    style SDK fill:#1e40af,stroke:#3b82f6,color:#dbeafe
+```
+
+## Core Components
+
+| Component | Technology | Port | Purpose |
+|-----------|------------|------|---------|
+| **Backend API** | Node.js/Express | 7130 | Function management, authentication, proxy |
+| **Runtime** | Deno v2.0.6 | 7133 | Secure JavaScript/TypeScript execution |
+| **Sandbox** | Web Workers | - | Isolated execution environment |
+| **Database** | PostgreSQL | 5432 | Function code and metadata storage |
+| **SDK** | @insforge/sdk | - | Pre-injected client for backend access |
+| **Secrets** | AES-256-GCM | - | Encrypted environment variables |
+
+## How It Works
+
+When a client makes a request to `/functions/{slug}`:
+
+1. The backend API receives and validates the request
+2. Request is proxied to the Deno runtime
+3. Function code is executed in an isolated Web Worker
+4. The function has access to the InsForge SDK and environment variables
+5. Response is returned to the client
+
+### Authentication Flow
+
+```mermaid
+sequenceDiagram
+    participant C as Client
+    participant B as Backend API
+    participant D as Deno Runtime
+    participant W as Worker
+    participant SDK as InsForge SDK
+
+    C->>B: POST /functions/my-api<br/>Authorization: Bearer TOKEN
+    B->>D: Proxy request with headers
+    D->>D: Fetch function code from DB
+    D->>D: Decrypt secrets
+    D->>W: Create worker with code + secrets
+    W->>SDK: createClient({token})
+    SDK->>B: Validate token
+    B-->>SDK: User data
+    W->>C: HTTP Response
+```
+
+## Performance Characteristics
+
+### Execution Limits
+
+| Limit | Value | Description |
+|-------|-------|-------------|
+| **Timeout** | 30 seconds | Maximum execution time per invocation |
+| **Memory** | Worker default | Depends on Deno worker allocation |
+| **Payload Size** | 10MB | Maximum request/response size |
+| **Concurrent Workers** | System dependent | Limited by server resources |
+| **Cold Start** | ~50-200ms | Time to create new worker |
+
+### Optimization Strategies
+
+1. **Worker Pooling**: Workers are created on-demand
+2. **Code Caching**: Function code cached in memory
+3. **Secret Caching**: Decrypted secrets cached per execution
+4. **SDK Reuse**: SDK client created once per worker
+
+## Best Practices
+
+1. **Keep Functions Small**: Single responsibility per function
+2. **Handle Errors Gracefully**: Always return proper HTTP responses
+3. **Validate Input**: Check request data before processing
+4. **Use Caching**: Cache frequently accessed data
+5. **Optimize Queries**: Use efficient database queries
+6. **Monitor Performance**: Track execution times and errors
+7. **Secure Secrets**: Never log sensitive data
+8. **Test Locally**: Test functions before deployment

package/docs/core-concepts/functions/sdk.mdx

@@ -0,0 +1,184 @@
+---
+title: Functions SDK Reference
+description: Invoke serverless functions with the InsForge SDK
+---
+
+import Installation from '/snippets/sdk-installation.mdx';
+
+<Installation />
+
+## invoke()
+
+Invoke a serverless function by slug.
+
+### Parameters
+
+- `slug` (string, required) - Function slug/name
+- `body` (any, optional) - Request body (JSON-serializable)
+- `headers` (object, optional) - Custom headers
+- `method` ('GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE', optional) - HTTP method (default: POST)
+
+### Returns
+
+```typescript
+{
+  data: any | null,  // Response from function
+  error: Error | null
+}
+```
+
+<Note>
+SDK automatically includes authentication token from logged-in user.
+</Note>
+
+### Example (POST with body)
+
+```javascript
+const { data, error } = await insforge.functions.invoke('hello-world', {
+  body: { name: 'World', greeting: 'Hello' }
+})
+
+console.log(data)
+```
+
+### Output (POST with body)
+
+```json
+{
+  "data": {
+    "message": "Hello, World!",
+    "timestamp": "2024-01-15T10:30:00Z"
+  },
+  "error": null
+}
+```
+
+### Example (GET request)
+
+```javascript
+const { data, error } = await insforge.functions.invoke('get-stats', {
+  method: 'GET'
+})
+
+console.log(data)
+```
+
+### Output (GET request)
+
+```json
+{
+  "data": {
+    "users": 100,
+    "posts": 500,
+    "comments": 1200
+  },
+  "error": null
+}
+```
+
+### Example (With custom headers)
+
+```javascript
+const { data, error } = await insforge.functions.invoke('api-endpoint', {
+  method: 'PUT',
+  body: { id: '123', status: 'active' },
+  headers: { 'X-Custom-Header': 'value' }
+})
+```
+
+### Output (With custom headers)
+
+```json
+{
+  "data": {
+    "updated": true,
+    "id": "123"
+  },
+  "error": null
+}
+```
+
+## Complete Serverless Function Examples
+
+### Example 1: Public Function (No Authentication Required)
+
+```javascript
+// Use anon token for public data access
+// No import needed - createClient is injected by the worker template
+module.exports = async function(request) {
+  const corsHeaders = {
+    'Access-Control-Allow-Origin': '*',
+    'Access-Control-Allow-Methods': 'GET, POST, OPTIONS',
+    'Access-Control-Allow-Headers': 'Content-Type, Authorization'
+  };
+
+  if (request.method === 'OPTIONS') {
+    return new Response(null, { status: 204, headers: corsHeaders });
+  }
+
+  // Create client with anon token - no authentication needed
+  const client = createClient({
+    baseUrl: Deno.env.get('INSFORGE_INTERNAL_URL') || 'http://insforge:7130',
+    anonKey: Deno.env.get('ANON_KEY')
+  });
+
+  // Access public data
+  const { data, error } = await client.database
+    .from('public_posts')
+    .select('*')
+    .limit(10);
+
+  return new Response(JSON.stringify({ data }), {
+    status: 200,
+    headers: { ...corsHeaders, 'Content-Type': 'application/json' }
+  });
+}
+```
+
+### Example 2: Authenticated Function (Access User Data)
+
+```javascript
+// Use functionToken to access authenticated user's data
+// No import needed - createClient is injected by the worker template
+module.exports = async function(request) {
+  const corsHeaders = {
+    'Access-Control-Allow-Origin': '*',
+    'Access-Control-Allow-Methods': 'GET, POST, OPTIONS',
+    'Access-Control-Allow-Headers': 'Content-Type, Authorization'
+  };
+
+  if (request.method === 'OPTIONS') {
+    return new Response(null, { status: 204, headers: corsHeaders });
+  }
+
+  // Extract token from request headers
+  const authHeader = request.headers.get('Authorization');
+  const userToken = authHeader ? authHeader.replace('Bearer ', '') : null;
+
+  // Create client with user's token for authenticated access
+  const client = createClient({
+    baseUrl: Deno.env.get('INSFORGE_INTERNAL_URL') || 'http://insforge:7130',
+    edgeFunctionToken: userToken
+  });
+
+  // Get authenticated user
+  const { data: userData } = await client.auth.getCurrentUser();
+  if (!userData?.user?.id) {
+    return new Response(JSON.stringify({ error: 'Unauthorized' }), {
+      status: 401,
+      headers: { ...corsHeaders, 'Content-Type': 'application/json' }
+    });
+  }
+
+  // Access user's private data or create records with user_id
+  await client.database.from('user_posts').insert([{
+    user_id: userData.user.id,
+    content: 'My post'
+  }]);
+
+  return new Response(JSON.stringify({ success: true }), {
+    status: 200,
+    headers: { ...corsHeaders, 'Content-Type': 'application/json' }
+  });
+}
+```

package/docs/core-concepts/storage/architecture.mdx

@@ -0,0 +1,243 @@
+---
+title: Architecture  
+description: Enterprise-grade storage system powered by AWS S3
+---
+
+## Overview
+
+InsForge provides a high-performance, scalable storage system built on AWS S3, delivering enterprise-grade reliability with 99.999999999% (11 9's) durability.
+
+## Technology Stack
+
+```mermaid
+graph TB
+    Client[Client Application] --> SDK[InsForge SDK]
+    SDK --> StorageAPI[Storage API]
+    
+    StorageAPI --> S3[AWS S3]
+    
+    StorageAPI --> DB[(PostgreSQL)]
+    DB --> Metadata[File Metadata]
+    DB --> Buckets[Bucket Configuration]
+    
+    S3 --> DirectUpload[Presigned URLs]
+    S3 --> SecureAccess[IAM Policies]
+    
+    style Client fill:#1e293b,stroke:#475569,color:#e2e8f0
+    style SDK fill:#1e40af,stroke:#3b82f6,color:#dbeafe
+    style StorageAPI fill:#166534,stroke:#22c55e,color:#dcfce7
+    style S3 fill:#ea580c,stroke:#f97316,color:#fed7aa
+    style DB fill:#0e7490,stroke:#06b6d4,color:#cffafe
+    style Metadata fill:#0e7490,stroke:#22d3ee,color:#cffafe
+    style Buckets fill:#0e7490,stroke:#22d3ee,color:#cffafe
+    style DirectUpload fill:#ea580c,stroke:#fb923c,color:#fed7aa
+    style SecureAccess fill:#ea580c,stroke:#fb923c,color:#fed7aa
+```
+
+## Core Components
+
+| Component | Technology | Purpose |
+|-----------|------------|---------|
+| **Storage Backend** | AWS S3 | Enterprise-grade object storage |
+| **Metadata Store** | PostgreSQL | File metadata, bucket configuration |
+| **Upload Handler** | Multer | Multipart form data parsing (both backends) |
+| **URL Strategy** | Presigned URLs | Secure direct uploads/downloads (S3 only) |
+| **Access Control** | JWT + Bucket visibility | Public/private bucket permissions |
+
+## AWS S3 Architecture
+
+### Enterprise Features
+
+- **Direct Uploads**: Presigned URLs bypass API server for unlimited scale
+- **IAM Security**: Role-based authentication without credential management
+- **Multi-Tenancy**: Secure isolation between projects using app key prefix
+- **Bucket Policies**: Public and private bucket configurations
+- **Automatic Cleanup**: Configurable lifecycle policies
+- **Metadata Tracking**: Rich file metadata stored in PostgreSQL
+
+## Upload Strategies
+
+### Direct Upload to S3
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant API
+    participant S3
+    participant DB
+    
+    Client->>API: POST/PUT with multipart/form-data
+    API->>S3: Upload file
+    S3-->>API: Success
+    API->>DB: Store metadata
+    API-->>Client: {url, key, size}
+```
+
+### Presigned URL Upload (Recommended)
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant API
+    participant S3
+    participant DB
+    
+    Client->>API: GET upload strategy
+    API->>S3: Generate presigned URL
+    S3-->>API: Presigned URL
+    API-->>Client: {uploadUrl, key}
+    Client->>S3: Direct upload
+    Client->>API: Confirm upload
+    API->>DB: Store metadata
+    API-->>Client: {url, key, size}
+```
+
+## Bucket Configuration
+
+### Bucket Types
+
+| Type | Access | Use Case |
+|------|--------|----------|
+| **Public** | No auth required for downloads | Public assets, images, static files |
+| **Private** | Auth required for all operations | User files, sensitive documents |
+
+
+## File Operations
+
+### Upload Flow
+
+1. **Request Upload**: Client requests upload permission
+2. **Validation**: Check auth, bucket permissions, file size
+3. **Strategy Selection**: Choose direct or presigned upload
+4. **Upload**: Client uploads via selected method
+5. **Confirmation**: Verify upload and store metadata
+6. **Response**: Return file URL and metadata
+
+### Download Flow
+
+1. **Request File**: Client requests file access
+2. **Permission Check**: Verify bucket/object permissions
+3. **Strategy Selection**: Direct serve or presigned URL
+4. **Delivery**: Stream file or redirect to URL
+
+## Security Features
+
+<CardGroup cols={2}>
+  <Card title="Bucket Policies" icon="shield">
+    Public, private, or protected bucket access control
+  </Card>
+  
+  <Card title="JWT Authentication" icon="key">
+    Token-based access for private resources
+  </Card>
+  
+  <Card title="Presigned URLs" icon="link">
+    Time-limited URLs for secure S3 access
+  </Card>
+  
+  <Card title="MIME Type Validation" icon="file-check">
+    Restrict uploads to allowed file types
+  </Card>
+  
+  <Card title="Size Limits" icon="weight">
+    10MB default, configurable via MAX_FILE_SIZE
+  </Card>
+  
+  <Card title="App Key Isolation" icon="key">
+    Multi-tenant isolation using app key prefix in S3
+  </Card>
+</CardGroup>
+
+## Intelligent Metadata Management
+
+InsForge maintains optimized metadata in PostgreSQL for instant queries:
+
+- **Fast Search**: Indexed metadata for sub-millisecond lookups
+- **Rich Metadata**: MIME types, sizes, timestamps, custom tags
+- **Usage Analytics**: Track downloads, bandwidth, popular files
+- **Access Control**: Fine-grained permissions per file or bucket
+
+### Secure URL Generation
+
+| Type | Use Case | Security |
+|------|----------|----------|
+| **Public Access** | Static assets, images | Direct S3 URLs |
+| **Presigned GET** | Private file access | Time-limited, single-use |
+| **Presigned POST** | Direct uploads | Validated, size-limited |
+
+## Performance Optimizations
+
+### Performance Features
+
+- **Direct S3 Access**: Bypass API server for uploads/downloads
+- **Browser Caching**: Cache-Control headers
+- **ETag Support**: Conditional requests for S3
+- **Parallel Uploads**: Support for multipart uploads
+
+### Upload Methods
+
+**1. Direct Upload through API Server:**
+- Client sends file to `/api/storage/buckets/{bucket}/objects`
+- File passes through API server memory
+- Server uploads to S3
+- Limited by server memory (10MB default)
+
+**2. Presigned URL Upload (Recommended):**
+- Client requests upload URL from `/api/storage/buckets/{bucket}/upload-strategy`
+- Server returns presigned POST URL
+- Client uploads directly to S3 (bypasses API server)
+- No server memory constraints
+- Client confirms upload via `/api/storage/buckets/{bucket}/objects/{key}/confirm-upload`
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `AWS_S3_BUCKET` | S3 bucket name | `my-app-storage` |
+| `AWS_REGION` | AWS region | `us-east-2` |
+| `APP_KEY` | App key for S3 multi-tenancy | `my-app-key` |
+
+### S3 Configuration
+
+```javascript
+// S3 client configuration
+const s3Client = new S3Client({
+  region: this.region, // e.g., 'us-east-2'
+  // IAM role credentials are automatically used on EC2
+  // No explicit credentials needed in production
+});
+
+// File paths use app key prefix for multi-tenancy
+const s3Key = `${this.appKey}/${bucket}/${key}`;
+```
+
+## Best Practices
+
+<CardGroup cols={2}>
+  <Card title="Use Buckets" icon="folder">
+    Organize files logically in buckets
+  </Card>
+  
+  <Card title="Set Limits" icon="gauge">
+    Configure appropriate size/type limits
+  </Card>
+  
+  <Card title="Clean URLs" icon="link">
+    Use consistent, SEO-friendly key naming
+  </Card>
+  
+  <Card title="Metadata" icon="tags">
+    Store searchable metadata in database
+  </Card>
+  
+  <Card title="Backup Strategy" icon="cloud-arrow-up">
+    Implement regular backups for production
+  </Card>
+  
+  <Card title="Monitor Usage" icon="chart-line">
+    Track storage costs and usage patterns
+  </Card>
+</CardGroup>
+