create-lego-one 2.0.12 → 2.0.14

package/template/docs/framework/research/02-data-migration-protocol.md

@@ -0,0 +1,699 @@
+# Data Migration Protocol: PocketBase Auto-Schema
+
+**Project:** Lego-One (Modern.js SaaS OS)
+**Document:** 02 - Data Migration Protocol
+**Status:** Research Phase
+
+## Executive Summary
+
+This document defines the **Migration Protocol** for automatically synchronizing PocketBase collections when plugins are enabled. Since PocketBase lacks traditional SQL migrations, we use a **startup schema check** pattern that leverages the PocketBase Admin API to programmatically create collections.
+
+---
+
+## 1. Problem Statement
+
+### 1.1 The Challenge
+
+PocketBase manages collections through:
+1. **Dashboard UI** - Manual creation/modification
+2. **Admin API** - Programmatic operations (superuser only)
+3. **JavaScript Hooks** - Server-side extensions
+
+**Problem:** Each plugin requires specific database collections, but:
+- Users shouldn't manually create collections via dashboard
+- Plugins must be "drop-in" - work immediately when enabled
+- Schema versioning is needed for plugin updates
+
+### 1.2 Requirements
+
+| Requirement | Description |
+|-------------|-------------|
+| **Auto-Discovery** | Host detects enabled plugins from `saas.config.ts` |
+| **Schema Check** | Verify required collections exist before app starts |
+| **Auto-Creation** | Create missing collections programmatically |
+| **Idempotency** | Safe to run multiple times without errors |
+| **Versioning** | Support schema migrations for plugin updates |
+| **Multi-Tenancy** | Apply PocketBase API Rules for Row Level Security |
+
+---
+
+## 2. Migration Architecture
+
+### 2.1 Startup Protocol Flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                     Host Application Boot                               │
+└─────────────────────────────────────────────────────────────────────────┘
+       │
+       ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 1. Read saas.config.ts                                                  │
+│    - Parse enabled plugins list                                        │
+│    - Load plugin.config.ts from each plugin                            │
+└──────┬──────────────────────────────────────────────────────────────────┘
+       │
+       ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 2. Initialize PocketBase Admin Client                                  │
+│    - Load admin credentials from env vars                               │
+│    - Authenticate as superuser                                          │
+└──────┬──────────────────────────────────────────────────────────────────┘
+       │
+       ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 3. For each enabled plugin:                                            │
+│    a. Load migration files from plugin/migrations/                     │
+│    b. Check if collection exists ($app.findCollectionByNameOrId)       │
+│    c. If missing, create collection ($app.save(new Collection(...)))   │
+│    d. If exists, check schema version, migrate if needed               │
+└──────┬──────────────────────────────────────────────────────────────────┘
+       │
+       ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 4. Store migration version in _lego_migrations table                   │
+│    - Track which migrations have been applied                           │
+│    - Prevent re-running same migrations                                 │
+└──────┬──────────────────────────────────────────────────────────────────┘
+       │
+       ▼
+┌─────────────────────────────────────────────────────────────────────────┐
+│ 5. Proceed to app initialization                                       │
+│    - All required collections guaranteed to exist                      │
+│    - Plugins can safely query their collections                        │
+└─────────────────────────────────────────────────────────────────────────┘
+```
+
+### 2.2 Migration File Structure
+
+```
+packages/plugins/@lego/plugin-inventory/
+├── migrations/
+│   ├── 001_initial.ts           # Initial schema
+│   ├── 002_add_categories.ts    # Feature addition
+│   └── 003_add_suppliers.ts     # Another feature
+├── plugin.config.ts
+└── src/
+```
+
+---
+
+## 3. Implementation
+
+### 3.1 Migration System Library
+
+**File:** `host/src/kernel/migration-system.ts`
+
+```typescript
+import PocketBase from 'pocketbase';
+import { Collection } from 'pocketbase';
+
+// ============================================================================
+// Types
+// ============================================================================
+
+export interface Migration {
+  version: number;
+  name: string;
+  collection: string;
+  up: () => Collection;
+  down?: () => void;
+}
+
+export interface PluginMigrations {
+  pluginName: string;
+  migrations: Migration[];
+}
+
+export interface MigrationRecord {
+  id: string;
+  pluginName: string;
+  collection: string;
+  version: number;
+  executedAt: string;
+}
+
+// ============================================================================
+// Migration Runner
+// ============================================================================
+
+export class MigrationSystem {
+  private pb: PocketBase;
+
+  constructor(adminUrl: string, adminEmail: string, adminPassword: string) {
+    this.pb = new PocketBase(adminUrl);
+  }
+
+  async initialize(): Promise<void> {
+    // Authenticate as admin
+    await this.pb.admins.authWithPassword(
+      process.env.VITE_POCKETBASE_ADMIN_EMAIL!,
+      process.env.VITE_POCKETBASE_ADMIN_PASSWORD!
+    );
+
+    // Ensure migrations tracking table exists
+    await this.ensureMigrationsTable();
+  }
+
+  private async ensureMigrationsTable(): Promise<void> {
+    // Check if _lego_migrations collection exists
+    const existing = await this.pb.collections.getList(1, 1, {
+      filter: `name = '_lego_migrations'`,
+    });
+
+    if (existing.items.length === 0) {
+      // Create the migrations tracking collection
+      const migrationCollection = new Collection({
+        name: '_lego_migrations',
+        type: 'base',
+        listRule: null,            // No list access via API
+        viewRule: null,            // No view access via API
+        createRule: null,          // No create access via API
+        updateRule: null,          // No update access via API
+        deleteRule: null,          // No delete access via API
+        fields: [
+          {
+            name: 'pluginName',
+            type: 'text',
+            required: true,
+          },
+          {
+            name: 'collection',
+            type: 'text',
+            required: true,
+          },
+          {
+            name: 'version',
+            type: 'number',
+            required: true,
+          },
+          {
+            name: 'executedAt',
+            type: 'autodate',
+            required: true,
+          },
+        ],
+        indexes: [
+          'CREATE UNIQUE INDEX idx_migrations_plugin_collection_version ON _lego_migrations (pluginName, collection, version)',
+        ],
+      });
+
+      await this.executeAdminRequest('/api/collections', 'POST', migrationCollection);
+    }
+  }
+
+  /**
+   * Check if a migration has already been executed
+   */
+  private async isExecuted(pluginName: string, collection: string, version: number): Promise<boolean> {
+    try {
+      const records = await this.pb.records.getList('_lego_migrations', 1, 1, {
+        filter: `pluginName = "${pluginName}" && collection = "${collection}" && version = ${version}`,
+      });
+      return records.totalItems > 0;
+    } catch {
+      return false;
+    }
+  }
+
+  /**
+   * Record a migration as executed
+   */
+  private async recordMigration(pluginName: string, collection: string, version: number): Promise<void> {
+    await this.pb.records.create('_lego_migrations', {
+      pluginName,
+      collection,
+      version,
+    });
+  }
+
+  /**
+   * Execute a PocketBase Admin API request
+   * Note: Collection operations require superuser privileges
+   */
+  private async executeAdminRequest(endpoint: string, method: string, body?: any): Promise<any> {
+    const url = `${this.pb.baseUrl}${endpoint}`;
+
+    const response = await fetch(url, {
+      method,
+      headers: {
+        'Content-Type': 'application/json',
+        'Authorization': this.pb.authStore.token,
+      },
+      body: body ? JSON.stringify(body) : undefined,
+    });
+
+    if (!response.ok) {
+      const error = await response.text();
+      throw new Error(`Admin API request failed: ${error}`);
+    }
+
+    return response.json();
+  }
+
+  /**
+   * Check if a collection exists
+   */
+  async collectionExists(name: string): Promise<boolean> {
+    try {
+      // Try to fetch via Admin API
+      await this.executeAdminRequest(`/api/collections/${name}`, 'GET');
+      return true;
+    } catch {
+      return false;
+    }
+  }
+
+  /**
+   * Create a new collection from a migration
+   */
+  async createCollection(collection: Collection): Promise<void> {
+    await this.executeAdminRequest('/api/collections', 'POST', collection);
+  }
+
+  /**
+   * Update an existing collection
+   */
+  async updateCollection(id: string, collection: Partial<Collection>): Promise<void> {
+    await this.executeAdminRequest(`/api/collections/${id}`, 'PATCH', collection);
+  }
+
+  /**
+   * Run migrations for a plugin
+   */
+  async runMigrations(pluginMigrations: PluginMigrations): Promise<void> {
+    const { pluginName, migrations } = pluginMigrations;
+
+    // Sort migrations by version
+    const sortedMigrations = migrations.sort((a, b) => a.version - b.version);
+
+    for (const migration of sortedMigrations) {
+      const executed = await this.isExecuted(pluginName, migration.collection, migration.version);
+
+      if (!executed) {
+        console.log(`[Migration] Running ${pluginName}/${migration.name} (v${migration.version})`);
+
+        // Execute the migration
+        const collection = migration.up();
+
+        // Check if collection exists
+        const exists = await this.collectionExists(migration.collection);
+
+        if (exists) {
+          // Update existing collection
+          const existing = await this.executeAdminRequest(`/api/collections/${migration.collection}`, 'GET');
+          await this.updateCollection(existing.id, collection);
+        } else {
+          // Create new collection
+          await this.createCollection(collection);
+        }
+
+        // Record as executed
+        await this.recordMigration(pluginName, migration.collection, migration.version);
+
+        console.log(`[Migration] Completed ${pluginName}/${migration.name}`);
+      } else {
+        console.log(`[Migration] Skipped ${pluginName}/${migration.name} (already executed)`);
+      }
+    }
+  }
+
+  /**
+   * Run all pending migrations for multiple plugins
+   */
+  async runAllMigrations(allMigrations: PluginMigrations[]): Promise<void> {
+    for (const pluginMigrations of allMigrations) {
+      await this.runMigrations(pluginMigrations);
+    }
+  }
+}
+```
+
+### 3.2 Plugin Migration Definition
+
+**File:** `packages/plugins/@lego/plugin-inventory/migrations/001_initial.ts`
+
+```typescript
+import { Collection } from 'pocketbase';
+import { BoolField, NumberField, TextField, RelationField } from 'pocketbase';
+
+export const version = 1;
+export const name = 'initial';
+
+export function up(): Collection {
+  return new Collection({
+    type: 'base',
+    name: 'inventory_items',
+    // Multi-tenancy: Users can only see their own items
+    listRule: '@request.auth.id != "" && owner = @request.auth.id',
+    viewRule: '@request.auth.id != "" && owner = @request.auth.id',
+    createRule: '@request.auth.id != ""',
+    updateRule: '@request.auth.id != "" && owner = @request.auth.id',
+    deleteRule: '@request.auth.id != "" && owner = @request.auth.id',
+    fields: [
+      {
+        name: 'name',
+        type: 'text',
+        required: true,
+        max: 200,
+      },
+      {
+        name: 'description',
+        type: 'text',
+        required: false,
+        max: 1000,
+      },
+      {
+        name: 'quantity',
+        type: 'number',
+        required: true,
+        min: 0,
+      },
+      {
+        name: 'price',
+        type: 'number',
+        required: true,
+        min: 0,
+      },
+      {
+        name: 'owner',
+        type: 'relation',
+        required: true,
+        maxSelect: 1,
+        collectionId: '_pb_users_auth_',
+        cascadeDelete: true,
+      },
+    ],
+    indexes: [
+      'CREATE INDEX idx_inventory_owner ON inventory_items (owner)',
+    ],
+  });
+}
+```
+
+**File:** `packages/plugins/@lego/plugin-inventory/migrations/002_add_categories.ts`
+
+```typescript
+import { Collection } from 'pocketbase';
+import { SelectField } from 'pocketbase';
+
+export const version = 2;
+export const name = 'add_categories';
+
+export function up(): Collection {
+  return new Collection({
+    type: 'base',
+    name: 'inventory_items',
+    // Preserve existing rules
+    listRule: '@request.auth.id != "" && owner = @request.auth.id',
+    viewRule: '@request.auth.id != "" && owner = @request.auth.id',
+    createRule: '@request.auth.id != ""',
+    updateRule: '@request.auth.id != "" && owner = @request.auth.id',
+    deleteRule: '@request.auth.id != "" && owner = @request.auth.id',
+    fields: [
+      // ... existing fields from 001_initial ...
+      {
+        name: 'name',
+        type: 'text',
+        required: true,
+        max: 200,
+      },
+      {
+        name: 'quantity',
+        type: 'number',
+        required: true,
+        min: 0,
+      },
+      {
+        name: 'price',
+        type: 'number',
+        required: true,
+        min: 0,
+      },
+      {
+        name: 'owner',
+        type: 'relation',
+        required: true,
+        maxSelect: 1,
+        collectionId: '_pb_users_auth_',
+        cascadeDelete: true,
+      },
+      // NEW FIELD
+      {
+        name: 'category',
+        type: 'select',
+        required: false,
+        values: ['electronics', 'clothing', 'food', 'other'],
+      },
+    ],
+    indexes: [
+      'CREATE INDEX idx_inventory_owner ON inventory_items (owner)',
+      'CREATE INDEX idx_inventory_category ON inventory_items (category)',
+    ],
+  });
+}
+```
+
+### 3.3 Plugin Config: Expose Migrations
+
+**File:** `packages/plugins/@lego/plugin-inventory/plugin.config.ts`
+
+```typescript
+import { definePluginConfig } from '@lego/kernel/plugin-config';
+
+export default definePluginConfig({
+  name: '@lego/plugin-inventory',
+  version: '1.0.0',
+  displayName: 'Inventory Management',
+  description: 'Track inventory items and stock levels',
+
+  // Expose migrations to host
+  migrations: {
+    collection: 'inventory_items',
+    files: [
+      './migrations/001_initial',
+      './migrations/002_add_categories',
+    ],
+  },
+
+  // ... other config ...
+});
+```
+
+### 3.4 Host: Load and Run Migrations
+
+**File:** `host/src/kernel/migration-loader.ts`
+
+```typescript
+import { MigrationSystem, PluginMigrations } from './migration-system';
+import { loadSaaSConfig } from './plugin-loader';
+
+export async function loadAndRunMigrations(): Promise<void> {
+  // Initialize migration system
+  const migrationSystem = new MigrationSystem(
+    import.meta.env.VITE_POCKETBASE_URL,
+    import.meta.env.VITE_POCKETBASE_ADMIN_EMAIL,
+    import.meta.env.VITE_POCKETBASE_ADMIN_PASSWORD
+  );
+
+  await migrationSystem.initialize();
+
+  // Load enabled plugins
+  const enabledPlugins = await loadSaaSConfig();
+
+  // Load migrations from each plugin
+  const allMigrations: PluginMigrations[] = [];
+
+  for (const plugin of enabledPlugins) {
+    try {
+      // Dynamically import plugin config
+      const pluginConfig = await import(/* @vite-ignore */ `${plugin.entry}/plugin.config.ts`);
+
+      if (pluginConfig.default?.migrations) {
+        const migrations = [];
+
+        for (const migrationFile of pluginConfig.default.migrations.files) {
+          const migrationModule = await import(/* @vite-ignore */ migrationFile);
+          migrations.push({
+            version: migrationModule.version,
+            name: migrationModule.name,
+            collection: pluginConfig.default.migrations.collection,
+            up: migrationModule.up,
+          });
+        }
+
+        allMigrations.push({
+          pluginName: plugin.name,
+          migrations,
+        });
+      }
+    } catch (error) {
+      console.warn(`[Migration] Could not load migrations for ${plugin.name}:`, error);
+    }
+  }
+
+  // Run all pending migrations
+  await migrationSystem.runAllMigrations(allMigrations);
+}
+```
+
+### 3.5 Host: Run Migrations on Bootstrap
+
+**File:** `host/src/bootstrap.tsx`
+
+```typescript
+import { StrictMode } from 'react';
+import ReactDOM from 'react-dom';
+import App from './App';
+import { loadAndRunMigrations } from './kernel/migration-loader';
+import { registerSharedState } from './kernel/shared-state-bridge';
+import './styles/globals.css';
+
+async function bootstrap() {
+  // 1. Run database migrations first
+  console.log('[Boot] Running database migrations...');
+  try {
+    await loadAndRunMigrations();
+    console.log('[Boot] Migrations complete');
+  } catch (error) {
+    console.error('[Boot] Migration failed:', error);
+    // You might want to block app startup or show error UI
+    return;
+  }
+
+  // 2. Register shared state bridge
+  registerSharedState();
+
+  // 3. Mount React app
+  ReactDOM.createRoot(document.getElementById('root')!).render(
+    <StrictMode>
+      <App />
+    </StrictMode>
+  );
+}
+
+bootstrap();
+```
+
+---
+
+## 4. PocketBase API Rules for Multi-Tenancy
+
+### 4.1 Row Level Security Pattern
+
+All plugin collections must follow this pattern:
+
+```typescript
+const collection = new Collection({
+  name: 'plugin_data',
+  // Users can only list their own records
+  listRule: '@request.auth.id != "" && owner = @request.auth.id',
+  // Users can only view their own records
+  viewRule: '@request.auth.id != "" && owner = @request.auth.id',
+  // Authenticated users can create
+  createRule: '@request.auth.id != ""',
+  // Users can only update their own records
+  updateRule: '@request.auth.id != "" && owner = @request.auth.id',
+  // Users can only delete their own records
+  deleteRule: '@request.auth.id != "" && owner = @request.auth.id',
+  fields: [
+    // ...
+    {
+      name: 'owner',
+      type: 'relation',
+      required: true,
+      maxSelect: 1,
+      collectionId: '_pb_users_auth_',
+    },
+  ],
+});
+```
+
+### 4.2 Admin-Only Collections
+
+For settings that apply across all users (app-level config):
+
+```typescript
+const adminCollection = new Collection({
+  name: 'plugin_settings',
+  // Only admins can access
+  listRule: 'id = @request.auth.id && @collection.permissions = "admin"',
+  viewRule: 'id = @request.auth.id && @collection.permissions = "admin"',
+  createRule: 'id = @request.auth.id && @collection.permissions = "admin"',
+  updateRule: 'id = @request.auth.id && @collection.permissions = "admin"',
+  deleteRule: 'id = @request.auth.id && @collection.permissions = "admin"',
+  fields: [ /* ... */ ],
+});
+```
+
+---
+
+## 5. Best Practices
+
+### 5.1 Migration Guidelines
+
+| Practice | Description |
+|----------|-------------|
+| **Version Numbers** | Use sequential integers (001, 002, 003...) |
+| **Idempotency** | Migrations should be safe to re-run |
+| **Backwards Compatible** | New migrations shouldn't break existing data |
+| **Test Locally** | Test migrations against a fresh PocketBase instance |
+| **Document Changes** | Add comments explaining schema changes |
+
+### 5.2 Error Handling
+
+```typescript
+// Always wrap migration execution in try-catch
+try {
+  await loadAndRunMigrations();
+} catch (error) {
+  if (error.message.includes('authentication')) {
+    showErrorDialog('Database authentication failed. Check admin credentials.');
+  } else if (error.message.includes('collection')) {
+    showErrorDialog('Failed to create collection. Check migration syntax.');
+  } else {
+    showErrorDialog('Migration failed. Check console for details.');
+  }
+  throw error; // Prevent app from starting with invalid schema
+}
+```
+
+### 5.3 Development vs Production
+
+**Development:**
+```bash
+# Start PocketBase with fresh data
+pocketbase serve --dev
+```
+
+**Production:**
+- Back up PocketBase data directory before migrations
+- Test migrations in staging first
+- Keep `_lego_migrations` table for audit trail
+
+---
+
+## 6. Alternative Approach: PB_MIGRATE Hook
+
+For production deployments, consider using PocketBase's **migrate** hook:
+
+**File:** `pocketbase/migrations/migrate.go` (Go - not covered in our TypeScript-only stack)
+
+Since we're using a **TypeScript-only** approach with the Admin API, the startup protocol is preferred over Go hooks.
+
+---
+
+## 7. Next Steps
+
+1. **`03-host-setup.md`**: Initialize the Modern.js host app
+2. **`04-plugin-architecture.md`**: Plugin development patterns
+3. **`05-slot-injection-pattern.md`**: UI extension system
+
+---
+
+## References
+
+- [PocketBase JS Collections Documentation](https://pocketbase.io/docs/js-collections/)
+- [PocketBase API Rules Guide](https://pocketbase.io/docs/collections/)
+- [PocketBase Admin API](https://pocketbase.io/docs/api-collections/)
+- [GitHub Discussion: Programmatically Create Collections](https://github.com/pocketbase/pocketbase/discussions/890)