@portel/photon 1.7.0 → 1.8.0

package/README.md

@@ -278,7 +278,7 @@ VideoProcessor requires the following CLI tools to be installed:
   - ffmpeg: Install from https://ffmpeg.org/download.html
 ```
 
-> See the full [Tag Reference](./DOCBLOCK-TAGS.md) for all available tags. There are 30+ covering validation, UI hints, scheduling, webhooks, and more.
+> See the full [Tag Reference](./docs/reference/DOCBLOCK-TAGS.md) for all available tags. There are 30+ covering validation, UI hints, scheduling, webhooks, and more.
 
 <div align="center">
 <img src="https://raw.githubusercontent.com/portel-dev/photon/main/assets/readme-step-4.png" alt="Step 4 — Validation and formatting in Beam" width="100%">
@@ -322,7 +322,7 @@ export default class Weather {
 
 **What changes in Beam:** Instead of the auto-generated table, results render inside your custom HTML (a weather dashboard with icons, charts, or any visualization you build). The `window.photon` API bridges your UI to the tool system.
 
-> Custom UIs follow the [MCP Apps Extension (SEP-1865)](https://github.com/nicolo-ribaudo/modelcontextprotocol/blob/nicolo/sep-1865/docs/specification/draft/extensions/apps.mdx) standard and work across compatible hosts. See the [Custom UI Guide](./CUSTOM-UI.md).
+> Custom UIs follow the [MCP Apps Extension (SEP-1865)](https://github.com/nicolo-ribaudo/modelcontextprotocol/blob/nicolo/sep-1865/docs/specification/draft/extensions/apps.mdx) standard and work across compatible hosts. See the [Custom UI Guide](./docs/guides/CUSTOM-UI.md).
 
 <div align="center">
 <img src="https://raw.githubusercontent.com/portel-dev/photon/main/assets/readme-step-5.png" alt="Step 5 — Custom UI result in Beam" width="100%">
@@ -351,12 +351,12 @@ If you are just skimming, here is what you need to know:
 | Concept | What it is | Learn more |
 |---------|-----------|------------|
 | **MCP** | A way for AI to use your tools. It's a standard. | [modelcontextprotocol.io](https://modelcontextprotocol.io/introduction) |
-| **Photon file** | A `.photon.ts` file. You define tools as methods in a class. | [Guide](./GUIDE.md) |
+| **Photon file** | A `.photon.ts` file. You define tools as methods in a class. | [Guide](./docs/GUIDE.md) |
 | **Beam** | A web dashboard. It shows your tools as forms. | [Beam UI](#beam) |
 | **Marketplace** | A way to get other people's photons. | [Marketplace](#marketplace) |
-| **Daemon** | A background thing that handles messages and jobs. | [Daemon Pub/Sub](./DAEMON-PUBSUB.md) |
-| **Tags** | JSDoc comments that tell Photon what to do. | [Tag Reference](./DOCBLOCK-TAGS.md) |
-| **Custom UI** | When the auto-generated forms aren't enough. | [Custom UI Guide](./CUSTOM-UI.md) |
+| **Daemon** | A background thing that handles messages and jobs. | [Daemon Pub/Sub](./docs/core/DAEMON-PUBSUB.md) |
+| **Tags** | JSDoc comments that tell Photon what to do. | [Tag Reference](./docs/reference/DOCBLOCK-TAGS.md) |
+| **Custom UI** | When the auto-generated forms aren't enough. | [Custom UI Guide](./docs/guides/CUSTOM-UI.md) |
 
 ---
 
@@ -377,7 +377,7 @@ photon maker new <name>           # Scaffold a new photon
 # Manage
 photon info                       # List all photons
 photon info <name> --mcp          # Get MCP client config (paste into Claude/Cursor)
-photon validate <name>            # Check for errors
+photon maker validate <name>      # Check for errors
 
 # Marketplace
 photon add <name>                 # Install photon
@@ -386,7 +386,6 @@ photon upgrade                    # Upgrade all
 
 # Ops
 photon doctor                     # Diagnose environment
-photon audit                      # Security audit
 photon test                       # Run tests
 ```
 
@@ -412,7 +411,7 @@ Tags are JSDoc annotations that control how Photon processes your code. Here are
 | `@mcp` | Class | Inject another MCP server as a dependency |
 | `@icon` | Class/Method | Set emoji icon |
 
-> This is a subset. See the full [Tag Reference](./DOCBLOCK-TAGS.md) for all 30+ tags with examples.
+> This is a subset. See the full [Tag Reference](./docs/reference/DOCBLOCK-TAGS.md) for all 30+ tags with examples.
 
 ---
 
@@ -422,33 +421,33 @@ Tags are JSDoc annotations that control how Photon processes your code. Here are
 
 | Guide | |
 |-------|-|
-| [Getting Started](./GUIDE.md) | Create your first photon, step by step |
-| [Tag Reference](./DOCBLOCK-TAGS.md) | Complete JSDoc tag reference with examples |
-| [Naming Conventions](./NAMING-CONVENTIONS.md) | How to name methods so they read naturally as CLI commands |
-| [Troubleshooting](./TROUBLESHOOTING.md) | Common issues and solutions |
+| [Getting Started](./docs/GUIDE.md) | Create your first photon, step by step |
+| [Tag Reference](./docs/reference/DOCBLOCK-TAGS.md) | Complete JSDoc tag reference with examples |
+| [Naming Conventions](./docs/guides/NAMING-CONVENTIONS.md) | How to name methods so they read naturally as CLI commands |
+| [Troubleshooting](./docs/TROUBLESHOOTING.md) | Common issues and solutions |
 
 **Build more:**
 
 | Topic | |
 |-------|-|
-| [Custom UI](./CUSTOM-UI.md) | Build rich interactive interfaces with `window.photon` |
-| [OAuth](./AUTH.md) | Built-in OAuth 2.1 with Google, GitHub, Microsoft |
-| [Daemon Pub/Sub](./DAEMON-PUBSUB.md) | Real-time cross-process messaging |
-| [Webhooks](./WEBHOOKS.md) | HTTP endpoints for external services |
-| [Locks](./LOCKS.md) | Distributed locks for exclusive access |
-| [Advanced Patterns](./ADVANCED.md) | Lifecycle hooks, dependency injection, interactive workflows |
-| [Deployment](./DEPLOYMENT.md) | Docker, Cloudflare Workers, AWS Lambda, Systemd |
+| [Custom UI](./docs/guides/CUSTOM-UI.md) | Build rich interactive interfaces with `window.photon` |
+| [OAuth](./docs/guides/AUTH.md) | Built-in OAuth 2.1 with Google, GitHub, Microsoft |
+| [Daemon Pub/Sub](./docs/core/DAEMON-PUBSUB.md) | Real-time cross-process messaging |
+| [Webhooks](./docs/reference/WEBHOOKS.md) | HTTP endpoints for external services |
+| [Locks](./docs/reference/LOCKS.md) | Distributed locks for exclusive access |
+| [Advanced Patterns](./docs/guides/ADVANCED.md) | Lifecycle hooks, dependency injection, interactive workflows |
+| [Deployment](./docs/guides/DEPLOYMENT.md) | Docker, Cloudflare Workers, AWS Lambda, Systemd |
 
 **Operate:**
 
 | Topic | |
 |-------|-|
 | [Security](./SECURITY.md) | Best practices and audit checklist |
-| [Marketplace Publishing](./MARKETPLACE-PUBLISHING.md) | Create and share team marketplaces |
-| [Best Practices](./PHOTON_BEST_PRACTICES.md) | Patterns for production photons |
-| [Comparison](./COMPARISON.md) | Benchmarks vs official MCP implementations |
+| [Marketplace Publishing](./docs/guides/MARKETPLACE-PUBLISHING.md) | Create and share team marketplaces |
+| [Best Practices](./docs/guides/BEST-PRACTICES.md) | Patterns for production photons |
+| [Comparison](./docs/COMPARISON.md) | Benchmarks vs official MCP implementations |
 
-**Reference:** [Architecture](./ARCHITECTURE.md) · [Changelog](./CHANGELOG.md) · [Contributing](./CONTRIBUTING.md)
+**Reference:** [Architecture](./docs/core/ARCHITECTURE.md) · [Changelog](./CHANGELOG.md) · [Contributing](./CONTRIBUTING.md)
 
 ---
 

package/dist/auto-ui/beam.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"beam.d.ts","sourceRoot":"","sources":["../../src/auto-ui/beam.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AAixBH,wBAAsB,SAAS,CAAC,aAAa,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CA0/ElF;AAiYD;;;GAGG;AACH,wBAAsB,QAAQ,IAAI,OAAO,CAAC,IAAI,CAAC,CAsB9C"}
+{"version":3,"file":"beam.d.ts","sourceRoot":"","sources":["../../src/auto-ui/beam.ts"],"names":[],"mappings":"AAAA;;;;;;GAMG;AA2xBH,wBAAsB,SAAS,CAAC,aAAa,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAwkFlF;AAiYD;;;GAGG;AACH,wBAAsB,QAAQ,IAAI,OAAO,CAAC,IAAI,CAAC,CAsB9C"}

package/dist/auto-ui/beam.js

@@ -14,7 +14,7 @@ import * as os from 'os';
 import { spawn } from 'child_process';
 import { fileURLToPath } from 'url';
 import { createHash } from 'crypto';
-import { isPathWithin, isLocalRequest, setSecurityHeaders, readBody, SimpleRateLimiter } from '../shared/security.js';
+import { isPathWithin, isLocalRequest, setSecurityHeaders, readBody, SimpleRateLimiter, } from '../shared/security.js';
 /**
  * Generate a unique ID for a photon based on its path.
  * This ensures photons with the same name from different paths are distinguishable.
@@ -29,15 +29,17 @@ const __dirname = path.dirname(__filename);
 import { listPhotonMCPs, resolvePhotonPath } from '../path-resolver.js';
 import { PhotonLoader } from '../loader.js';
 import { logger, createLogger } from '../shared/logger.js';
+import { getErrorMessage } from '../shared/error-handler.js';
 import { toEnvVarName } from '../shared/config-docs.js';
 import { MarketplaceManager } from '../marketplace-manager.js';
 import { PhotonDocExtractor } from '../photon-doc-extractor.js';
 import { TemplateManager } from '../template-manager.js';
 import { subscribeChannel, pingDaemon } from '../daemon/client.js';
+import { ensureDaemon } from '../daemon/manager.js';
 import { SchemaExtractor, } from '@portel/photon-core';
 import { generateOpenAPISpec } from './openapi-generator.js';
 import { handleStreamableHTTP, broadcastNotification, broadcastToBeam, sendToSession, requestExternalElicitation, } from './streamable-http-transport.js';
-import { SDKMCPClientFactory } from '../mcp-client.js';
+import { SDKMCPClientFactory } from '@portel/photon-core';
 import { getBundledPhotonPath, BEAM_BUNDLED_PHOTONS } from '../shared-utils.js';
 // SDK imports for direct resource access (transport wrapper doesn't expose these yet)
 import { Client } from '@modelcontextprotocol/sdk/client/index.js';
@@ -46,7 +48,7 @@ import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/
 import { SSEClientTransport } from '@modelcontextprotocol/sdk/client/sse.js';
 import { ElicitRequestSchema } from '@modelcontextprotocol/sdk/types.js';
 // Config file path
-const CONFIG_FILE = path.join(os.homedir(), '.photon', 'config.json');
+const CONFIG_FILE = process.env.PHOTON_CONFIG_FILE || path.join(os.homedir(), '.photon', 'config.json');
 // ════════════════════════════════════════════════════════════════════════════════
 // EXTERNAL MCP STATE (module-level for MCP transport access)
 // ════════════════════════════════════════════════════════════════════════════════
@@ -277,19 +279,10 @@ async function loadExternalMCPs(config) {
                     mcpInfo.methods = methods;
                 }
                 catch (sdkError) {
-                    // SDK client failed - fall back to wrapper client
-                    logger.debug(`SDK client failed for ${name}, using wrapper: ${sdkError}`);
-                    // Try wrapper client as fallback
-                    const tools = await client.list();
-                    methods = (tools || []).map((tool) => ({
-                        name: tool.name,
-                        description: tool.description || '',
-                        params: tool.inputSchema || { type: 'object', properties: {} },
-                        returns: { type: 'object' },
-                        icon: tool['x-icon'],
-                    }));
-                    mcpInfo.connected = true;
-                    mcpInfo.methods = methods;
+                    // SDK client failed — don't fall back to wrapper for stdio MCPs
+                    // (same command would fail identically, and wrapper spawns a process
+                    // without stderr suppression, leaking raw Node.js stack traces)
+                    throw sdkError;
                 }
             }
             else {
@@ -318,7 +311,17 @@ async function loadExternalMCPs(config) {
         catch (error) {
             const errorMsg = error instanceof Error ? error.message : String(error);
             mcpInfo.errorMessage = errorMsg.slice(0, 200);
-            logger.warn(`⚠️ Failed to connect to external MCP: ${name} - ${errorMsg}`);
+            // User-friendly error messages for common failures
+            const shortMsg = errorMsg.includes('Cannot find module')
+                ? `Module not found (run npm build in the MCP directory)`
+                : errorMsg.includes('ENOENT')
+                    ? `Command not found: ${serverConfig.command}`
+                    : errorMsg.includes('Connection timeout')
+                        ? `Connection timed out (server may not be running)`
+                        : errorMsg.includes('Connection closed')
+                            ? `Server exited immediately (check configuration)`
+                            : errorMsg.slice(0, 120);
+            logger.warn(`⚠️  External MCP "${name}" — ${shortMsg}`);
         }
         results.push(mcpInfo);
     }
@@ -799,6 +802,7 @@ export async function startBeam(rawWorkingDir, port) {
             catch {
                 // No install metadata - that's fine
             }
+            const isStateful = schemaSource ? /@stateful\b/.test(schemaSource) : false;
             return {
                 id: generatePhotonId(photonPath),
                 name,
@@ -818,6 +822,7 @@ export async function startBeam(rawWorkingDir, port) {
                 resourceCount,
                 promptCount,
                 installSource,
+                ...(isStateful && { stateful: true }),
                 ...(constructorParams.length > 0 && { requiredParams: constructorParams }),
                 ...(mcp.injectedPhotons &&
                     mcp.injectedPhotons.length > 0 && { injectedPhotons: mcp.injectedPhotons }),
@@ -840,57 +845,58 @@ export async function startBeam(rawWorkingDir, port) {
         }
     }
     const channelSubscriptions = new Map();
-    const EVENT_BUFFER_SIZE = 30; // Keep last 30 events per channel
+    /** Buffer retention window — events older than this are purged */
+    const EVENT_BUFFER_DURATION_MS = 5 * 60 * 1000; // 5 minutes
     const channelEventBuffers = new Map();
     // Store an event in the channel buffer
     function bufferEvent(channel, method, params) {
         let buffer = channelEventBuffers.get(channel);
         if (!buffer) {
-            buffer = { events: [], nextId: 1 };
+            buffer = { events: [] };
             channelEventBuffers.set(channel, buffer);
         }
-        const eventId = buffer.nextId++;
+        const now = Date.now();
         const event = {
-            id: eventId,
+            id: now,
             method,
             params,
-            timestamp: Date.now(),
+            timestamp: now,
         };
         buffer.events.push(event);
-        // Keep only last N events (circular buffer)
-        if (buffer.events.length > EVENT_BUFFER_SIZE) {
+        // Purge events older than retention window
+        const cutoff = now - EVENT_BUFFER_DURATION_MS;
+        while (buffer.events.length > 0 && buffer.events[0].timestamp < cutoff) {
             buffer.events.shift();
         }
-        return eventId;
+        return now;
     }
-    // Replay missed events to a specific session, or signal refresh needed
-    function replayEventsToSession(sessionId, channel, lastEventId) {
+    // Replay missed events to a specific session, or signal full sync needed
+    function replayEventsToSession(sessionId, channel, lastTimestamp) {
         const buffer = channelEventBuffers.get(channel);
         // No buffer = no events ever sent on this channel
         if (!buffer || buffer.events.length === 0) {
             return { replayed: 0, refreshNeeded: false };
         }
-        // No lastEventId = client is fresh, no replay needed
-        if (lastEventId === undefined) {
+        // No lastTimestamp = client is fresh, no replay needed
+        if (lastTimestamp === undefined) {
             return { replayed: 0, refreshNeeded: false };
         }
         const oldestEvent = buffer.events[0];
-        // If lastEventId is older than our oldest buffered event, signal refresh needed
-        if (lastEventId < oldestEvent.id) {
+        // Stale: client's timestamp is older than buffer window → full sync needed
+        if (lastTimestamp < oldestEvent.timestamp) {
             sendToSession(sessionId, 'photon/refresh-needed', { channel });
-            logger.info(`📡 Replay: ${channel} - lastEventId ${lastEventId} too old (oldest: ${oldestEvent.id}), refresh needed`);
+            logger.info(`📡 Stale client on ${channel} - last seen ${new Date(lastTimestamp).toISOString()}, oldest buffered ${new Date(oldestEvent.timestamp).toISOString()}, full sync needed`);
             return { replayed: 0, refreshNeeded: true };
         }
-        // Find events to replay (all events after lastEventId)
-        const eventsToReplay = buffer.events.filter((e) => e.id > lastEventId);
+        // Delta sync: replay events after client's last timestamp
+        const eventsToReplay = buffer.events.filter((e) => e.timestamp > lastTimestamp);
         if (eventsToReplay.length === 0) {
             return { replayed: 0, refreshNeeded: false };
         }
-        // Replay each missed event to this session
         for (const event of eventsToReplay) {
-            sendToSession(sessionId, event.method, { ...event.params, _eventId: event.id });
+            sendToSession(sessionId, event.method, { ...event.params, _eventId: event.timestamp });
         }
-        logger.info(`📡 Replay: ${channel} - replayed ${eventsToReplay.length} events (${lastEventId + 1} to ${buffer.nextId - 1})`);
+        logger.info(`📡 Delta sync: ${channel} - replayed ${eventsToReplay.length} events`);
         return { replayed: eventsToReplay.length, refreshNeeded: false };
     }
     // ══════════════════════════════════════════════════════════════════════════════
@@ -964,8 +970,8 @@ export async function startBeam(rawWorkingDir, port) {
     // Called when a client starts viewing a board (from MCP notification)
     // photonId: hash of photon path (unique across servers)
     // itemId: whatever the photon uses to identify the item (e.g., board name)
-    // lastEventId: optional - if provided, replay missed events or signal refresh needed
-    function onClientViewingBoard(sessionId, photonId, itemId, lastEventId) {
+    // lastTimestamp: optional - if provided, delta sync missed events or signal full sync needed
+    function onClientViewingBoard(sessionId, photonId, itemId, lastTimestamp) {
         const prevState = sessionViewState.get(sessionId);
         // Unsubscribe from previous item if different
         if (prevState?.itemId && (prevState.photonId !== photonId || prevState.itemId !== itemId)) {
@@ -976,9 +982,9 @@ export async function startBeam(rawWorkingDir, port) {
         const channel = `${photonId}:${itemId}`;
         sessionViewState.set(sessionId, { photonId, itemId });
         subscribeToChannel(channel);
-        // Replay missed events if lastEventId is provided
-        if (lastEventId !== undefined) {
-            replayEventsToSession(sessionId, channel, lastEventId);
+        // Delta sync missed events if lastTimestamp is provided
+        if (lastTimestamp !== undefined) {
+            replayEventsToSession(sessionId, channel, lastTimestamp);
         }
     }
     // Called when a client disconnects
@@ -2265,6 +2271,26 @@ export async function startBeam(rawWorkingDir, port) {
             logger.info(isNewPhoton
                 ? `✨ New photon detected: ${photonName}`
                 : `🔄 File change detected, reloading ${photonName}...`);
+            // Auto-scaffold empty photon files with a starter template
+            if (isNewPhoton) {
+                try {
+                    const rawContent = await fs.readFile(photonPath, 'utf-8');
+                    if (rawContent.trim().length === 0) {
+                        const className = photonName
+                            .split(/[-_]/)
+                            .map((w) => w.charAt(0).toUpperCase() + w.slice(1))
+                            .join('');
+                        const scaffold = `/**\n * ${className} Photon\n */\n\nexport default class ${className} {\n  /**\n   * Example tool\n   * @param message Message to echo\n   */\n  async echo(params: { message: string }) {\n    return \`Echo: \${params.message}\`;\n  }\n}\n`;
+                        await fs.writeFile(photonPath, scaffold, 'utf-8');
+                        logger.info(`📝 Scaffolded empty file: ${photonName}.photon.ts`);
+                        // The write triggers another watcher event which will load the scaffolded photon
+                        return;
+                    }
+                }
+                catch {
+                    // File read failed, continue with normal load attempt
+                }
+            }
             // For new photons, check if configuration is needed first
             if (isNewPhoton) {
                 const extractor = new SchemaExtractor();
@@ -2375,6 +2401,7 @@ export async function startBeam(rawWorkingDir, port) {
                     // Can't extract params
                 }
                 backfillEnvDefaults(mcp.instance, reloadConstructorParams);
+                const isStateful = /@stateful\b/.test(reloadSource);
                 const reloadedPhoton = {
                     id: generatePhotonId(photonPath),
                     name: photonName,
@@ -2386,6 +2413,7 @@ export async function startBeam(rawWorkingDir, port) {
                     description: reloadClassMeta.description,
                     icon: reloadClassMeta.icon,
                     internal: reloadClassMeta.internal,
+                    ...(isStateful && { stateful: true }),
                     ...(reloadConstructorParams.length > 0 && { requiredParams: reloadConstructorParams }),
                     ...(mcp.injectedPhotons &&
                         mcp.injectedPhotons.length > 0 && { injectedPhotons: mcp.injectedPhotons }),
@@ -2541,6 +2569,14 @@ export async function startBeam(rawWorkingDir, port) {
                 console.log(`\n⚡ Photon Beam → ${url} (loading photons...)\n`);
                 resolve();
             });
+            // Configure server and socket timeouts to prevent premature disconnections
+            // Disable server timeout for long-lived SSE connections (0 = no timeout)
+            server.setTimeout(0);
+            // Enable TCP keepalive on all connections to prevent intermediary timeouts
+            server.on('connection', (socket) => {
+                socket.setKeepAlive(true, 60000); // Send keepalive probe every 60s
+                socket.setTimeout(0); // Disable socket inactivity timeout
+            });
         };
         tryListen();
     });
@@ -2569,6 +2605,45 @@ export async function startBeam(rawWorkingDir, port) {
     console.log(`⚡ Photon Beam ready (${photonStatus}${mcpStatus})`);
     // Notify connected clients that photon list is now available
     broadcastPhotonChange();
+    // Auto-start daemon and subscribe to state-changed events for stateful photons
+    // Uses reconnect: true so subscriptions survive daemon restarts
+    const statefulPhotons = photons.filter((p) => p.stateful && p.configured);
+    if (statefulPhotons.length > 0) {
+        try {
+            await ensureDaemon();
+            for (const photon of statefulPhotons) {
+                const photonName = photon.name;
+                const channel = `${photonName}:state-changed`;
+                subscribeChannel(photonName, channel, (message) => {
+                    broadcastToBeam('photon/state-changed', {
+                        photon: photonName,
+                        method: message?.method,
+                        data: message?.data,
+                    });
+                }, {
+                    reconnect: true,
+                    onReconnect: () => logger.info(`📡 Reconnected ${channel} subscription`),
+                    onRefreshNeeded: () => {
+                        logger.info(`📡 Refresh needed for ${channel} (events lost during daemon restart)`);
+                        broadcastToBeam('photon/state-changed', {
+                            photon: photonName,
+                            method: '_refresh',
+                            data: {},
+                        });
+                    },
+                })
+                    .then(() => {
+                    logger.info(`📡 Subscribed to ${channel} for cross-client sync`);
+                })
+                    .catch((err) => {
+                    logger.warn(`Failed to subscribe to ${channel}: ${getErrorMessage(err)}`);
+                });
+            }
+        }
+        catch (err) {
+            logger.warn(`Failed to start daemon for stateful photons: ${getErrorMessage(err)}`);
+        }
+    }
     // Set up file watchers for symlinked and bundled photon assets (now that photons are loaded)
     for (const photon of photons) {
         if (!photon.path) {