@push.rocks/smartproxy 19.4.2 → 19.5.2

package/readme.plan.md

@@ -1,384 +1,316 @@
 # SmartProxy Development Plan
 
-## ACME Route Port Binding Intelligence Improvement
-
-### Problem Statement
-Currently, SmartProxy has an issue with port binding conflicts between regular routes and ACME challenge routes. While SmartProxy is designed to support multiple routes sharing the same port (differentiated by host, path, etc.), there's a specific conflict when adding ACME challenge routes to a port that is already in use by other routes.
-
-This results in the error: `Port 80 is already in use for ACME challenges` when SmartProxy tries to bind the ACME challenge route to a port that it's already using.
-
-### Root Cause Analysis
-1. **Double Binding Attempt**: SmartProxy tries to bind to port 80 twice - once for application routes and once for ACME challenge routes.
-2. **Overlapping Route Updates**: When adding a challenge route, it triggers a port binding operation without checking if the port is already bound.
-3. **Naive Error Handling**: The code detects EADDRINUSE but doesn't distinguish between external conflicts and internal conflicts.
-4. **Port Binding Semantics**: The port manager doesn't recognize that a port already bound by SmartProxy can be reused for additional routes.
-
-### Solution Architecture
-We need a more intelligent approach to port binding that understands when a port can be shared between routes vs. when a new binding is needed:
-
-1. **Port Binding Awareness**: Track what ports are already bound by SmartProxy itself.
-2. **Smart Route Updates**: Only attempt to bind to ports that aren't already bound by SmartProxy.
-3. **Route Merging Logic**: When adding ACME challenge routes, merge them with existing routes on the same ports.
-4. **Dynamic Port Management**: Release port bindings when no routes are using them and rebind when needed.
-5. **Improved Error Recovery**: Handle port conflicts gracefully, with distinct handling for internal vs. external conflicts.
-
-### Implementation Plan
-
-#### Phase 1: Improve Port Manager Intelligence
-- [x] Enhance `PortManager` to distinguish between ports that need new bindings vs ports that can reuse existing bindings
-- [x] Add an internal tracking mechanism to detect when a requested port is already bound internally
-- [x] Modify port addition logic to skip binding operations for ports already bound by SmartProxy
-- [x] Implement reference counting for port bindings to track how many routes use each port
-- [x] Add logic to release port bindings when no routes are using them anymore
-- [x] Update error handling to provide more context for port binding failures
-
-#### Phase 2: Refine ACME Challenge Route Integration
-- [x] Modify `addChallengeRoute()` to check if the port is already in use by SmartProxy
-- [x] Ensure route updates don't trigger unnecessary port binding operations
-- [x] Implement a merging strategy for ACME routes with existing routes on the same port
-- [x] Add diagnostic logging to track route and port binding relationships
-
-#### Phase 3: Enhance Proxy Route Management
-- [x] Restructure route update process to group routes by port
-- [x] Implement a more efficient route update mechanism that minimizes port binding operations
-- [x] Develop port lifecycle management to track usage across route changes
-- [x] Add validation to detect potential binding conflicts before attempting operations
-- [ ] Create a proper route dependency graph to understand the relationships between routes
-- [x] Implement efficient detection of "orphaned" ports that no longer have associated routes
-
-#### Phase 4: Improve Error Handling and Recovery
-- [x] Enhance error messages to be more specific about the nature of port conflicts
-- [x] Add recovery mechanisms for common port binding scenarios
-- [ ] Implement a fallback port selection strategy for ACME challenges
-- [x] Create a more robust validation system to catch issues before they cause runtime errors
-
-### Detailed Technical Tasks
-
-#### Phase 1: Improve Port Manager Intelligence
-1. Modify `/ts/proxies/smart-proxy/port-manager.ts`:
-   - Add a new method `isPortBoundBySmartProxy(port: number): boolean`
-   - Refactor `addPort()` to check if the port is already bound
-   - Update `updatePorts()` to be more intelligent about which ports need binding
-   - Add reference counting for port usage
-
-2. Implement Port Reference Counting:
-   ```typescript
-   // Add to PortManager class
-   private portRefCounts: Map<number, number> = new Map();
-   
-   public incrementPortRefCount(port: number): void {
-     const currentCount = this.portRefCounts.get(port) || 0;
-     this.portRefCounts.set(port, currentCount + 1);
-     logger.log('debug', `Port ${port} reference count increased to ${currentCount + 1}`, { port, refCount: currentCount + 1 });
-   }
-   
-   public decrementPortRefCount(port: number): number {
-     const currentCount = this.portRefCounts.get(port) || 0;
-     if (currentCount <= 0) {
-       logger.log('warn', `Attempted to decrement reference count for port ${port} below zero`, { port });
-       return 0;
-     }
-     
-     const newCount = currentCount - 1;
-     this.portRefCounts.set(port, newCount);
-     logger.log('debug', `Port ${port} reference count decreased to ${newCount}`, { port, refCount: newCount });
-     return newCount;
-   }
-   
-   public getPortRefCount(port: number): number {
-     return this.portRefCounts.get(port) || 0;
-   }
-   ```
-
-3. Port Binding Logic Enhancements:
-   ```typescript
-   public async addPort(port: number): Promise<void> {
-     // If already bound by this instance, just increment ref count and return
-     if (this.servers.has(port)) {
-       this.incrementPortRefCount(port);
-       logger.log('debug', `Port ${port} is already bound by SmartProxy, reusing binding`, { port });
-       return;
-     }
-     
-     // Initialize ref count for new port
-     this.portRefCounts.set(port, 1);
-     
-     // Continue with normal binding...
-   }
-   
-   public async removePort(port: number): Promise<void> {
-     // Decrement reference count
-     const newCount = this.decrementPortRefCount(port);
-     
-     // If port is still in use by other routes, keep it
-     if (newCount > 0) {
-       logger.log('debug', `Port ${port} still in use by ${newCount} routes, keeping binding open`, { port, refCount: newCount });
-       return;
-     }
-     
-     // No more references, can actually close the port
-     const server = this.servers.get(port);
-     if (!server) {
-       logger.log('warn', `Port ${port} not found in servers map`, { port });
-       return;
-     }
-     
-     // Continue with normal unbinding logic...
-   }
-   ```
-
-4. Add Smarter Port Conflict Detection:
-   ```typescript
-   private isPortConflict(error: any): { isConflict: boolean; isExternal: boolean } {
-     if (error.code !== 'EADDRINUSE') {
-       return { isConflict: false, isExternal: false };
-     }
-     
-     // Check if we already have this port
-     const isBoundInternally = this.servers.has(Number(error.port));
-     return { isConflict: true, isExternal: !isBoundInternally };
-   }
-   ```
-
-#### Phase 2: Refine ACME Challenge Route Integration
-1. Modify `/ts/proxies/smart-proxy/certificate-manager.ts`:
-   - Enhance `addChallengeRoute()` to be aware of existing port bindings
-   - Add port verification before attempting to add challenge routes
-
-2. Smart Route Merging Logic:
-   ```typescript
-   private async addChallengeRoute(): Promise<void> {
-     // Check if route is already active
-     if (this.challengeRouteActive) {
-       return;
-     }
-
-     // Create challenge route
-     const challengeRoute = this.challengeRoute;
-     const challengePort = this.globalAcmeDefaults?.port || 80;
-     
-     // Check if port is already in use by another route
-     const portAlreadyUsed = this.routes.some(r => 
-       Array.isArray(r.match.ports) 
-       ? r.match.ports.includes(challengePort)
-       : r.match.ports === challengePort
-     );
-     
-     if (portAlreadyUsed) {
-       logger.log('info', `Port ${challengePort} is already used by an existing route, merging ACME challenge route`);
-     }
-     
-     // Continue with route update...
-   }
-   ```
-
-3. Update Route Manager Communication:
-   ```typescript
-   // Add this method to smart-proxy.ts
-   private async addRouteWithoutRebinding(route: IRouteConfig): Promise<void> {
-     // Add route to configuration without triggering a port rebind
-     this.settings.routes.push(route);
-     this.routeManager.updateRoutes(this.settings.routes);
-     
-     // Update HttpProxy if needed, but skip port binding updates
-     if (this.httpProxyBridge.getHttpProxy()) {
-       await this.httpProxyBridge.syncRoutesToHttpProxy(this.settings.routes);
-     }
-   }
-   ```
-
-#### Phase 3: Enhance Proxy Route Management
-1. Modify `/ts/proxies/smart-proxy/smart-proxy.ts`:
-   - Refactor `updateRoutes()` to group routes by port
-   - Implement incremental updates that preserve port bindings
-   - Add orphaned port detection and cleanup
-
-2. Group Routes by Port:
-   ```typescript
-   private groupRoutesByPort(routes: IRouteConfig[]): Map<number, IRouteConfig[]> {
-     const portMap = new Map<number, IRouteConfig[]>();
-     
-     for (const route of routes) {
-       const ports = Array.isArray(route.match.ports) 
-         ? route.match.ports 
-         : [route.match.ports];
-         
-       for (const port of ports) {
-         if (!portMap.has(port)) {
-           portMap.set(port, []);
-         }
-         portMap.get(port)!.push(route);
-       }
-     }
-     
-     return portMap;
-   }
-   ```
-
-3. Implement Port Usage Tracking:
-   ```typescript
-   private updatePortUsageMap(routes: IRouteConfig[]): Map<number, Set<string>> {
-     // Map of port -> Set of route names using that port
-     const portUsage = new Map<number, Set<string>>();
-     
-     for (const route of routes) {
-       const ports = Array.isArray(route.match.ports) 
-         ? route.match.ports 
-         : [route.match.ports];
-         
-       const routeName = route.name || `unnamed_${Math.random().toString(36).substring(2, 9)}`;
-       
-       for (const port of ports) {
-         if (!portUsage.has(port)) {
-           portUsage.set(port, new Set());
-         }
-         portUsage.get(port)!.add(routeName);
-       }
-     }
-     
-     return portUsage;
-   }
-   
-   private findOrphanedPorts(oldUsage: Map<number, Set<string>>, newUsage: Map<number, Set<string>>): number[] {
-     // Find ports that have no routes in new configuration
-     const orphanedPorts: number[] = [];
-     
-     for (const [port, routes] of oldUsage.entries()) {
-       if (!newUsage.has(port) || newUsage.get(port)!.size === 0) {
-         orphanedPorts.push(port);
-         logger.log('info', `Port ${port} no longer has any associated routes, will be released`, { port });
-       }
-     }
-     
-     return orphanedPorts;
-   }
-   ```
-
-4. Implement Incremental Update Logic:
-   ```typescript
-   public async updateRoutesIncremental(newRoutes: IRouteConfig[]): Promise<void> {
-     // Track port usage before and after update
-     const oldPortUsage = this.updatePortUsageMap(this.settings.routes);
-     const newPortUsage = this.updatePortUsageMap(newRoutes);
-     
-     // Find orphaned ports - ports that no longer have any routes
-     const orphanedPorts = this.findOrphanedPorts(oldPortUsage, newPortUsage);
-     
-     // Ports that need new bindings - not in old configuration
-     const newBindingPorts = [...newPortUsage.keys()].filter(p => !oldPortUsage.has(p));
-     
-     // Close orphaned ports
-     if (orphanedPorts.length > 0) {
-       logger.log('info', `Releasing ${orphanedPorts.length} orphaned ports: ${orphanedPorts.join(', ')}`, { ports: orphanedPorts });
-       await this.portManager.removePorts(orphanedPorts);
-     }
-     
-     // Bind to new ports
-     if (newBindingPorts.length > 0) {
-       logger.log('info', `Binding to ${newBindingPorts.length} new ports: ${newBindingPorts.join(', ')}`, { ports: newBindingPorts });
-       await this.portManager.addPorts(newBindingPorts);
-     }
-     
-     // Update route configuration
-     this.settings.routes = newRoutes;
-     this.routeManager.updateRoutes(newRoutes);
-     
-     // Update other components...
-   }
-   ```
-
-#### Phase 4: Improve Error Handling and Recovery
-1. Enhance Error Reporting:
-   ```typescript
-   private handlePortBindingError(port: number, error: any): void {
-     if (error.code === 'EADDRINUSE') {
-       const isInternalConflict = this.portManager.isPortBoundBySmartProxy(port);
-       if (isInternalConflict) {
-         logger.log('warn', `Port ${port} is already bound by SmartProxy. This is likely a route configuration issue.`, { port });
-       } else {
-         logger.log('error', `Port ${port} is in use by another application. Please choose a different port.`, { port });
-       }
-     } else {
-       logger.log('error', `Failed to bind to port ${port}: ${error.message}`, { port, error });
-     }
-   }
-   ```
-
-2. Implement ACME Port Fallback Strategy:
-   ```typescript
-   private async selectAcmePort(): Promise<number> {
-     const preferredPort = this.globalAcmeDefaults?.port || 80;
-     
-     // Check if preferred port is already bound internally
-     if (this.portManager.isPortBoundBySmartProxy(preferredPort)) {
-       // We can use it without a new binding
-       return preferredPort;
-     }
-     
-     // Try to bind to preferred port
-     try {
-       // Temporary test binding
-       const server = plugins.net.createServer();
-       await new Promise<void>((resolve, reject) => {
-         server.listen(preferredPort, () => {
-           server.close();
-           resolve();
-         }).on('error', reject);
-       });
-       
-       // If we get here, port is available
-       return preferredPort;
-     } catch (error) {
-       if (error.code === 'EADDRINUSE') {
-         // Port is unavailable, try fallback ports
-         for (const fallbackPort of [8080, 8081, 8082, 8083, 8084]) {
-           try {
-             // Test if we can bind to fallback
-             const server = plugins.net.createServer();
-             await new Promise<void>((resolve, reject) => {
-               server.listen(fallbackPort, () => {
-                 server.close();
-                 resolve();
-               }).on('error', reject);
-             });
-             
-             logger.log('warn', `Primary ACME port ${preferredPort} is unavailable, using fallback port ${fallbackPort}`);
-             return fallbackPort;
-           } catch {
-             // Try next fallback
-           }
-         }
-       }
-       
-       // All attempts failed
-       throw new Error(`Could not find an available port for ACME challenges`);
-     }
-   }
-   ```
-
-### Testing Strategy
-1. **Unit Tests**:
-   - Test port binding intelligence
-   - Test route merging logic
-   - Test error handling mechanisms
-   - Test port reference counting
-   - Test orphaned port detection and cleanup
-
-2. **Integration Tests**:
-   - Test multiple routes on the same port
-   - Test ACME challenges on ports with existing routes
-   - Test dynamic route addition and removal
-   - Test port lifecycle (bind → share → release)
-   - Test various recovery scenarios
-
-3. **Stress Tests**:
-   - Test rapid route updates
-   - Test concurrent operations
-   - Test large scale route changes (add/remove many at once)
-   - Test frequent changes to see if ports are properly released
-   - Test recovery from port conflicts
-
-### Release Plan
-1. **19.4.0** - Phase 1 & 2: Port Manager and ACME Route Improvements
-2. **19.5.0** - Phase 3: Enhanced Route Management
-3. **19.6.0** - Phase 4: Improved Error Handling and Recovery
+## Implementation Plan: Socket Handler Function Support (Simplified) ✅ COMPLETED
+
+### Overview
+Add support for custom socket handler functions with the simplest possible API - just pass a function that receives the socket.
+
+### User Experience Goal
+```typescript
+const proxy = new SmartProxy({
+  routes: [{
+    name: 'my-custom-protocol',
+    match: { ports: 9000, domains: 'custom.example.com' },
+    action: {
+      type: 'socket-handler',
+      socketHandler: (socket) => {
+        // User has full control of the socket
+        socket.write('Welcome!\n');
+        socket.on('data', (data) => {
+          socket.write(`Echo: ${data}`);
+        });
+      }
+    }
+  }]
+});
+```
+
+That's it. Simple and powerful.
+
+---
+
+## Phase 1: Minimal Type Changes
+
+### 1.1 Add Socket Handler Action Type
+**File:** `ts/proxies/smart-proxy/models/route-types.ts`
+
+```typescript
+// Update action type
+export type TRouteActionType = 'forward' | 'redirect' | 'block' | 'static' | 'socket-handler';
+
+// Add simple socket handler type
+export type TSocketHandler = (socket: net.Socket) => void | Promise<void>;
+
+// Extend IRouteAction
+export interface IRouteAction {
+  // ... existing properties
+  
+  // Socket handler function (when type is 'socket-handler')
+  socketHandler?: TSocketHandler;
+}
+```
+
+---
+
+## Phase 2: Simple Implementation
+
+### 2.1 Update Route Connection Handler
+**File:** `ts/proxies/smart-proxy/route-connection-handler.ts`
+
+In the `handleConnection` method, add handling for socket-handler:
+
+```typescript
+// After route matching...
+if (matchedRoute) {
+  const action = matchedRoute.action;
+  
+  if (action.type === 'socket-handler') {
+    if (!action.socketHandler) {
+      logger.error('socket-handler action missing socketHandler function');
+      socket.destroy();
+      return;
+    }
+    
+    try {
+      // Simply call the handler with the socket
+      const result = action.socketHandler(socket);
+      
+      // If it returns a promise, handle errors
+      if (result instanceof Promise) {
+        result.catch(error => {
+          logger.error('Socket handler error:', error);
+          if (!socket.destroyed) {
+            socket.destroy();
+          }
+        });
+      }
+    } catch (error) {
+      logger.error('Socket handler error:', error);
+      if (!socket.destroyed) {
+        socket.destroy();
+      }
+    }
+    
+    return; // Done - user has control now
+  }
+  
+  // ... rest of existing action handling
+}
+```
+
+---
+
+## Phase 3: Optional Context (If Needed)
+
+If users need more info, we can optionally pass a minimal context as a second parameter:
+
+```typescript
+export type TSocketHandler = (
+  socket: net.Socket, 
+  context?: {
+    route: IRouteConfig;
+    clientIp: string;
+    localPort: number;
+  }
+) => void | Promise<void>;
+```
+
+Usage:
+```typescript
+socketHandler: (socket, context) => {
+  console.log(`Connection from ${context.clientIp} to port ${context.localPort}`);
+  // Handle socket...
+}
+```
+
+---
+
+## Phase 4: Helper Utilities (Optional)
+
+### 4.1 Common Patterns
+**File:** `ts/proxies/smart-proxy/utils/route-helpers.ts`
+
+```typescript
+// Simple helper to create socket handler routes
+export function createSocketHandlerRoute(
+  domains: string | string[],
+  ports: TPortRange,
+  handler: TSocketHandler,
+  options?: { name?: string; priority?: number }
+): IRouteConfig {
+  return {
+    name: options?.name || 'socket-handler-route',
+    priority: options?.priority || 50,
+    match: { domains, ports },
+    action: {
+      type: 'socket-handler',
+      socketHandler: handler
+    }
+  };
+}
+
+// Pre-built handlers for common cases
+export const SocketHandlers = {
+  // Simple echo server
+  echo: (socket: net.Socket) => {
+    socket.on('data', data => socket.write(data));
+  },
+  
+  // TCP proxy
+  proxy: (targetHost: string, targetPort: number) => (socket: net.Socket) => {
+    const target = net.connect(targetPort, targetHost);
+    socket.pipe(target);
+    target.pipe(socket);
+    socket.on('close', () => target.destroy());
+    target.on('close', () => socket.destroy());
+  },
+  
+  // Line-based protocol
+  lineProtocol: (handler: (line: string, socket: net.Socket) => void) => (socket: net.Socket) => {
+    let buffer = '';
+    socket.on('data', (data) => {
+      buffer += data.toString();
+      const lines = buffer.split('\n');
+      buffer = lines.pop() || '';
+      lines.forEach(line => handler(line, socket));
+    });
+  }
+};
+```
+
+---
+
+## Usage Examples
+
+### Example 1: Custom Protocol
+```typescript
+{
+  name: 'custom-protocol',
+  match: { ports: 9000 },
+  action: {
+    type: 'socket-handler',
+    socketHandler: (socket) => {
+      socket.write('READY\n');
+      socket.on('data', (data) => {
+        const cmd = data.toString().trim();
+        if (cmd === 'PING') socket.write('PONG\n');
+        else if (cmd === 'QUIT') socket.end();
+        else socket.write('ERROR: Unknown command\n');
+      });
+    }
+  }
+}
+```
+
+### Example 2: Simple TCP Proxy
+```typescript
+{
+  name: 'tcp-proxy',
+  match: { ports: 8080, domains: 'proxy.example.com' },
+  action: {
+    type: 'socket-handler',
+    socketHandler: SocketHandlers.proxy('backend.local', 3000)
+  }
+}
+```
+
+### Example 3: WebSocket with Custom Auth
+```typescript
+{
+  name: 'custom-websocket',
+  match: { ports: [80, 443], path: '/ws' },
+  action: {
+    type: 'socket-handler',
+    socketHandler: async (socket) => {
+      // Read HTTP headers
+      const headers = await readHttpHeaders(socket);
+      
+      // Custom auth check
+      if (!headers.authorization || !validateToken(headers.authorization)) {
+        socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');
+        socket.end();
+        return;
+      }
+      
+      // Proceed with WebSocket upgrade
+      const ws = new WebSocket(socket, headers);
+      // ... handle WebSocket
+    }
+  }
+}
+```
+
+---
+
+## Benefits of This Approach
+
+1. **Dead Simple API**: Just pass a function that gets the socket
+2. **No New Classes**: No ForwardingHandler subclass needed
+3. **Minimal Changes**: Only touches type definitions and one handler method
+4. **Full Power**: Users have complete control over the socket
+5. **Backward Compatible**: No changes to existing functionality
+6. **Easy to Test**: Just test the socket handler functions directly
+
+---
+
+## Implementation Steps
+
+1. Add `'socket-handler'` to `TRouteActionType` (5 minutes)
+2. Add `socketHandler?: TSocketHandler` to `IRouteAction` (5 minutes)
+3. Add socket-handler case in `RouteConnectionHandler.handleConnection()` (15 minutes)
+4. Add helper functions (optional, 30 minutes)
+5. Write tests (2 hours)
+6. Update documentation (1 hour)
+
+**Total implementation time: ~4 hours** (vs 6 weeks for the complex version)
+
+---
+
+## What We're NOT Doing
+
+- ❌ Creating new ForwardingHandler classes
+- ❌ Complex context objects with utils
+- ❌ HTTP request handling for socket handlers
+- ❌ Complex protocol detection mechanisms
+- ❌ Middleware patterns
+- ❌ Lifecycle hooks
+
+Keep it simple. The user just wants to handle a socket.
+
+---
+
+## Success Criteria
+
+- ✅ Users can define a route with `type: 'socket-handler'`
+- ✅ Users can provide a function that receives the socket
+- ✅ The function is called when a connection matches the route
+- ✅ Error handling prevents crashes
+- ✅ No performance impact on existing routes
+- ✅ Clean, simple API that's easy to understand
+
+---
+
+## Implementation Notes (Completed)
+
+### What Was Implemented
+1. **Type Definitions** - Added 'socket-handler' to TRouteActionType and TSocketHandler type
+2. **Route Handler** - Added socket-handler case in RouteConnectionHandler switch statement  
+3. **Error Handling** - Both sync and async errors are caught and logged
+4. **Initial Data Handling** - Initial chunks are re-emitted to handler's listeners
+5. **Helper Functions** - Added createSocketHandlerRoute and pre-built handlers (echo, proxy, etc.)
+6. **Full Test Coverage** - All test cases pass including async handlers and error handling
+
+### Key Implementation Details
+- Socket handlers require initial data from client to trigger routing (not TLS handshake)
+- The handler receives the raw socket after route matching
+- Both sync and async handlers are supported  
+- Errors in handlers terminate the connection gracefully
+- Helper utilities provide common patterns (echo server, TCP proxy, line protocol)
+
+### Usage Notes
+- Clients must send initial data to trigger the handler (even just a newline)
+- The socket is passed directly to the handler function
+- Handler has complete control over the socket lifecycle
+- No special context object needed - keeps it simple
+
+**Total implementation time: ~3 hours**