@push.rocks/smartproxy 18.0.0 → 18.0.2

package/readme.plan.md

@@ -1,146 +1,654 @@
-# SmartProxy Interface Consolidation Plan
+# NFTables-SmartProxy Integration Plan
 
 ## Overview
 
-This document outlines a plan to consolidate duplicate and inconsistent interfaces in the SmartProxy codebase, specifically the `IRouteSecurity` interface which is defined twice with different properties. This inconsistency caused issues with security checks for port forwarding. The goal is to unify these interfaces, use consistent property naming, and improve code maintainability.
+This document outlines a comprehensive plan to integrate the existing NFTables functionality with the SmartProxy core to provide advanced network-level routing capabilities. The NFTables proxy already exists in the codebase but is not fully integrated with the SmartProxy routing system. This integration will allow SmartProxy to leverage the power of Linux's NFTables firewall system for high-performance port forwarding, load balancing, and security filtering.
 
-## Problem Description
+## Current State
 
-We currently have two separate `IRouteSecurity` interfaces defined in `ts/proxies/smart-proxy/models/route-types.ts`:
+1. **NFTablesProxy**: A standalone implementation exists in `ts/proxies/nftables-proxy/` with its own configuration and API.
+2. **SmartProxy**: The main routing system with route-based configuration.
+3. **No Integration**: Currently, these systems operate independently with no shared configuration or coordination.
 
-1. **First definition** (lines 116-122) - Used in IRouteAction:
+## Goals
+
+1. Create a unified configuration system where SmartProxy routes can specify NFTables-based forwarding.
+2. Allow SmartProxy to dynamically provision and manage NFTables rules based on route configuration.
+3. Support advanced filtering and security rules through NFTables for better performance.
+4. Ensure backward compatibility with existing setups.
+5. Provide metrics integration between the systems.
+
+## Implementation Plan
+
+### Phase 1: Route Configuration Schema Extension
+
+1. **Extend Route Configuration Schema**:
+   - Add new `forwardingEngine` option to IRouteAction to specify the forwarding implementation.
+   - Support values: 'node' (current NodeJS implementation) and 'nftables' (Linux NFTables).
+   - Add NFTables-specific configuration options to IRouteAction.
+
+2. **Update Type Definitions**:
    ```typescript
-   export interface IRouteSecurity {
-     allowedIps?: string[];
-     blockedIps?: string[];
-     maxConnections?: number;
-     authentication?: IRouteAuthentication;
+   // In route-types.ts
+   export interface IRouteAction {
+     type: 'forward' | 'redirect' | 'block';
+     target?: IRouteTarget;
+     security?: IRouteSecurity;
+     options?: IRouteOptions;
+     tls?: IRouteTlsOptions;
+     forwardingEngine?: 'node' | 'nftables';  // New field
+     nftables?: INfTablesOptions;             // New field
+   }
+
+   export interface INfTablesOptions {
+     preserveSourceIP?: boolean;
+     protocol?: 'tcp' | 'udp' | 'all';
+     maxRate?: string;                // QoS rate limiting
+     priority?: number;               // QoS priority
+     tableName?: string;              // Optional custom table name
+     useIPSets?: boolean;             // Use IP sets for performance
+     useAdvancedNAT?: boolean;        // Use connection tracking
    }
    ```
 
-2. **Second definition** (lines 253-272) - Used directly in IRouteConfig:
+### Phase 2: NFTablesManager Implementation
+
+1. **Create NFTablesManager Class**:
+   - Create a new class to manage NFTables rules based on SmartProxy routes.
+   - Add methods to create, update, and remove NFTables rules.
+   - Design a rule naming scheme to track which rules correspond to which routes.
+
+2. **Implementation**:
    ```typescript
-   export interface IRouteSecurity {
-     rateLimit?: IRouteRateLimit;
-     basicAuth?: {...};
-     jwtAuth?: {...};
-     ipAllowList?: string[];
-     ipBlockList?: string[];
+   // In ts/proxies/smart-proxy/nftables-manager.ts
+   export class NFTablesManager {
+     private rulesMap: Map<string, NfTablesProxy> = new Map();
+     
+     constructor(private options: ISmartProxyOptions) {}
+     
+     /**
+      * Provision NFTables rules for a route
+      */
+     public async provisionRoute(route: IRouteConfig): Promise<boolean> {
+       // Generate a unique ID for this route
+       const routeId = this.generateRouteId(route);
+       
+       // Skip if route doesn't use NFTables
+       if (route.action.forwardingEngine !== 'nftables') {
+         return true;
+       }
+       
+       // Create NFTables options from route configuration
+       const nftOptions = this.createNfTablesOptions(route);
+       
+       // Create and start an NFTablesProxy instance
+       const proxy = new NfTablesProxy(nftOptions);
+       
+       try {
+         await proxy.start();
+         this.rulesMap.set(routeId, proxy);
+         return true;
+       } catch (err) {
+         console.error(`Failed to provision NFTables rules for route ${route.name}: ${err.message}`);
+         return false;
+       }
+     }
+     
+     /**
+      * Remove NFTables rules for a route
+      */
+     public async deprovisionRoute(route: IRouteConfig): Promise<boolean> {
+       const routeId = this.generateRouteId(route);
+       
+       const proxy = this.rulesMap.get(routeId);
+       if (!proxy) {
+         return true; // Nothing to remove
+       }
+       
+       try {
+         await proxy.stop();
+         this.rulesMap.delete(routeId);
+         return true;
+       } catch (err) {
+         console.error(`Failed to deprovision NFTables rules for route ${route.name}: ${err.message}`);
+         return false;
+       }
+     }
+     
+     /**
+      * Update NFTables rules when route changes
+      */
+     public async updateRoute(oldRoute: IRouteConfig, newRoute: IRouteConfig): Promise<boolean> {
+       // Remove old rules and add new ones
+       await this.deprovisionRoute(oldRoute);
+       return this.provisionRoute(newRoute);
+     }
+     
+     /**
+      * Generate a unique ID for a route
+      */
+     private generateRouteId(route: IRouteConfig): string {
+       // Generate a unique ID based on route properties
+       return `${route.name || 'unnamed'}-${JSON.stringify(route.match)}-${Date.now()}`;
+     }
+     
+     /**
+      * Create NFTablesProxy options from a route configuration
+      */
+     private createNfTablesOptions(route: IRouteConfig): NfTableProxyOptions {
+       const { action } = route;
+       
+       // Ensure we have a target
+       if (!action.target) {
+         throw new Error('Route must have a target to use NFTables forwarding');
+       }
+       
+       // Convert port specifications
+       const fromPorts = this.expandPortRange(route.match.ports);
+       
+       // Determine target port
+       let toPorts;
+       if (action.target.port === 'preserve') {
+         // 'preserve' means use the same ports as the source
+         toPorts = fromPorts;
+       } else if (typeof action.target.port === 'function') {
+         // For function-based ports, we can't determine at setup time
+         // Use the "preserve" approach and let NFTables handle it
+         toPorts = fromPorts;
+       } else {
+         toPorts = action.target.port;
+       }
+       
+       // Create options
+       const options: NfTableProxyOptions = {
+         fromPort: fromPorts,
+         toPort: toPorts,
+         toHost: typeof action.target.host === 'function' 
+           ? 'localhost' // Can't determine at setup time, use localhost
+           : (Array.isArray(action.target.host) 
+              ? action.target.host[0] // Use first host for now
+              : action.target.host),
+         protocol: action.nftables?.protocol || 'tcp',
+         preserveSourceIP: action.nftables?.preserveSourceIP,
+         useIPSets: action.nftables?.useIPSets !== false,
+         useAdvancedNAT: action.nftables?.useAdvancedNAT,
+         enableLogging: this.options.enableDetailedLogging,
+         deleteOnExit: true,
+         tableName: action.nftables?.tableName || 'smartproxy'
+       };
+       
+       // Add security-related options
+       if (action.security?.ipAllowList?.length) {
+         options.allowedSourceIPs = action.security.ipAllowList;
+       }
+       
+       if (action.security?.ipBlockList?.length) {
+         options.bannedSourceIPs = action.security.ipBlockList;
+       }
+       
+       // Add QoS options
+       if (action.nftables?.maxRate || action.nftables?.priority) {
+         options.qos = {
+           enabled: true,
+           maxRate: action.nftables.maxRate,
+           priority: action.nftables.priority
+         };
+       }
+       
+       return options;
+     }
+     
+     /**
+      * Expand port range specifications
+      */
+     private expandPortRange(ports: TPortRange): number | PortRange | Array<number | PortRange> {
+       // Use RouteManager's expandPortRange to convert to actual port numbers
+       const routeManager = new RouteManager(this.options);
+       
+       // Process different port specifications
+       if (typeof ports === 'number') {
+         return ports;
+       } else if (Array.isArray(ports)) {
+         const result: Array<number | PortRange> = [];
+         
+         for (const item of ports) {
+           if (typeof item === 'number') {
+             result.push(item);
+           } else if ('from' in item && 'to' in item) {
+             result.push({ from: item.from, to: item.to });
+           }
+         }
+         
+         return result;
+       } else if ('from' in ports && 'to' in ports) {
+         return { from: ports.from, to: ports.to };
+       }
+       
+       // Fallback
+       return 80;
+     }
+     
+     /**
+      * Get status of all managed rules
+      */
+     public async getStatus(): Promise<Record<string, NfTablesStatus>> {
+       const result: Record<string, NfTablesStatus> = {};
+       
+       for (const [routeId, proxy] of this.rulesMap.entries()) {
+         result[routeId] = await proxy.getStatus();
+       }
+       
+       return result;
+     }
+     
+     /**
+      * Stop all NFTables rules
+      */
+     public async stop(): Promise<void> {
+       // Stop all NFTables proxies
+       const stopPromises = Array.from(this.rulesMap.values()).map(proxy => proxy.stop());
+       await Promise.all(stopPromises);
+       
+       this.rulesMap.clear();
+     }
    }
    ```
 
-This duplication with inconsistent naming (`allowedIps` vs `ipAllowList` and `blockedIps` vs `ipBlockList`) caused routing issues when IP security checks were used, as we had to implement a workaround to check both property names.
+### Phase 3: SmartProxy Integration
 
-## Implementation Plan
+1. **Extend SmartProxy Class**:
+   - Add NFTablesManager as a property of SmartProxy.
+   - Hook into route configuration to provision NFTables rules.
+   - Add methods to manage NFTables functionality.
 
-### Phase 1: Interface Consolidation
+2. **Implementation**:
+   ```typescript
+   // In ts/proxies/smart-proxy/smart-proxy.ts
+   import { NFTablesManager } from './nftables-manager.js';
 
-1. **Create a unified interface definition:**
-   - Create one comprehensive `IRouteSecurity` interface that includes all properties
-   - Use consistent property naming (standardize on `ipAllowList` and `ipBlockList`)
-   - Add proper documentation for each property
-   - Remove the duplicate interface definition
-   
-2. **Update references to use the unified interface:**
-   - Update all code that references the old interface properties
-   - Update all configurations to use the new property names
-   - Ensure implementation in `route-manager.ts` uses the correct property names
+   export class SmartProxy {
+     // Existing properties
+     private nftablesManager: NFTablesManager;
+     
+     constructor(options: ISmartProxyOptions) {
+       // Existing initialization
+       
+       // Initialize NFTablesManager
+       this.nftablesManager = new NFTablesManager(options);
+     }
+     
+     /**
+      * Start the SmartProxy server
+      */
+     public async start(): Promise<void> {
+       // Existing initialization
+       
+       // If we have routes, provision NFTables rules for them
+       for (const route of this.settings.routes) {
+         if (route.action.forwardingEngine === 'nftables') {
+           await this.nftablesManager.provisionRoute(route);
+         }
+       }
+       
+       // Rest of existing start method
+     }
+     
+     /**
+      * Stop the SmartProxy server
+      */
+     public async stop(): Promise<void> {
+       // Stop NFTablesManager first
+       await this.nftablesManager.stop();
+       
+       // Rest of existing stop method
+     }
+     
+     /**
+      * Update routes
+      */
+     public async updateRoutes(routes: IRouteConfig[]): Promise<void> {
+       // Get existing routes that use NFTables
+       const oldNfTablesRoutes = this.settings.routes.filter(
+         r => r.action.forwardingEngine === 'nftables'
+       );
+       
+       // Get new routes that use NFTables
+       const newNfTablesRoutes = routes.filter(
+         r => r.action.forwardingEngine === 'nftables'
+       );
+       
+       // Find routes to remove, update, or add
+       for (const oldRoute of oldNfTablesRoutes) {
+         const newRoute = newNfTablesRoutes.find(r => r.name === oldRoute.name);
+         
+         if (!newRoute) {
+           // Route was removed
+           await this.nftablesManager.deprovisionRoute(oldRoute);
+         } else {
+           // Route was updated
+           await this.nftablesManager.updateRoute(oldRoute, newRoute);
+         }
+       }
+       
+       // Find new routes to add
+       for (const newRoute of newNfTablesRoutes) {
+         const oldRoute = oldNfTablesRoutes.find(r => r.name === newRoute.name);
+         
+         if (!oldRoute) {
+           // New route
+           await this.nftablesManager.provisionRoute(newRoute);
+         }
+       }
+       
+       // Update settings with the new routes
+       this.settings.routes = routes;
+       
+       // Update route manager with new routes
+       this.routeManager.updateRoutes(routes);
+     }
+     
+     /**
+      * Get NFTables status
+      */
+     public async getNfTablesStatus(): Promise<Record<string, NfTablesStatus>> {
+       return this.nftablesManager.getStatus();
+     }
+   }
+   ```
 
-### Phase 2: Code and Documentation Updates
+### Phase 4: Routing System Integration
 
-1. **Update type usages and documentation:**
-   - Update all code that creates or uses security configurations
-   - Update documentation to reflect the new interface structure
-   - Add examples of the correct property usage
-   - Document the breaking change in changelog.md
+1. **Extend the Route-Connection-Handler**:
+   - Modify to check if a route uses NFTables.
+   - Skip Node.js-based connection handling for NFTables routes.
 
-2. **Add tests:**
-   - Update existing tests to use the new property names
-   - Add test cases for all security configuration scenarios
-   - Verify that port range configurations with security settings work correctly
+2. **Implementation**:
+   ```typescript
+   // In ts/proxies/smart-proxy/route-connection-handler.ts
+   export class RouteConnectionHandler {
+     // Existing methods
+     
+     /**
+      * Route the connection based on match criteria
+      */
+     private routeConnection(
+       socket: plugins.net.Socket, 
+       record: IConnectionRecord,
+       serverName: string,
+       initialChunk?: Buffer
+     ): void {
+       // Find matching route
+       const routeMatch = this.routeManager.findMatchingRoute({
+         port: record.localPort,
+         domain: serverName,
+         clientIp: record.remoteIP,
+         path: undefined,
+         tlsVersion: undefined
+       });
+       
+       if (!routeMatch) {
+         // Existing code for no matching route
+         return;
+       }
+       
+       const route = routeMatch.route;
+       
+       // Check if this route uses NFTables for forwarding
+       if (route.action.forwardingEngine === 'nftables') {
+         // For NFTables routes, we don't need to do anything at the application level
+         // The packet is forwarded at the kernel level
+         
+         // Log the connection
+         console.log(
+           `[${record.id}] Connection forwarded by NFTables: ${record.remoteIP} -> port ${record.localPort}`
+         );
+         
+         // Just close the socket in our application since it's handled at kernel level
+         socket.end();
+         this.connectionManager.initiateCleanupOnce(record, 'nftables_handled');
+         return;
+       }
+       
+       // Existing code for handling the route
+     }
+   }
+   ```
+
+### Phase 5: CLI and Configuration Helpers
 
-## Implementation Steps
+1. **Add Helper Functions**:
+   - Create helper functions for easy route creation with NFTables.
+   - Update the route-helpers.ts utility file.
+
+2. **Implementation**:
+   ```typescript
+   // In ts/proxies/smart-proxy/utils/route-helpers.ts
+   
+   /**
+    * Create an NFTables-based route
+    */
+   export function createNfTablesRoute(
+     nameOrDomains: string | string[],
+     target: { host: string; port: number | 'preserve' },
+     options: {
+       ports?: TPortRange;
+       protocol?: 'tcp' | 'udp' | 'all';
+       preserveSourceIP?: boolean;
+       allowedIps?: string[];
+       maxRate?: string;
+       priority?: number;
+       useTls?: boolean;
+     } = {}
+   ): IRouteConfig {
+     // Determine if this is a name or domain
+     let name: string;
+     let domains: string | string[];
+     
+     if (Array.isArray(nameOrDomains) || nameOrDomains.includes('.')) {
+       domains = nameOrDomains;
+       name = Array.isArray(nameOrDomains) ? nameOrDomains[0] : nameOrDomains;
+     } else {
+       name = nameOrDomains;
+       domains = []; // No domains
+     }
+     
+     const route: IRouteConfig = {
+       name,
+       match: {
+         domains,
+         ports: options.ports || 80
+       },
+       action: {
+         type: 'forward',
+         target: {
+           host: target.host,
+           port: target.port
+         },
+         forwardingEngine: 'nftables',
+         nftables: {
+           protocol: options.protocol || 'tcp',
+           preserveSourceIP: options.preserveSourceIP,
+           maxRate: options.maxRate,
+           priority: options.priority
+         }
+       }
+     };
+     
+     // Add security if allowed IPs are specified
+     if (options.allowedIps?.length) {
+       route.action.security = {
+         ipAllowList: options.allowedIps
+       };
+     }
+     
+     // Add TLS options if needed
+     if (options.useTls) {
+       route.action.tls = {
+         mode: 'passthrough'
+       };
+     }
+     
+     return route;
+   }
+   
+   /**
+    * Create an NFTables-based TLS termination route
+    */
+   export function createNfTablesTerminateRoute(
+     nameOrDomains: string | string[],
+     target: { host: string; port: number | 'preserve' },
+     options: {
+       ports?: TPortRange;
+       protocol?: 'tcp' | 'udp' | 'all';
+       preserveSourceIP?: boolean;
+       allowedIps?: string[];
+       maxRate?: string;
+       priority?: number;
+       certificate?: string | { cert: string; key: string };
+     } = {}
+   ): IRouteConfig {
+     const route = createNfTablesRoute(
+       nameOrDomains,
+       target,
+       {
+         ...options,
+         ports: options.ports || 443,
+         useTls: false
+       }
+     );
+     
+     // Set TLS termination
+     route.action.tls = {
+       mode: 'terminate',
+       certificate: options.certificate || 'auto'
+     };
+     
+     return route;
+   }
+   ```
+
+### Phase 6: Documentation and Testing
+
+1. **Update Documentation**:
+   - Add NFTables integration documentation to README and API docs.
+   - Document the implementation and use cases.
+
+2. **Test Cases**:
+   - Create test cases for NFTables-based routing.
+   - Test performance comparison with Node.js-based forwarding.
+   - Test security features with IP allowlists/blocklists.
 
 ```typescript
-// Step 1: Define the unified interface
-export interface IRouteSecurity {
-  // Access control lists
-  ipAllowList?: string[];        // IP addresses that are allowed to connect
-  ipBlockList?: string[];        // IP addresses that are blocked from connecting
-  
-  // Connection limits
-  maxConnections?: number;       // Maximum concurrent connections
-  
-  // Authentication
-  authentication?: IRouteAuthentication;
+// In test/test.nftables-integration.ts
+import { SmartProxy } from '../ts/proxies/smart-proxy/index.js';
+import { createNfTablesRoute } from '../ts/proxies/smart-proxy/utils/route-helpers.js';
+import { expect, tap } from '@push.rocks/tapbundle';
+import * as net from 'net';
+
+// Test server and client utilities
+let testServer: net.Server;
+let smartProxy: SmartProxy;
+
+const TEST_PORT = 4000;
+const PROXY_PORT = 5000;
+const TEST_DATA = 'Hello through NFTables!';
+
+tap.test('setup NFTables integration test environment', async () => {
+  // Create a test TCP server
+  testServer = net.createServer((socket) => {
+    socket.on('data', (data) => {
+      socket.write(`Server says: ${data.toString()}`);
+    });
+  });
   
-  // Rate limiting
-  rateLimit?: IRouteRateLimit;
+  await new Promise<void>((resolve) => {
+    testServer.listen(TEST_PORT, () => {
+      console.log(`Test server listening on port ${TEST_PORT}`);
+      resolve();
+    });
+  });
   
-  // Authentication methods
-  basicAuth?: {
-    enabled: boolean;
-    users: Array<{ username: string; password: string }>;
-    realm?: string;
-    excludePaths?: string[];
-  };
+  // Create SmartProxy with NFTables route
+  smartProxy = new SmartProxy({
+    routes: [
+      createNfTablesRoute('test-nftables', {
+        host: 'localhost',
+        port: TEST_PORT
+      }, {
+        ports: PROXY_PORT,
+        protocol: 'tcp'
+      })
+    ]
+  });
   
-  jwtAuth?: {
-    enabled: boolean;
-    secret: string;
-    algorithm?: string;
-    issuer?: string;
-    audience?: string;
-    expiresIn?: number;
-    excludePaths?: string[];
-  };
-}
-```
-
-Update `isClientIpAllowed` method to use only the new property names:
+  // Start the proxy
+  await smartProxy.start();
+});
 
-```typescript
-private isClientIpAllowed(route: IRouteConfig, clientIp: string): boolean {
-  const security = route.action.security;
-  
-  if (!security) {
-    return true; // No security settings means allowed
-  }
+tap.test('should forward TCP connections through NFTables', async () => {
+  // Connect to the proxy port
+  const client = new net.Socket();
   
-  // Check blocked IPs first
-  if (security.ipBlockList && security.ipBlockList.length > 0) {
-    for (const pattern of security.ipBlockList) {
-      if (this.matchIpPattern(pattern, clientIp)) {
-        return false; // IP is blocked
-      }
-    }
-  }
+  const response = await new Promise<string>((resolve, reject) => {
+    let responseData = '';
+    
+    client.connect(PROXY_PORT, 'localhost', () => {
+      client.write(TEST_DATA);
+    });
+    
+    client.on('data', (data) => {
+      responseData += data.toString();
+      client.end();
+    });
+    
+    client.on('end', () => {
+      resolve(responseData);
+    });
+    
+    client.on('error', (err) => {
+      reject(err);
+    });
+  });
   
-  // If there are allowed IPs, check them
-  if (security.ipAllowList && security.ipAllowList.length > 0) {
-    for (const pattern of security.ipAllowList) {
-      if (this.matchIpPattern(pattern, clientIp)) {
-        return true; // IP is allowed
-      }
-    }
-    return false; // IP not in allowed list
-  }
+  expect(response).toEqual(`Server says: ${TEST_DATA}`);
+});
+
+tap.test('cleanup NFTables integration test environment', async () => {
+  // Stop the proxy and test server
+  await smartProxy.stop();
   
-  // No allowed IPs specified, so IP is allowed
-  return true;
-}
+  await new Promise<void>((resolve) => {
+    testServer.close(() => {
+      resolve();
+    });
+  });
+});
+
+export default tap.start();
 ```
 
 ## Expected Benefits
 
-- **Improved Consistency**: Single, unified interface with consistent property naming
-- **Better Type Safety**: Eliminating confusing duplicate interface definitions
-- **Reduced Errors**: Prevent misunderstandings about which property names to use
-- **Forward Compatibility**: Clearer path for future security enhancements
-- **Better Developer Experience**: Simplified interface with comprehensive documentation
+1. **Performance**: NFTables operates at the kernel level, offering much higher performance than Node.js-based routing.
+2. **Scalability**: Handle more connections with less CPU and memory usage.
+3. **Security**: Leverage kernel-level security features for better protection.
+4. **Integration**: Unified configuration model between application and network layers.
+5. **Advanced Features**: Support for QoS, rate limiting, and other advanced networking features.
+
+## Implementation Notes
+
+- This integration requires root/sudo access to configure NFTables rules.
+- Consider adding a capability check to gracefully fall back to Node.js routing if NFTables is not available.
+- The NFTables integration should be optional and SmartProxy should continue to work without it.
+- The integration provides a path for future extensions to other kernel-level networking features.
+
+## Timeline
 
-## Testing Plan
+- Phase 1 (Route Configuration Schema): 1-2 days
+- Phase 2 (NFTablesManager): 2-3 days
+- Phase 3 (SmartProxy Integration): 1-2 days
+- Phase 4 (Routing System Integration): 1 day
+- Phase 5 (CLI and Helpers): 1 day
+- Phase 6 (Documentation and Testing): 2 days
 
-1. Test with existing configurations using both old and new property names
-2. Create specific test cases for port ranges with different security configurations
-3. Verify that port forwarding with IP allow lists works correctly with the unified interface
+**Total Estimated Time: 8-11 days**