@push.rocks/smartproxy 19.5.18 → 19.5.20

package/readme.problems.md

@@ -1,170 +0,0 @@
-# SmartProxy Performance Issues Report
-
-## Executive Summary
-This report identifies performance issues and blocking operations in the SmartProxy codebase that could impact scalability and responsiveness under high load.
-
-## Critical Issues
-
-### 1. **Synchronous Filesystem Operations**
-These operations block the event loop and should be replaced with async alternatives:
-
-#### Certificate Management
-- `ts/proxies/http-proxy/certificate-manager.ts:29`: `fs.existsSync()` 
-- `ts/proxies/http-proxy/certificate-manager.ts:30`: `fs.mkdirSync()`
-- `ts/proxies/http-proxy/certificate-manager.ts:49-50`: `fs.readFileSync()` for loading certificates
-
-#### NFTables Proxy
-- `ts/proxies/nftables-proxy/nftables-proxy.ts`: Multiple uses of `execSync()` for system commands
-- `ts/proxies/nftables-proxy/nftables-proxy.ts`: Multiple `fs.writeFileSync()` and `fs.unlinkSync()` operations
-
-#### Certificate Store
-- `ts/proxies/smart-proxy/cert-store.ts:8`: `ensureDirSync()`
-- `ts/proxies/smart-proxy/cert-store.ts:15,31,76`: `fileExistsSync()`
-- `ts/proxies/smart-proxy/cert-store.ts:77`: `removeManySync()`
-
-### 2. **Event Loop Blocking Operations**
-
-#### Busy Wait Loop
-- `ts/proxies/nftables-proxy/nftables-proxy.ts:235-238`: 
-  ```typescript
-  const waitUntil = Date.now() + retryDelayMs;
-  while (Date.now() < waitUntil) {
-    // busy wait - blocks event loop completely
-  }
-  ```
-  This is extremely problematic as it blocks the entire Node.js event loop.
-
-### 3. **Potential Memory Leaks**
-
-#### Timer Management Issues
-Several timers are created without proper cleanup:
-- `ts/proxies/http-proxy/function-cache.ts`: `setInterval()` without storing reference for cleanup
-- `ts/proxies/http-proxy/request-handler.ts`: `setInterval()` for rate limit cleanup without cleanup
-- `ts/core/utils/shared-security-manager.ts`: `cleanupInterval` stored but no cleanup method
-
-#### Event Listener Accumulation
-- Multiple instances of event listeners being added without corresponding cleanup
-- Connection handlers add listeners without always removing them on connection close
-
-### 4. **Connection Pool Management**
-
-#### ConnectionPool (ts/proxies/http-proxy/connection-pool.ts)
-**Good practices observed:**
-- Proper connection lifecycle management
-- Periodic cleanup of idle connections
-- Connection limits enforcement
-
-**Potential issues:**
-- No backpressure mechanism when pool is full
-- Synchronous sorting operation in `cleanupConnectionPool()` could be slow with many connections
-
-### 5. **Resource Management Issues**
-
-#### Socket Cleanup
-- Some error paths don't properly clean up sockets
-- Missing `removeAllListeners()` in some error scenarios could lead to memory leaks
-
-#### Timeout Management
-- Inconsistent timeout handling across different components
-- Some sockets created without timeout settings
-
-### 6. **JSON Operations on Large Objects**
-- `ts/proxies/smart-proxy/cert-store.ts:21`: `JSON.parse()` on certificate metadata
-- `ts/proxies/smart-proxy/cert-store.ts:71`: `JSON.stringify()` with pretty printing
-- `ts/proxies/http-proxy/function-cache.ts:76`: `JSON.stringify()` for cache keys (called frequently)
-
-## Recommendations
-
-### Immediate Actions (High Priority)
-
-1. **Replace Synchronous Operations**
-   ```typescript
-   // Instead of:
-   if (fs.existsSync(path)) { ... }
-   
-   // Use:
-   try {
-     await fs.promises.access(path);
-     // file exists
-   } catch {
-     // file doesn't exist
-   }
-   ```
-
-2. **Fix Busy Wait Loop**
-   ```typescript
-   // Instead of:
-   while (Date.now() < waitUntil) { }
-   
-   // Use:
-   await new Promise(resolve => setTimeout(resolve, retryDelayMs));
-   ```
-
-3. **Add Timer Cleanup**
-   ```typescript
-   class Component {
-     private cleanupTimer?: NodeJS.Timeout;
-     
-     start() {
-       this.cleanupTimer = setInterval(() => { ... }, 60000);
-     }
-     
-     stop() {
-       if (this.cleanupTimer) {
-         clearInterval(this.cleanupTimer);
-         this.cleanupTimer = undefined;
-       }
-     }
-   }
-   ```
-
-### Medium Priority
-
-1. **Optimize JSON Operations**
-   - Cache JSON.stringify results for frequently used objects
-   - Consider using faster hashing for cache keys (e.g., crypto.createHash)
-   - Use streaming JSON parsers for large objects
-
-2. **Improve Connection Pool**
-   - Implement backpressure/queueing when pool is full
-   - Use a heap or priority queue for connection management instead of sorting
-
-3. **Standardize Resource Cleanup**
-   - Create a base class for components with lifecycle management
-   - Ensure all event listeners are removed on cleanup
-   - Add abort controllers for better cancellation support
-
-### Long-term Improvements
-
-1. **Worker Threads**
-   - Move CPU-intensive operations to worker threads
-   - Consider using worker pools for NFTables operations
-
-2. **Monitoring and Metrics**
-   - Add performance monitoring for event loop lag
-   - Track connection pool utilization
-   - Monitor memory usage patterns
-
-3. **Graceful Degradation**
-   - Implement circuit breakers for backend connections
-   - Add request queuing with overflow protection
-   - Implement adaptive timeout strategies
-
-## Impact Assessment
-
-These issues primarily affect:
-- **Scalability**: Blocking operations limit concurrent connection handling
-- **Responsiveness**: Event loop blocking causes latency spikes
-- **Stability**: Memory leaks could cause crashes under sustained load
-- **Resource Usage**: Inefficient resource management increases memory/CPU usage
-
-## Testing Recommendations
-
-1. Load test with high connection counts (10k+ concurrent)
-2. Monitor event loop lag under stress
-3. Test long-running scenarios to detect memory leaks
-4. Benchmark with async vs sync operations to measure improvement
-
-## Conclusion
-
-While SmartProxy has good architectural design and many best practices, the identified blocking operations and resource management issues could significantly impact performance under high load. The most critical issues (busy wait loop and synchronous filesystem operations) should be addressed immediately.

package/ts/core/utils/route-utils.ts

@@ -1,312 +0,0 @@
-/**
- * Route matching utilities for SmartProxy components
- * 
- * Contains shared logic for domain matching, path matching, and IP matching
- * to be used by different proxy components throughout the system.
- */
-
-/**
- * Match a domain pattern against a domain
- * 
- * @param pattern Domain pattern with optional wildcards (e.g., "*.example.com")
- * @param domain Domain to match against the pattern
- * @returns Whether the domain matches the pattern
- */
-export function matchDomain(pattern: string, domain: string): boolean {
-  // Handle exact match (case-insensitive)
-  if (pattern.toLowerCase() === domain.toLowerCase()) {
-    return true;
-  }
-
-  // Handle wildcard pattern
-  if (pattern.includes('*')) {
-    const regexPattern = pattern
-      .replace(/\./g, '\\.')  // Escape dots
-      .replace(/\*/g, '.*');  // Convert * to .*
-
-    const regex = new RegExp(`^${regexPattern}$`, 'i');
-    return regex.test(domain);
-  }
-
-  return false;
-}
-
-/**
- * Match domains from a route against a given domain
- * 
- * @param domains Array or single domain pattern to match against
- * @param domain Domain to match
- * @returns Whether the domain matches any of the patterns
- */
-export function matchRouteDomain(domains: string | string[] | undefined, domain: string | undefined): boolean {
-  // If no domains specified in the route, match all domains
-  if (!domains) {
-    return true;
-  }
-
-  // If no domain in the request, can't match domain-specific routes
-  if (!domain) {
-    return false;
-  }
-  
-  const patterns = Array.isArray(domains) ? domains : [domains];
-  return patterns.some(pattern => matchDomain(pattern, domain));
-}
-
-/**
- * Match a path pattern against a path
- * 
- * @param pattern Path pattern with optional wildcards
- * @param path Path to match against the pattern
- * @returns Whether the path matches the pattern
- */
-export function matchPath(pattern: string, path: string): boolean {
-  // Handle exact match
-  if (pattern === path) {
-    return true;
-  }
-
-  // Handle simple wildcard at the end (like /api/*)
-  if (pattern.endsWith('*')) {
-    const prefix = pattern.slice(0, -1);
-    return path.startsWith(prefix);
-  }
-
-  // Handle more complex wildcard patterns
-  if (pattern.includes('*')) {
-    const regexPattern = pattern
-      .replace(/\./g, '\\.')   // Escape dots
-      .replace(/\*/g, '.*')    // Convert * to .*
-      .replace(/\//g, '\\/');  // Escape slashes
-    
-    const regex = new RegExp(`^${regexPattern}$`);
-    return regex.test(path);
-  }
-
-  return false;
-}
-
-/**
- * Parse CIDR notation into subnet and mask bits
- * 
- * @param cidr CIDR string (e.g., "192.168.1.0/24")
- * @returns Object with subnet and bits, or null if invalid
- */
-export function parseCidr(cidr: string): { subnet: string; bits: number } | null {
-  try {
-    const [subnet, bitsStr] = cidr.split('/');
-    const bits = parseInt(bitsStr, 10);
-    
-    if (isNaN(bits) || bits < 0 || bits > 32) {
-      return null;
-    }
-    
-    return { subnet, bits };
-  } catch (e) {
-    return null;
-  }
-}
-
-/**
- * Convert an IP address to a numeric value
- * 
- * @param ip IPv4 address string (e.g., "192.168.1.1")
- * @returns Numeric representation of the IP
- */
-export function ipToNumber(ip: string): number {
-  // Handle IPv6-mapped IPv4 addresses (::ffff:192.168.1.1)
-  if (ip.startsWith('::ffff:')) {
-    ip = ip.slice(7);
-  }
-  
-  const parts = ip.split('.').map(part => parseInt(part, 10));
-  return (parts[0] << 24) | (parts[1] << 16) | (parts[2] << 8) | parts[3];
-}
-
-/**
- * Match an IP against a CIDR pattern
- * 
- * @param cidr CIDR pattern (e.g., "192.168.1.0/24")
- * @param ip IP to match against the pattern
- * @returns Whether the IP is in the CIDR range
- */
-export function matchIpCidr(cidr: string, ip: string): boolean {
-  const parsed = parseCidr(cidr);
-  if (!parsed) {
-    return false;
-  }
-  
-  try {
-    const { subnet, bits } = parsed;
-    
-    // Normalize IPv6-mapped IPv4 addresses
-    const normalizedIp = ip.startsWith('::ffff:') ? ip.substring(7) : ip;
-    const normalizedSubnet = subnet.startsWith('::ffff:') ? subnet.substring(7) : subnet;
-    
-    // Convert IP addresses to numeric values
-    const ipNum = ipToNumber(normalizedIp);
-    const subnetNum = ipToNumber(normalizedSubnet);
-    
-    // Calculate subnet mask
-    const maskNum = ~(2 ** (32 - bits) - 1);
-    
-    // Check if IP is in subnet
-    return (ipNum & maskNum) === (subnetNum & maskNum);
-  } catch (e) {
-    return false;
-  }
-}
-
-/**
- * Match an IP pattern against an IP
- * 
- * @param pattern IP pattern (exact, CIDR, or with wildcards)
- * @param ip IP to match against the pattern
- * @returns Whether the IP matches the pattern
- */
-export function matchIpPattern(pattern: string, ip: string): boolean {
-  // Normalize IPv6-mapped IPv4 addresses
-  const normalizedIp = ip.startsWith('::ffff:') ? ip.substring(7) : ip;
-  const normalizedPattern = pattern.startsWith('::ffff:') ? pattern.substring(7) : pattern;
-  
-  // Handle exact match with all variations
-  if (pattern === ip || normalizedPattern === normalizedIp || 
-      pattern === normalizedIp || normalizedPattern === ip) {
-    return true;
-  }
-  
-  // Handle "all" wildcard
-  if (pattern === '*' || normalizedPattern === '*') {
-    return true;
-  }
-  
-  // Handle CIDR notation (e.g., 192.168.1.0/24)
-  if (pattern.includes('/')) {
-    return matchIpCidr(pattern, normalizedIp) || 
-           (normalizedPattern !== pattern && matchIpCidr(normalizedPattern, normalizedIp));
-  }
-  
-  // Handle glob pattern (e.g., 192.168.1.*)
-  if (pattern.includes('*')) {
-    const regexPattern = pattern.replace(/\./g, '\\.').replace(/\*/g, '.*');
-    const regex = new RegExp(`^${regexPattern}$`);
-    if (regex.test(ip) || regex.test(normalizedIp)) {
-      return true;
-    }
-    
-    // If pattern was normalized, also test with normalized pattern
-    if (normalizedPattern !== pattern) {
-      const normalizedRegexPattern = normalizedPattern.replace(/\./g, '\\.').replace(/\*/g, '.*');
-      const normalizedRegex = new RegExp(`^${normalizedRegexPattern}$`);
-      return normalizedRegex.test(ip) || normalizedRegex.test(normalizedIp);
-    }
-  }
-  
-  return false;
-}
-
-/**
- * Match an IP against allowed and blocked IP patterns
- * 
- * @param ip IP to check
- * @param ipAllowList Array of allowed IP patterns
- * @param ipBlockList Array of blocked IP patterns
- * @returns Whether the IP is allowed
- */
-export function isIpAuthorized(
-  ip: string, 
-  ipAllowList: string[] = ['*'], 
-  ipBlockList: string[] = []
-): boolean {
-  // Check blocked IPs first
-  if (ipBlockList.length > 0) {
-    for (const pattern of ipBlockList) {
-      if (matchIpPattern(pattern, ip)) {
-        return false; // IP is blocked
-      }
-    }
-  }
-  
-  // If there are allowed IPs, check them
-  if (ipAllowList.length > 0) {
-    // Special case: if '*' is in allowed IPs, all non-blocked IPs are allowed
-    if (ipAllowList.includes('*')) {
-      return true;
-    }
-    
-    for (const pattern of ipAllowList) {
-      if (matchIpPattern(pattern, ip)) {
-        return true; // IP is allowed
-      }
-    }
-    return false; // IP not in allowed list
-  }
-  
-  // No allowed IPs specified, so IP is allowed by default
-  return true;
-}
-
-/**
- * Match an HTTP header pattern against a header value
- * 
- * @param pattern Expected header value (string or RegExp)
- * @param value Actual header value
- * @returns Whether the header matches the pattern
- */
-export function matchHeader(pattern: string | RegExp, value: string): boolean {
-  if (typeof pattern === 'string') {
-    return pattern === value;
-  } else if (pattern instanceof RegExp) {
-    return pattern.test(value);
-  }
-  return false;
-}
-
-/**
- * Calculate route specificity score
- * Higher score means more specific matching criteria
- * 
- * @param match Match criteria to evaluate
- * @returns Numeric specificity score
- */
-export function calculateRouteSpecificity(match: {
-  domains?: string | string[];
-  path?: string;
-  clientIp?: string[];
-  tlsVersion?: string[];
-  headers?: Record<string, string | RegExp>;
-}): number {
-  let score = 0;
-  
-  // Path is very specific
-  if (match.path) {
-    // More specific if it doesn't use wildcards
-    score += match.path.includes('*') ? 3 : 4;
-  }
-  
-  // Domain is next most specific
-  if (match.domains) {
-    const domains = Array.isArray(match.domains) ? match.domains : [match.domains];
-    // More domains or more specific domains (without wildcards) increase specificity
-    score += domains.length;
-    // Add bonus for exact domains (without wildcards)
-    score += domains.some(d => !d.includes('*')) ? 1 : 0;
-  }
-  
-  // Headers are quite specific
-  if (match.headers) {
-    score += Object.keys(match.headers).length * 2;
-  }
-  
-  // Client IP adds some specificity
-  if (match.clientIp && match.clientIp.length > 0) {
-    score += 1;
-  }
-  
-  // TLS version adds minimal specificity
-  if (match.tlsVersion && match.tlsVersion.length > 0) {
-    score += 1;
-  }
-  
-  return score;
-}