@push.rocks/smartproxy 27.21.0 → 28.0.0

package/readme.hints.md

@@ -1,561 +0,0 @@
-# SmartProxy Development Hints
-
-## Byte Tracking and Metrics
-
-### Throughput Drift Issue (Fixed)
-
-**Problem**: Throughput numbers were gradually increasing over time for long-lived connections.
-
-**Root Cause**: The `byRoute()` and `byIP()` methods were dividing cumulative total bytes (since connection start) by the window duration, causing rates to appear higher as connections aged:
-- Hour 1: 1GB total / 60s = 17 MB/s ✓
-- Hour 2: 2GB total / 60s = 34 MB/s ✗ (appears doubled!)
-- Hour 3: 3GB total / 60s = 50 MB/s ✗ (keeps rising!)
-
-**Solution**: Implemented dedicated ThroughputTracker instances for each route and IP address:
-- Each route and IP gets its own throughput tracker with per-second sampling
-- Samples are taken every second and stored in a circular buffer
-- Rate calculations use actual samples within the requested window
-- Default window is now 1 second for real-time accuracy
-
-### What Gets Counted (Network Interface Throughput)
-
-The byte tracking is designed to match network interface throughput (what Unifi/network monitoring tools show):
-
-**Counted bytes include:**
-- All application data
-- TLS handshakes and protocol overhead
-- TLS record headers and encryption padding
-- HTTP headers and protocol data
-- WebSocket frames and protocol overhead
-- TLS alerts sent to clients
-
-**NOT counted:**
-- PROXY protocol headers (sent to backend, not client)
-- TCP/IP headers (handled by OS, not visible at application layer)
-
-**Byte direction:**
-- `bytesReceived`: All bytes received FROM the client on the incoming connection
-- `bytesSent`: All bytes sent TO the client on the incoming connection
-- Backend connections are separate and not mixed with client metrics
-
-### Double Counting Issue (Fixed)
-
-**Problem**: Initial data chunks were being counted twice in the byte tracking:
-1. Once when stored in `pendingData` in `setupDirectConnection()`
-2. Again when the data flowed through bidirectional forwarding
-
-**Solution**: Removed the byte counting when storing initial chunks. Bytes are now only counted when they actually flow through the `setupBidirectionalForwarding()` callbacks.
-
-### HttpProxy Metrics (Fixed)
-
-**Problem**: HttpProxy forwarding was updating connection record byte counts but not calling `metricsCollector.recordBytes()`, resulting in missing throughput data.
-
-**Solution**: Added `metricsCollector.recordBytes()` calls to the HttpProxy bidirectional forwarding callbacks.
-
-### Metrics Architecture
-
-The metrics system has multiple layers:
-1. **Connection Records** (`record.bytesReceived/bytesSent`): Track total bytes per connection
-2. **Global ThroughputTracker**: Accumulates bytes between samples for overall rate calculations
-3. **Per-Route ThroughputTrackers**: Dedicated tracker for each route with per-second sampling
-4. **Per-IP ThroughputTrackers**: Dedicated tracker for each IP with per-second sampling
-5. **connectionByteTrackers**: Track cumulative bytes and metadata for active connections
-
-Key features:
-- All throughput trackers sample every second (1Hz)
-- Each tracker maintains a circular buffer of samples (default: 1 hour retention)
-- Rate calculations are accurate for any requested window (default: 1 second)
-- All byte counting happens exactly once at the data flow point
-- Unused route/IP trackers are automatically cleaned up when connections close
-
-### Understanding "High" Byte Counts
-
-If byte counts seem high compared to actual application data, remember:
-- TLS handshakes can be 1-5KB depending on cipher suites and certificates
-- Each TLS record has 5 bytes of header overhead
-- TLS encryption adds 16-48 bytes of padding/MAC per record
-- HTTP/2 has additional framing overhead
-- WebSocket has frame headers (2-14 bytes per message)
-
-This overhead is real network traffic and should be counted for accurate throughput metrics.
-
-### Byte Counting Paths
-
-There are two mutually exclusive paths for connections:
-
-1. **Direct forwarding** (route-connection-handler.ts):
-   - Used for TCP passthrough, TLS passthrough, and direct connections
-   - Bytes counted in `setupBidirectionalForwarding` callbacks
-   - Initial chunk NOT counted separately (flows through bidirectional forwarding)
-
-2. **HttpProxy forwarding** (http-proxy-bridge.ts):
-   - Used for TLS termination (terminate, terminate-and-reencrypt)
-   - Initial chunk counted when written to proxy
-   - All subsequent bytes counted in `setupBidirectionalForwarding` callbacks
-   - This is the ONLY counting point for these connections
-
-### Byte Counting Audit (2025-01-06)
-
-A comprehensive audit was performed to verify byte counting accuracy:
-
-**Audit Results:**
-- ✅ No double counting detected in any connection flow
-- ✅ Each byte counted exactly once in each direction
-- ✅ Connection records and metrics updated consistently
-- ✅ PROXY protocol headers correctly excluded from client metrics
-- ✅ NFTables forwarded connections correctly not counted (kernel handles)
-
-**Key Implementation Points:**
-- All byte counting happens in only 2 files: `route-connection-handler.ts` and `http-proxy-bridge.ts`
-- Both use the same pattern: increment `record.bytesReceived/Sent` AND call `metricsCollector.recordBytes()`
-- Initial chunks handled correctly: stored but not counted until forwarded
-- TLS alerts counted as sent bytes (correct - they are sent to client)
-
-For full audit details, see `readme.byte-counting-audit.md`
-
-## Connection Cleanup
-
-### Zombie Connection Detection
-
-The connection manager performs comprehensive zombie detection every 10 seconds:
-- **Full zombies**: Both incoming and outgoing sockets destroyed but connection not cleaned up
-- **Half zombies**: One socket destroyed, grace period expired (5 minutes for TLS, 30 seconds for non-TLS)
-- **Stuck connections**: Data received but none sent back after threshold (5 minutes for TLS, 60 seconds for non-TLS)
-
-### Cleanup Queue
-
-Connections are cleaned up through a batched queue system:
-- Batch size: 100 connections
-- Processing triggered immediately when batch size reached
-- Otherwise processed after 100ms delay
-- Prevents overwhelming the system during mass disconnections
-
-## Keep-Alive Handling
-
-Keep-alive connections receive special treatment based on `keepAliveTreatment` setting:
-- **standard**: Normal timeout applies
-- **extended**: Timeout multiplied by `keepAliveInactivityMultiplier` (default 6x)
-- **immortal**: No timeout, connections persist indefinitely
-
-## PROXY Protocol
-
-The system supports both receiving and sending PROXY protocol:
-- **Receiving**: Enabled explicitly per listener through `match.inboundProxyProtocol`; the global `trustedProxyIPs` list only supplies trust entries for explicit `optional` or `required` policies
-- **Sending**: Enabled per-target, per-route, or globally via `sendProxyProtocol`
-- Real client IP is preserved and used for all connection tracking and security checks
-
-## Metrics and Throughput Calculation
-
-The metrics system tracks throughput using per-second sampling:
-
-1. **Byte Recording**: Bytes are recorded as data flows through connections
-2. **Sampling**: Every second, accumulated bytes are stored as a sample
-3. **Rate Calculation**: Throughput is calculated by summing bytes over a time window
-4. **Per-Route/IP Tracking**: Separate ThroughputTracker instances for each route and IP
-
-Key implementation details:
-- Bytes are recorded in the bidirectional forwarding callbacks
-- The instant() method returns throughput over the last 1 second
-- The recent() method returns throughput over the last 10 seconds
-- Custom windows can be specified for different averaging periods
-
-### Throughput Spikes Issue
-
-There's a fundamental difference between application-layer and network-layer throughput:
-
-**Application Layer (what we measure)**:
-- Bytes are recorded when delivered to/from the application
-- Large chunks can arrive "instantly" due to kernel/Node.js buffering
-- Shows spikes when buffers are flushed (e.g., 20MB in 1 second = 160 Mbit/s)
-
-**Network Layer (what Unifi shows)**:
-- Actual packet flow through the network interface
-- Limited by physical network speed (e.g., 20 Mbit/s)
-- Data transfers over time, not in bursts
-
-The spikes occur because:
-1. Data flows over network at 20 Mbit/s (takes 8 seconds for 20MB)
-2. Kernel/Node.js buffers this incoming data
-3. When buffer is flushed, application receives large chunk at once
-4. We record entire chunk in current second, creating artificial spike
-
-**Potential Solutions**:
-1. Use longer window for "instant" measurements (e.g., 5 seconds instead of 1)
-2. Track socket write backpressure to estimate actual network flow
-3. Implement bandwidth estimation based on connection duration
-4. Accept that application-layer != network-layer throughput
-
-## Connection Limiting
-
-### Per-IP Connection Limits
-- SmartProxy tracks connections per IP address in the SecurityManager
-- Default limit is 100 connections per IP (configurable via `maxConnectionsPerIP`)
-- Connection rate limiting is also enforced (default 300 connections/minute per IP)
-- HttpProxy has been enhanced to also enforce per-IP limits when forwarding from SmartProxy
-
-### Route-Level Connection Limits
-- Routes can define `security.maxConnections` to limit connections per route
-- Rust `HttpProxyService` enforces HTTP route limits after the route has matched and before forwarding
-- Exceeded route limits return terminal `429` responses; lower-priority fallback routes are not considered
-- Route counters are keyed by route id, then route name, and retained while non-zero across route updates
-- Streamed HTTP/H3 response bodies and WebSocket tunnels hold the route slot until the body/tunnel closes
-
-### HttpProxy Integration
-- When SmartProxy forwards to HttpProxy for TLS termination, it sends a `CLIENT_IP:<ip>\r\n` header
-- HttpProxy parses this header to track the real client IP, not the localhost IP
-- This ensures per-IP limits are enforced even for forwarded connections
-- The header is parsed in the connection handler before any data processing
-
-### Memory Optimization
-- Periodic cleanup runs every 60 seconds to remove:
-  - IPs with no active connections
-  - Expired rate limit timestamps (older than 1 minute)
-- Prevents memory accumulation from many unique IPs over time
-- Cleanup is automatic and runs in background with `unref()` to not keep process alive
-
-### Connection Cleanup Queue
-- Cleanup queue processes connections in batches to prevent overwhelming the system
-- Race condition prevention using `isProcessingCleanup` flag
-- Try-finally block ensures flag is always reset even if errors occur
-- New connections added during processing are queued for next batch
-
-### Important Implementation Notes
-- Always use `NodeJS.Timeout` type instead of `NodeJS.Timer` for interval/timeout references
-- IPv4/IPv6 normalization is handled (e.g., `::ffff:127.0.0.1` and `127.0.0.1` are treated as the same IP)
-- Connection limits are checked before route matching to prevent DoS attacks
-- SharedSecurityManager supports checking route-level limits via optional parameter
-
-## Log Deduplication
-
-To reduce log spam during high-traffic scenarios or attacks, SmartProxy implements log deduplication for repetitive events:
-
-### How It Works
-- Similar log events are batched and aggregated over a 5-second window
-- Instead of logging each event individually, a summary is emitted
-- Events are grouped by type and deduplicated by key (e.g., IP address, reason)
-
-### Deduplicated Event Types
-1. **Connection Rejections** (`connection-rejected`):
-   - Groups by rejection reason (global-limit, route-limit, etc.)
-   - Example: "Rejected 150 connections (reasons: global-limit: 100, route-limit: 50)"
-
-2. **IP Rejections** (`ip-rejected`):
-   - Groups by IP address
-   - Shows top offenders with rejection counts and reasons
-   - Example: "Rejected 500 connections from 10 IPs (top offenders: 192.168.1.100 (200x, rate-limit), ...)"
-
-3. **Connection Cleanups** (`connection-cleanup`):
-   - Groups by cleanup reason (normal, timeout, error, zombie, etc.)
-   - Example: "Cleaned up 250 connections (reasons: normal: 200, timeout: 30, error: 20)"
-
-4. **IP Tracking Cleanup** (`ip-cleanup`):
-   - Summarizes periodic IP cleanup operations
-   - Example: "IP tracking cleanup: removed 50 entries across 5 cleanup cycles"
-
-### Configuration
-- Default flush interval: 5 seconds
-- Maximum batch size: 100 events (triggers immediate flush)
-- Global periodic flush: Every 10 seconds (ensures logs are emitted regularly)
-- Process exit handling: Logs are flushed on SIGINT/SIGTERM
-
-### Benefits
-- Reduces log volume during attacks or high traffic
-- Provides better overview of patterns (e.g., which IPs are attacking)
-- Improves log readability and analysis
-- Prevents log storage overflow
-- Maintains detailed information in aggregated form
-
-### Log Output Examples
-
-Instead of hundreds of individual logs:
-```
-Connection rejected
-Connection rejected
-Connection rejected
-... (repeated 500 times)
-```
-
-You'll see:
-```
-[SUMMARY] Rejected 500 connections from 10 IPs in 5s (rate-limit: 350, per-ip-limit: 150) (top offenders: 192.168.1.100 (200x, rate-limit), 10.0.0.1 (150x, per-ip-limit))
-```
-
-Instead of:
-```
-Connection terminated: ::ffff:127.0.0.1 (client_closed). Active: 266
-Connection terminated: ::ffff:127.0.0.1 (client_closed). Active: 265
-... (repeated 266 times)
-```
-
-You'll see:
-```
-[SUMMARY] 266 HttpProxy connections terminated in 5s (reasons: client_closed: 266, activeConnections: 0)
-```
-
-### Rapid Event Handling
-- During attacks or high-volume scenarios, logs are flushed more frequently
-- If 50+ events occur within 1 second, immediate flush is triggered
-- Prevents memory buildup during flooding attacks
-- Maintains real-time visibility during incidents
-
-## Custom Certificate Provision Function
-
-The `certProvisionFunction` feature has been implemented to allow users to provide their own certificate generation logic.
-
-### Implementation Details
-
-1. **Type Definition**: The function must return `Promise<TSmartProxyCertProvisionObject>` where:
-   - `TSmartProxyCertProvisionObject = plugins.tsclass.network.ICert | 'http01'`
-   - Return `'http01'` to fallback to Let's Encrypt
-   - Return a certificate object for custom certificates
-
-2. **Certificate Manager Changes**:
-   - Added `certProvisionFunction` property to CertificateManager
-   - Modified `provisionAcmeCertificate()` to check custom function first
-   - Custom certificates are stored with source type 'custom'
-   - Expiry date extraction currently defaults to 90 days
-
-3. **Configuration Options**:
-   - `certProvisionFunction`: The custom provision function
-   - `certProvisionFallbackToAcme`: Whether to fallback to ACME on error (default: true)
-
-4. **Usage Example**:
-```typescript
-new SmartProxy({
-  certProvisionFunction: async (domain: string) => {
-    if (domain === 'internal.example.com') {
-      return {
-        cert: customCert,
-        key: customKey,
-        ca: customCA
-      } as unknown as TSmartProxyCertProvisionObject;
-    }
-    return 'http01'; // Use Let's Encrypt
-  },
-  certProvisionFallbackToAcme: true
-})
-```
-
-5. **Testing Notes**:
-   - Type assertions through `unknown` are needed in tests due to strict interface typing
-   - Mock certificate objects work for testing but need proper type casting
-   - The actual certificate parsing for expiry dates would need a proper X.509 parser
-
-### Future Improvements
-
-1. Implement proper certificate expiry date extraction using X.509 parsing
-2. Add support for returning expiry date with custom certificates
-3. Consider adding validation for custom certificate format
-4. Add events/hooks for certificate provisioning lifecycle
-
-## HTTPS/TLS Configuration Guide
-
-SmartProxy supports three TLS modes for handling HTTPS traffic. Understanding when to use each mode is crucial for correct configuration.
-
-### TLS Mode: Passthrough (SNI Routing)
-
-**When to use**: Backend server handles its own TLS certificates.
-
-**How it works**:
-1. Client connects with TLS ClientHello containing SNI (Server Name Indication)
-2. SmartProxy extracts the SNI hostname without decrypting
-3. Connection is forwarded to backend as-is (still encrypted)
-4. Backend server terminates TLS with its own certificate
-
-**Configuration**:
-```typescript
-{
-  match: { ports: 443, domains: 'backend.example.com' },
-  action: {
-    type: 'forward',
-    targets: [{ host: 'backend-server', port: 443 }],
-    tls: { mode: 'passthrough' }
-  }
-}
-```
-
-**Requirements**:
-- Backend must have valid TLS certificate for the domain
-- Client's SNI must be present (session tickets without SNI will be rejected)
-- No HTTP-level inspection possible (encrypted end-to-end)
-
-### TLS Mode: Terminate
-
-**When to use**: SmartProxy handles TLS, backend receives plain HTTP.
-
-**How it works**:
-1. Client connects with TLS ClientHello
-2. SmartProxy terminates TLS (decrypts traffic)
-3. Decrypted HTTP is forwarded to backend on plain HTTP port
-4. Backend receives unencrypted traffic
-
-**Configuration**:
-```typescript
-{
-  match: { ports: 443, domains: 'api.example.com' },
-  action: {
-    type: 'forward',
-    targets: [{ host: 'localhost', port: 8080 }],  // HTTP backend
-    tls: {
-      mode: 'terminate',
-      certificate: 'auto'  // Let's Encrypt, or provide { key, cert }
-    }
-  }
-}
-```
-
-**Requirements**:
-- ACME email configured for auto certificates: `acme: { email: 'admin@example.com' }`
-- Port 80 available for HTTP-01 challenges (or use DNS-01)
-- Backend accessible on HTTP port
-
-### TLS Mode: Terminate and Re-encrypt
-
-**When to use**: SmartProxy handles client TLS, but backend also requires TLS.
-
-**How it works**:
-1. Client connects with TLS ClientHello
-2. SmartProxy terminates client TLS (decrypts)
-3. SmartProxy creates new TLS connection to backend
-4. Traffic is re-encrypted for the backend connection
-
-**Configuration**:
-```typescript
-{
-  match: { ports: 443, domains: 'secure.example.com' },
-  action: {
-    type: 'forward',
-    targets: [{ host: 'backend-tls', port: 443 }],  // HTTPS backend
-    tls: {
-      mode: 'terminate-and-reencrypt',
-      certificate: 'auto'
-    }
-  }
-}
-```
-
-**Requirements**:
-- Same as 'terminate' mode
-- Backend must have valid TLS (can be self-signed for internal use)
-
-### HttpProxy Integration
-
-For TLS termination modes (`terminate` and `terminate-and-reencrypt`), SmartProxy uses an internal HttpProxy component:
-
-- HttpProxy listens on an internal port (default: 8443)
-- SmartProxy forwards TLS connections to HttpProxy for termination
-- Client IP is preserved via `CLIENT_IP:` header protocol
-- HTTP/2 and WebSocket are supported after TLS termination
-
-**Configuration**:
-```typescript
-{
-  useHttpProxy: [443],        // Ports that use HttpProxy for TLS termination
-  httpProxyPort: 8443,        // Internal HttpProxy port
-  acme: {
-    email: 'admin@example.com',
-    useProduction: true       // false for Let's Encrypt staging
-  }
-}
-```
-
-### Common Configuration Patterns
-
-**HTTP to HTTPS Redirect**:
-```typescript
-import { SocketHandlers } from '@push.rocks/smartproxy';
-
-const redirectRoute = {
-  name: 'http-to-https',
-  match: { ports: 80, domains: ['example.com', 'www.example.com'] },
-  action: {
-    type: 'socket-handler' as const,
-    socketHandler: SocketHandlers.httpRedirect('https://{domain}:443{path}', 301)
-  }
-};
-```
-
-**Complete HTTPS Server (with redirect)**:
-```typescript
-const routes = [
-  {
-    name: 'https-server',
-    match: { ports: 443, domains: 'example.com' },
-    action: {
-      type: 'forward' as const,
-      targets: [{ host: 'localhost', port: 8080 }],
-      tls: { mode: 'terminate' as const, certificate: 'auto' as const }
-    }
-  },
-  {
-    name: 'http-redirect',
-    match: { ports: 80, domains: 'example.com' },
-    action: {
-      type: 'socket-handler' as const,
-      socketHandler: SocketHandlers.httpRedirect('https://{domain}:443{path}', 301)
-    }
-  }
-];
-```
-
-**Load Balancer with Health Checks**:
-```typescript
-const lbRoute = {
-  name: 'load-balancer',
-  match: { ports: 443, domains: 'api.example.com' },
-  action: {
-    type: 'forward' as const,
-    targets: [
-      { host: 'backend1', port: 8080 },
-      { host: 'backend2', port: 8080 },
-      { host: 'backend3', port: 8080 }
-    ],
-    tls: { mode: 'terminate' as const, certificate: 'auto' as const },
-    loadBalancing: { algorithm: 'round-robin' as const }
-  }
-};
-```
-
-### Smart SNI Requirement (v22.3+)
-
-SmartProxy automatically determines when SNI is required for routing. Session tickets (TLS resumption without SNI) are now allowed in more scenarios:
-
-**SNI NOT required (session tickets allowed):**
-- Single passthrough route with static target(s) and no domain restriction
-- Single passthrough route with wildcard-only domain (`*` or `['*']`)
-- TLS termination routes (`terminate` or `terminate-and-reencrypt`)
-- Mixed terminate + passthrough routes (termination takes precedence)
-
-**SNI IS required (session tickets blocked):**
-- Multiple passthrough routes on the same port (need SNI to pick correct route)
-- Route has dynamic host function (e.g., `host: (ctx) => ctx.domain === 'api.example.com' ? 'api-backend' : 'web-backend'`)
-- Route has specific domain restriction (e.g., `domains: 'api.example.com'` or `domains: '*.example.com'`)
-
-This allows simple single-target passthrough setups to work with TLS session resumption, improving performance for clients that reuse connections.
-
-### Troubleshooting
-
-**"No SNI detected" errors**:
-- Client is using TLS session resumption without SNI
-- Solution: Configure route for TLS termination (allows session resumption), or ensure you have a single-target passthrough route with no domain restrictions
-
-**"HttpProxy not available" errors**:
-- `useHttpProxy` not configured for the port
-- Solution: Add port to `useHttpProxy` array in settings
-
-**Certificate provisioning failures**:
-- Port 80 not accessible for HTTP-01 challenges
-- ACME email not configured
-- Solution: Ensure port 80 is available and `acme.email` is set
-
-**Connection timeouts to HttpProxy**:
-- CLIENT_IP header parsing timeout (default: 2000ms)
-- Network congestion between SmartProxy and HttpProxy
-- Solution: Check localhost connectivity, increase timeout if needed
-
-## Static Rust Binaries
-
-- The Rust binaries in `dist_rust/` are statically linked (static-pie) via `"static": true` in the `@git.zone/tsrust` block of `.smartconfig.json` (tsrust >= 1.4.1). They run on both glibc (Debian/Ubuntu) and musl (Alpine) systems.
-- tsrust injects `RUSTFLAGS="-C target-feature=+crt-static"` only into its own per-target cargo invocations and verifies the result (no `PT_INTERP` ELF header). Keep `rust/.cargo/config.toml` free of `rustflags` entries: the injected env variable would replace them, and a repo-wide `rustflags` would also break plain `cargo test`/`cargo check` (proc-macros cannot build with `+crt-static` on linux-gnu without an explicit `--target`). The repo previously carried such `rustflags` lines; they were removed when the tsrust `static` option was adopted.
-- `pnpm run test:rust` runs the Rust unit tests; plain `cargo test` in `rust/` works as well.
-- Verify linkage manually with `ldd dist_rust/<binary>_linux_amd64` → "statically linked".