@serve.zone/dcrouter 2.12.4

package/readme.opsserver.md

@@ -0,0 +1,351 @@
+# DCRouter OpsServer Implementation Plan
+
+**Command to reread CLAUDE.md: `cat /home/philkunz/.claude/CLAUDE.md`**
+
+## Overview
+
+This document outlines the implementation plan for adding a TypedRequest-based API to the DCRouter OpsServer, following the patterns established in the cloudly project. The goal is to create a type-safe, reactive management dashboard with real-time statistics and monitoring capabilities.
+
+## Architecture Overview
+
+The implementation follows a clear separation of concerns:
+- **Backend**: TypedRequest handlers in OpsServer
+- **Frontend**: Reactive web components with Smartstate
+- **Communication**: Type-safe requests via TypedRequest pattern
+- **State Management**: Centralized state with reactive updates
+
+## Implementation Phases
+
+### Phase 1: Interface Definition ✓
+
+Create TypeScript interfaces for all API operations:
+
+#### Directory Structure ✓
+```
+ts_interfaces/
+  plugins.ts         # TypedRequest interfaces import
+  data/              # Data type definitions
+    auth.ts          # IIdentity interface
+    stats.ts         # Server, Email, DNS, Security types
+    index.ts         # Exports
+  requests/          # Request interfaces
+    admin.ts         # Authentication requests
+    config.ts        # Configuration management  
+    logs.ts          # Log retrieval with IVirtualStream
+    stats.ts         # Statistics endpoints
+    index.ts         # Exports
+```
+
+#### Key Interfaces Defined ✓
+- **Server Statistics**
+  - [x] `IReq_GetServerStatistics` - Server metrics with history
+  
+- **Email Operations**
+  - [x] `IReq_GetEmailStatistics` - Email delivery stats
+  - [x] `IReq_GetQueueStatus` - Queue monitoring
+  
+- **DNS Management**
+  - [x] `IReq_GetDnsStatistics` - DNS query metrics
+  
+- **Rate Limiting**
+  - [x] `IReq_GetRateLimitStatus` - Rate limit info
+  
+- **Security Metrics**
+  - [x] `IReq_GetSecurityMetrics` - Security stats and trends
+  - [x] `IReq_GetActiveConnections` - Connection monitoring
+  
+- **Logging**
+  - [x] `IReq_GetRecentLogs` - Paginated log retrieval
+  - [x] `IReq_GetLogStream` - Real-time log streaming with IVirtualStream
+  
+- **Configuration**
+  - [x] `IReq_GetConfiguration` - Read config
+  - [x] `IReq_UpdateConfiguration` - Update config
+  
+- **Authentication**
+  - [x] `IReq_AdminLoginWithUsernameAndPassword` - Admin login
+  - [x] `IReq_AdminLogout` - Logout
+  - [x] `IReq_VerifyIdentity` - Token verification
+  
+- **Health Check**
+  - [x] `IReq_GetHealthStatus` - Service health monitoring
+
+### Phase 2: Backend Implementation ✓
+
+#### 2.1 Enhance OpsServer (`ts/opsserver/classes.opsserver.ts`) ✓
+
+- [x] Add TypedRouter initialization
+- [x] Use TypedServer's built-in typedrouter
+- [x] CORS is already handled by TypedServer
+- [x] Add handler registration method
+
+```typescript
+// Example structure following cloudly pattern
+public typedrouter = new plugins.typedrequest.TypedRouter();
+
+constructor(private dcRouterRef: DcRouter) {
+  // Add our typedrouter to the dcRouter's main typedrouter
+  this.dcRouterRef.typedrouter.addTypedRouter(this.typedrouter);
+}
+
+public async start() {
+  // TypedServer already has a built-in typedrouter at /typedrequest
+  this.server = new plugins.typedserver.utilityservers.UtilityWebsiteServer({
+    domain: 'localhost',
+    feedMetadata: null,
+    serveDir: paths.distServe,
+  });
+  
+  // The server's typedrouter is automatically available
+  // Add the main dcRouter typedrouter to the server's typedrouter
+  this.server.typedrouter.addTypedRouter(this.dcRouterRef.typedrouter);
+  
+  this.setupHandlers();
+  await this.server.start(3000);
+}
+```
+
+**Note**: TypedServer automatically provides the `/typedrequest` endpoint with its built-in typedrouter. We just need to add our routers to it using the `addTypedRouter()` method.
+
+#### Hierarchical TypedRouter Structure
+
+Following cloudly's pattern, we'll use a hierarchical router structure:
+
+```
+TypedServer (built-in typedrouter at /typedrequest)
+  └── DcRouter.typedrouter (main router)
+      └── OpsServer.typedrouter (ops-specific handlers)
+          ├── StatsHandler.typedrouter
+          ├── ConfigHandler.typedrouter
+          └── SecurityHandler.typedrouter
+```
+
+This allows clean separation of concerns while keeping all handlers accessible through the single `/typedrequest` endpoint.
+
+#### 2.2 Create Handler Classes ✓
+
+Create modular handlers in `ts/opsserver/handlers/`:
+
+- [x] `stats.handler.ts` - Server and performance statistics
+- [x] `security.handler.ts` - Security and reputation metrics
+- [x] `config.handler.ts` - Configuration management
+- [x] `logs.handler.ts` - Log retrieval and streaming
+- [x] `admin.handler.ts` - Authentication and session management
+
+Each handler should:
+- Have its own typedrouter that gets added to OpsServer's router
+- Access the main DCRouter instance
+- Register handlers using TypedHandler instances
+- Format responses according to interfaces
+- Handle errors gracefully
+
+Example handler structure:
+```typescript
+export class StatsHandler {
+  public typedrouter = new plugins.typedrequest.TypedRouter();
+  
+  constructor(private opsServerRef: OpsServer) {
+    // Add this handler's router to the parent
+    this.opsServerRef.typedrouter.addTypedRouter(this.typedrouter);
+    this.registerHandlers();
+  }
+  
+  private registerHandlers() {
+    this.typedrouter.addTypedHandler(
+      new plugins.typedrequest.TypedHandler<IReq_GetServerStatistics>(
+        'getServerStatistics',
+        async (dataArg, toolsArg) => {
+          const stats = await this.collectServerStats();
+          return stats;
+        }
+      )
+    );
+  }
+}
+```
+
+### Phase 3: Frontend State Management ✓
+
+#### 3.1 Set up Smartstate (`ts_web/appstate.ts`) ✓
+
+- [x] Initialize Smartstate instance
+- [x] Create state parts with appropriate persistence
+- [x] Define initial state structures
+
+```typescript
+// State structure example
+interface IStatsState {
+  serverStats: IRes_ServerStatistics | null;
+  emailStats: IRes_EmailStatistics | null;
+  dnsStats: IRes_DnsStatistics | null;
+  lastUpdated: number;
+  isLoading: boolean;
+  error: string | null;
+}
+```
+
+#### 3.2 State Parts to Create ✓
+
+- [x] `statsState` - Runtime statistics (soft persistence)
+- [x] `configState` - Configuration data (soft persistence)
+- [x] `uiState` - UI preferences (persistent)
+- [x] `loginState` - Authentication state (persistent)
+
+### Phase 4: Frontend Integration ✓
+
+#### 4.1 API Client Setup ✓
+
+- [x] TypedRequest instances created inline within actions
+- [x] Base URL handled through relative paths
+- [x] Error handling integrated in actions
+- [x] Following cloudly pattern of creating requests within actions
+
+#### 4.2 Create Actions (`ts_web/appstate.ts`) ✓
+
+- [x] `loginAction` - Authentication with JWT
+- [x] `logoutAction` - Clear authentication state
+- [x] `fetchAllStatsAction` - Batch fetch all statistics
+- [x] `fetchConfigurationAction` - Get configuration
+- [x] `updateConfigurationAction` - Update configuration
+- [x] `fetchRecentLogsAction` - Get recent logs
+- [x] `toggleAutoRefreshAction` - Toggle auto-refresh
+- [x] `setActiveViewAction` - Change active view
+- [x] Error handling in all actions
+
+#### 4.3 Update Dashboard Component (`ts_web/elements/ops-dashboard.ts`) ✓
+
+- [x] Subscribe to state changes (login and UI state)
+- [x] Implement reactive UI updates
+- [x] Use dees-simple-login and dees-simple-appdash components
+- [x] Create view components for different sections
+- [x] Implement auto-refresh timer functionality
+
+### Phase 5: Component Structure ✓
+
+Created modular view components in `ts_web/elements/`:
+
+- [x] `ops-view-overview.ts` - Overview with server, email, and DNS statistics
+- [x] `ops-view-stats.ts` - Detailed statistics with tables and metrics
+- [x] `ops-view-logs.ts` - Log viewer with filtering and search
+- [x] `ops-view-config.ts` - Configuration editor with JSON editing
+- [x] `ops-view-security.ts` - Security metrics and threat monitoring
+- [x] `shared/ops-sectionheading.ts` - Reusable section heading component
+- [x] `shared/css.ts` - Shared CSS styles
+
+### Phase 6: Optional Enhancements
+
+#### 6.1 Authentication ✓ (Implemented)
+- [x] JWT-based authentication using `@push.rocks/smartjwt`
+- [x] Guards for identity validation and admin access
+- [x] Login/logout endpoints following cloudly pattern
+- [ ] Login component (frontend)
+- [ ] Protected route handling (frontend)
+- [ ] Session persistence (frontend)
+
+#### 6.2 Real-time Updates (future)
+- [ ] WebSocket integration for live stats
+- [ ] Push notifications for critical events
+- [ ] Event streaming for logs
+
+## Technical Stack
+
+### Dependencies to Use
+- `@api.global/typedserver` - Server with built-in typedrouter at `/typedrequest`
+- `@api.global/typedrequest` - TypedRouter and TypedHandler classes
+- `@design.estate/dees-domtools` - Frontend TypedRequest client
+- `@push.rocks/smartstate` - State management
+- `@design.estate/dees-element` - Web components
+- `@design.estate/dees-catalog` - UI components
+
+### Existing Dependencies to Leverage
+- Current DCRouter instance and statistics
+- Existing error handling patterns
+- Logger infrastructure
+- Security modules
+
+## Development Workflow
+
+1. **Start with interfaces** - Define all types first
+2. **Implement one handler** - Start with server stats
+3. **Create minimal frontend** - Test with one endpoint
+4. **Iterate** - Add more handlers and UI components
+5. **Polish** - Add error handling, loading states, etc.
+
+## Testing Strategy
+
+- [ ] Unit tests for handlers
+- [ ] Integration tests for API endpoints
+- [ ] Frontend component tests
+- [ ] End-to-end testing with real DCRouter instance
+
+## Success Criteria
+
+- Type-safe communication between frontend and backend
+- Real-time statistics display
+- Responsive and reactive UI
+- Clean, maintainable code structure
+- Consistent with cloudly patterns
+- Easy to extend with new features
+
+## Notes
+
+- Follow existing code conventions in the project
+- Use pnpm for all package management
+- Ensure all tests pass before marking complete
+- Document any deviations from the plan
+
+---
+
+## Progress Status
+
+### Completed ✓
+- **Phase 1: Interface Definition** - All TypedRequest interfaces created following cloudly pattern
+  - Created proper TypedRequest interfaces with `method`, `request`, and `response` properties
+  - Used `IVirtualStream` for log streaming
+  - Added `@api.global/typedrequest-interfaces` dependency
+  - All interfaces compile successfully
+
+- **Phase 2: Backend Implementation** - TypedRouter integration and handlers
+  - Enhanced OpsServer with hierarchical TypedRouter structure
+  - Created all handler classes with proper TypedHandler registration
+  - Implemented mock data responses for all endpoints
+  - Fixed all TypeScript compilation errors
+  - VirtualStream used for log streaming with Uint8Array encoding
+  - **JWT Authentication** - Following cloudly pattern:
+    - Added `@push.rocks/smartjwt` and `@push.rocks/smartguard` dependencies
+    - Updated IIdentity interface to match cloudly structure
+    - Implemented JWT-based authentication with RSA keypairs
+    - Created validIdentityGuard and adminIdentityGuard
+    - Added guard helpers for protecting endpoints
+    - Full test coverage for JWT authentication flows
+
+- **Phase 3: Frontend State Management** - Smartstate implementation
+  - Initialized Smartstate with proper state parts
+  - Created state interfaces for all data types
+  - Implemented persistent vs soft state persistence
+  - Set up reactive subscriptions
+
+- **Phase 4: Frontend Integration** - Complete dashboard implementation
+  - Created all state management actions with TypedRequest
+  - Implemented JWT authentication flow in frontend
+  - Built reactive dashboard with dees-simple-login and dees-simple-appdash
+  - Added auto-refresh functionality
+  - Fixed all interface import issues (using dist_ts_interfaces)
+
+- **Phase 5: Component Structure** - View components
+  - Created all view components following cloudly patterns
+  - Implemented reactive data binding with state subscriptions
+  - Added interactive features (filtering, editing, refresh controls)
+  - Used @design.estate/dees-catalog components throughout
+  - Created shared components and styles
+
+### Next Steps
+- Write comprehensive tests for handlers and frontend components
+- Implement real data sources (replace mock data)
+- Add WebSocket support for real-time updates
+- Enhance error handling and user feedback
+- Add more detailed charts and visualizations
+
+---
+
+*This plan is a living document. Update it as implementation progresses.*

package/test/helpers/server.loader.ts

@@ -0,0 +1,347 @@
+import * as plugins from '../../ts/plugins.js';
+import { UnifiedEmailServer } from '../../ts/mail/routing/classes.unified.email.server.js';
+import { createSmtpServer } from '../../ts/mail/delivery/smtpserver/index.js';
+import type { ISmtpServerOptions } from '../../ts/mail/delivery/smtpserver/interfaces.js';
+import type { net } from '../../ts/plugins.js';
+
+export interface ITestServerConfig {
+  port: number;
+  hostname?: string;
+  tlsEnabled?: boolean;
+  authRequired?: boolean;
+  timeout?: number;
+  testCertPath?: string;
+  testKeyPath?: string;
+  maxConnections?: number;
+  size?: number;
+  maxRecipients?: number;
+}
+
+export interface ITestServer {
+  server: any;
+  smtpServer: any;
+  port: number;
+  hostname: string;
+  config: ITestServerConfig;
+  startTime: number;
+}
+
+/**
+ * Starts a test SMTP server with the given configuration
+ */
+export async function startTestServer(config: ITestServerConfig): Promise<ITestServer> {
+  const serverConfig = {
+    port: config.port || 2525,
+    hostname: config.hostname || 'localhost',
+    tlsEnabled: config.tlsEnabled || false,
+    authRequired: config.authRequired || false,
+    timeout: config.timeout || 30000,
+    maxConnections: config.maxConnections || 100,
+    size: config.size || 10 * 1024 * 1024, // 10MB default
+    maxRecipients: config.maxRecipients || 100
+  };
+
+  // Create a mock email server for testing
+  const mockEmailServer = {
+    processEmailByMode: async (emailData: any) => {
+      console.log(`📧 [Test Server] Processing email:`, emailData.subject || 'No subject');
+      return emailData;
+    },
+    getRateLimiter: () => {
+      // Return a mock rate limiter for testing
+      return {
+        recordConnection: (_ip: string) => ({ allowed: true, remaining: 100 }),
+        checkConnectionLimit: async (_ip: string) => ({ allowed: true, remaining: 100 }),
+        checkMessageLimit: (_senderAddress: string, _ip: string, _recipientCount?: number, _pattern?: string, _domain?: string) => ({ allowed: true, remaining: 1000 }),
+        checkRecipientLimit: async (_session: any) => ({ allowed: true, remaining: 50 }),
+        recordAuthenticationFailure: async (_ip: string) => {},
+        recordSyntaxError: async (_ip: string) => {},
+        recordCommandError: async (_ip: string) => {},
+        isBlocked: async (_ip: string) => false,
+        cleanup: async () => {}
+      };
+    }
+  } as any;
+
+  // Load test certificates
+  let key: string;
+  let cert: string;
+  
+  if (serverConfig.tlsEnabled) {
+    try {
+      const certPath = config.testCertPath || './test/fixtures/test-cert.pem';
+      const keyPath = config.testKeyPath || './test/fixtures/test-key.pem';
+      
+      cert = await plugins.fs.promises.readFile(certPath, 'utf8');
+      key = await plugins.fs.promises.readFile(keyPath, 'utf8');
+    } catch (error) {
+      console.warn('⚠️ Failed to load TLS certificates, falling back to self-signed');
+      // Generate self-signed certificate for testing
+      const forge = await import('node-forge');
+      const pki = forge.default.pki;
+      
+      // Generate key pair
+      const keys = pki.rsa.generateKeyPair(2048);
+      
+      // Create certificate
+      const certificate = pki.createCertificate();
+      certificate.publicKey = keys.publicKey;
+      certificate.serialNumber = '01';
+      certificate.validity.notBefore = new Date();
+      certificate.validity.notAfter = new Date();
+      certificate.validity.notAfter.setFullYear(certificate.validity.notBefore.getFullYear() + 1);
+      
+      const attrs = [{
+        name: 'commonName',
+        value: serverConfig.hostname
+      }];
+      certificate.setSubject(attrs);
+      certificate.setIssuer(attrs);
+      certificate.sign(keys.privateKey);
+      
+      // Convert to PEM
+      cert = pki.certificateToPem(certificate);
+      key = pki.privateKeyToPem(keys.privateKey);
+    }
+  } else {
+    // Always provide a self-signed certificate for non-TLS servers
+    // This is required by the interface
+    const forge = await import('node-forge');
+    const pki = forge.default.pki;
+    
+    // Generate key pair
+    const keys = pki.rsa.generateKeyPair(2048);
+    
+    // Create certificate
+    const certificate = pki.createCertificate();
+    certificate.publicKey = keys.publicKey;
+    certificate.serialNumber = '01';
+    certificate.validity.notBefore = new Date();
+    certificate.validity.notAfter = new Date();
+    certificate.validity.notAfter.setFullYear(certificate.validity.notBefore.getFullYear() + 1);
+    
+    const attrs = [{
+      name: 'commonName',
+      value: serverConfig.hostname
+    }];
+    certificate.setSubject(attrs);
+    certificate.setIssuer(attrs);
+    certificate.sign(keys.privateKey);
+    
+    // Convert to PEM
+    cert = pki.certificateToPem(certificate);
+    key = pki.privateKeyToPem(keys.privateKey);
+  }
+
+  // SMTP server options
+  const smtpOptions: ISmtpServerOptions = {
+    port: serverConfig.port,
+    hostname: serverConfig.hostname,
+    key: key,
+    cert: cert,
+    maxConnections: serverConfig.maxConnections,
+    size: serverConfig.size,
+    maxRecipients: serverConfig.maxRecipients,
+    socketTimeout: serverConfig.timeout,
+    connectionTimeout: serverConfig.timeout * 2,
+    cleanupInterval: 300000,
+    auth: serverConfig.authRequired ? ({
+      required: true,
+      methods: ['PLAIN', 'LOGIN'] as ('PLAIN' | 'LOGIN' | 'OAUTH2')[],
+      validateUser: async (username: string, password: string) => {
+        // Test server accepts these credentials
+        return username === 'testuser' && password === 'testpass';
+      }
+    } as any) : undefined
+  };
+
+  // Create SMTP server
+  const smtpServer = await createSmtpServer(mockEmailServer, smtpOptions);
+  
+  // Start the server
+  await smtpServer.listen();
+  
+  // Wait for server to be ready
+  await waitForServerReady(serverConfig.hostname, serverConfig.port);
+  
+  console.log(`✅ Test SMTP server started on ${serverConfig.hostname}:${serverConfig.port}`);
+  
+  return {
+    server: mockEmailServer,
+    smtpServer: smtpServer,
+    port: serverConfig.port,
+    hostname: serverConfig.hostname,
+    config: serverConfig,
+    startTime: Date.now()
+  };
+}
+
+/**
+ * Stops a test SMTP server
+ */
+export async function stopTestServer(testServer: ITestServer): Promise<void> {
+  if (!testServer || !testServer.smtpServer) {
+    console.warn('⚠️ No test server to stop');
+    return;
+  }
+  
+  try {
+    console.log(`🛑 Stopping test SMTP server on ${testServer.hostname}:${testServer.port}`);
+    
+    // Stop the SMTP server
+    if (testServer.smtpServer.close && typeof testServer.smtpServer.close === 'function') {
+      await testServer.smtpServer.close();
+    }
+    
+    // Wait for port to be free
+    await waitForPortFree(testServer.port);
+    
+    console.log(`✅ Test SMTP server stopped`);
+  } catch (error) {
+    console.error('❌ Error stopping test server:', error);
+    throw error;
+  }
+}
+
+/**
+ * Wait for server to be ready to accept connections
+ */
+async function waitForServerReady(hostname: string, port: number, timeout: number = 10000): Promise<void> {
+  const startTime = Date.now();
+  
+  while (Date.now() - startTime < timeout) {
+    try {
+      await new Promise<void>((resolve, reject) => {
+        const socket = plugins.net.createConnection({ port, host: hostname });
+        
+        socket.on('connect', () => {
+          socket.end();
+          resolve();
+        });
+        
+        socket.on('error', reject);
+        
+        setTimeout(() => {
+          socket.destroy();
+          reject(new Error('Connection timeout'));
+        }, 1000);
+      });
+      
+      return; // Server is ready
+    } catch {
+      // Server not ready yet, wait and retry
+      await new Promise(resolve => setTimeout(resolve, 100));
+    }
+  }
+  
+  throw new Error(`Server did not become ready within ${timeout}ms`);
+}
+
+/**
+ * Wait for port to be free
+ */
+async function waitForPortFree(port: number, timeout: number = 5000): Promise<void> {
+  const startTime = Date.now();
+  
+  while (Date.now() - startTime < timeout) {
+    const isFree = await isPortFree(port);
+    if (isFree) {
+      return;
+    }
+    await new Promise(resolve => setTimeout(resolve, 100));
+  }
+  
+  console.warn(`⚠️ Port ${port} still in use after ${timeout}ms`);
+}
+
+/**
+ * Check if a port is free
+ */
+async function isPortFree(port: number): Promise<boolean> {
+  return new Promise((resolve) => {
+    const server = plugins.net.createServer();
+    
+    server.listen(port, () => {
+      server.close(() => resolve(true));
+    });
+    
+    server.on('error', () => resolve(false));
+  });
+}
+
+/**
+ * Get an available port for testing
+ */
+export async function getAvailablePort(startPort: number = 25000): Promise<number> {
+  for (let port = startPort; port < startPort + 1000; port++) {
+    if (await isPortFree(port)) {
+      return port;
+    }
+  }
+  throw new Error(`No available ports found starting from ${startPort}`);
+}
+
+/**
+ * Create test email data
+ */
+export function createTestEmail(options: {
+  from?: string;
+  to?: string | string[];
+  subject?: string;
+  text?: string;
+  html?: string;
+  attachments?: any[];
+} = {}): any {
+  return {
+    from: options.from || 'test@example.com',
+    to: options.to || 'recipient@example.com',
+    subject: options.subject || 'Test Email',
+    text: options.text || 'This is a test email',
+    html: options.html || '<p>This is a test email</p>',
+    attachments: options.attachments || [],
+    date: new Date(),
+    messageId: `<${Date.now()}@test.example.com>`
+  };
+}
+
+/**
+ * Simple test server for custom protocol testing
+ */
+export interface ISimpleTestServer {
+  server: any;
+  hostname: string;
+  port: number;
+}
+
+export async function createTestServer(options: {
+  onConnection?: (socket: any) => void | Promise<void>;
+  port?: number;
+  hostname?: string;
+}): Promise<ISimpleTestServer> {
+  const hostname = options.hostname || 'localhost';
+  const port = options.port || await getAvailablePort();
+  
+  const server = plugins.net.createServer((socket) => {
+    if (options.onConnection) {
+      const result = options.onConnection(socket);
+      if (result && typeof result.then === 'function') {
+        result.catch(error => {
+          console.error('Error in onConnection handler:', error);
+          socket.destroy();
+        });
+      }
+    }
+  });
+  
+  return new Promise((resolve, reject) => {
+    server.listen(port, hostname, () => {
+      resolve({
+        server,
+        hostname,
+        port
+      });
+    });
+    
+    server.on('error', reject);
+  });
+}