jinaga-server 3.5.1 → 3.5.2

package/docs/analysis/DIAGNOSTIC_LOGGING_README.md

@@ -0,0 +1,201 @@
+# Diagnostic Logging for StreamFeed Push Notification Issues
+
+This document describes the comprehensive diagnostic logging system implemented to identify why some clients fail to receive push notifications when multiple clients are simultaneously connected to streaming feeds.
+
+## Overview
+
+The diagnostic logging has been added to track:
+1. **Listener Management** - Registration/deregistration of specification listeners
+2. **Event Broadcasting** - How events are distributed to listeners
+3. **Stream State Changes** - Stream lifecycle and data flow
+4. **Connection Management** - HTTP connection lifecycle and cleanup
+5. **Concurrent Access Patterns** - Race conditions and timing issues
+
+## Components Modified
+
+### 1. ObservableSource (`../jinaga.js/src/observable/observable.ts`)
+
+**Added logging for:**
+- Listener registration with counts and specification details
+- Listener removal with timing and race condition detection
+- Event notification sequences with performance metrics
+- Race condition detection during listener array modifications
+
+**Key log patterns to watch for:**
+```
+[ObservableSource] RACE CONDITION DETECTED - Listener count changed during snapshot
+[ObservableSource] Listener NOT FOUND during removal
+[ObservableSource] SLOW notification - Duration: >100ms
+```
+
+### 2. Stream Class (`src/http/stream.ts`)
+
+**Added logging for:**
+- Stream creation with unique IDs
+- Handler registration and data replay
+- Feed operations with performance tracking
+- Stream closure and cleanup
+- Error handling in handlers
+
+**Key log patterns to watch for:**
+```
+[Stream:ID] Attempted to feed data to closed stream
+[Stream:ID] SLOW handler - Duration: >50ms
+[Stream:ID] ERROR in handler
+```
+
+### 3. StreamFeed Function (`src/http/router.ts`)
+
+**Added logging for:**
+- Connection establishment with unique IDs
+- Initial data streaming progress
+- Real-time listener setup
+- Event processing and filtering
+- Connection cleanup and resource management
+
+**Key log patterns to watch for:**
+```
+[StreamFeed:ID] EVENT RECEIVED - Results: X
+[StreamFeed:ID] No matching results - skipping response
+[StreamFeed:ID] SLOW event processing - Duration: >100ms
+```
+
+### 4. HTTP Connection Management (`src/http/router.ts`)
+
+**Added logging for:**
+- Client connection/disconnection events
+- Message transmission with timing
+- Timeout handling
+- Socket management
+- Error conditions
+
+**Key log patterns to watch for:**
+```
+[HttpConnection:ID] CLIENT DISCONNECTED
+[HttpConnection:ID] TIMEOUT TRIGGERED
+[HttpConnection:ID] SLOW write operation - Duration: >50ms
+```
+
+## How to Use the Diagnostic Logging
+
+### 1. Enable Logging
+
+The logging uses both `console.log` and the existing `Trace` system. Ensure your application is configured to capture both:
+
+```javascript
+// For console logs
+process.env.NODE_ENV = 'development';
+
+// For Trace logs (if using structured logging)
+// Configure your trace system to capture info/warn/error levels
+```
+
+### 2. Run the Test Script
+
+Use the provided test script to simulate concurrent connections:
+
+```bash
+# Replace 'your-feed-hash' with an actual feed hash from your system
+node test-concurrent-streams.js your-feed-hash-here
+```
+
+The test script will:
+- Create 5 concurrent streaming connections
+- Stagger connections by 1 second each
+- Monitor message distribution for 30 seconds
+- Report statistics and detect potential issues
+
+### 3. Analyze the Logs
+
+Look for these critical patterns in the logs:
+
+#### Race Condition Indicators
+```
+[ObservableSource] RACE CONDITION DETECTED - Listener count changed during snapshot
+[ObservableSource] Listener NOT FOUND during removal
+```
+
+#### Performance Issues
+```
+[StreamFeed:ID] SLOW event processing - Duration: >100ms
+[HttpConnection:ID] SLOW write operation - Duration: >50ms
+[Stream:ID] SLOW handler - Duration: >50ms
+```
+
+#### Connection Issues
+```
+[HttpConnection:ID] Attempted to write to disconnected client
+[Stream:ID] Attempted to feed data to closed stream
+```
+
+#### Event Distribution Problems
+```
+[StreamFeed:ID] No matching results - skipping response
+[ObservableSource] Processing spec X/Y - Listeners: 0
+```
+
+### 4. Expected Behavior vs Issues
+
+**Normal Operation:**
+- All clients should receive similar message counts
+- Event processing should be fast (<100ms)
+- No race condition warnings
+- Clean connection cleanup
+
+**Problem Indicators:**
+- Significant variance in message counts between clients
+- Race condition warnings in ObservableSource
+- Slow event processing times
+- Errors during listener removal
+- Messages sent to closed streams
+
+## Monitoring Recommendations
+
+### 1. Real-time Monitoring
+
+Set up log aggregation to monitor these patterns in production:
+
+```bash
+# Example: Monitor for race conditions
+tail -f application.log | grep "RACE CONDITION DETECTED"
+
+# Example: Monitor for slow operations
+tail -f application.log | grep "SLOW"
+
+# Example: Monitor connection issues
+tail -f application.log | grep "CLIENT DISCONNECTED\|TIMEOUT TRIGGERED"
+```
+
+### 2. Metrics to Track
+
+- **Message Distribution Variance**: Difference between max and min messages received by clients
+- **Event Processing Time**: Average time to process and distribute events
+- **Connection Cleanup Time**: Time taken to remove listeners during disconnection
+- **Race Condition Frequency**: Number of race condition warnings per hour
+
+### 3. Alerting Thresholds
+
+Consider setting up alerts for:
+- Race condition warnings (any occurrence)
+- Event processing time >500ms
+- Message distribution variance >10 messages
+- Connection cleanup time >1000ms
+
+## Next Steps
+
+After collecting diagnostic data:
+
+1. **Identify Root Cause**: Use the logs to confirm which of the suspected issues is occurring
+2. **Implement Fixes**: Based on findings, implement appropriate synchronization mechanisms
+3. **Validate Fixes**: Re-run tests to confirm issues are resolved
+4. **Performance Tuning**: Optimize any slow operations identified
+
+## Suspected Issues Being Tracked
+
+1. **Race Conditions in Listener Management**: ObservableSource listener array modifications during concurrent access
+2. **Shared State Corruption**: Event broadcasting while listeners are being modified
+3. **Stream State Management**: Race conditions between stream closure and data feeding
+4. **Connection Cleanup Timing**: Double cleanup or cleanup during active operations
+5. **Async Event Processing**: Out-of-order bookmark updates causing missed notifications
+
+The diagnostic logging will help confirm which of these issues is the primary cause of the selective push notification failures.

package/docs/analysis/security-audit-feeds-hash-handler.md

@@ -0,0 +1,407 @@
+# Security and Reliability Audit: `/feeds/:hash` Handler - Streaming Path
+
+## Executive Summary
+
+This document presents a comprehensive security and reliability audit of the `/feeds/:hash` handler in `src/http/router.ts`, specifically analyzing the execution path when the Accept header contains `application/x-jinaga-feed-stream`. The audit identifies critical security vulnerabilities, reliability issues, and provides detailed remediation strategies.
+
+## Scope of Analysis
+
+**Target Handler**: `/feeds/:hash` GET endpoint  
+**Execution Path**: `application/x-jinaga-feed-stream` Accept header flow  
+**Key Methods Analyzed**:
+- `getOrStream()` (lines 38-92)
+- `streamFeed()` (lines 468-509)
+- `streamAllInitialResults()` (lines 511-568)
+- `streamFeedResponse()` (lines 570-583)
+
+## Critical Security Findings
+
+### 1. Authentication Bypass Vulnerability
+**CVSS Score: 9.1 (Critical)**  
+**CWE-287: Improper Authentication**
+
+**Location**: Lines 43, 468-472  
+**Issue**: The handler accepts null users without explicit authentication validation.
+
+```typescript
+const user = <RequestUser>(req as any).user;  // Line 43
+// No authentication check before proceeding
+```
+
+**Proof of Concept**:
+```bash
+curl -H "Accept: application/x-jinaga-feed-stream" \
+     http://target/feeds/arbitrary_hash
+```
+
+**Business Impact**: 
+- Unauthorized access to sensitive feed data
+- Potential data exfiltration
+- Compliance violations (GDPR, HIPAA)
+
+**Remediation**:
+```typescript
+// Add explicit authentication check
+if (!user && requiresAuthentication(feedHash)) {
+    res.sendStatus(401);
+    return;
+}
+```
+
+### 2. Hash Parameter Injection Vulnerability
+**CVSS Score: 8.2 (High)**  
+**CWE-20: Improper Input Validation**
+
+**Location**: Lines 469, 441  
+**Issue**: Hash parameter lacks validation and sanitization.
+
+```typescript
+const feedHash = params["hash"];  // No validation
+if (!feedHash) {
+    return null;  // Only checks for existence
+}
+```
+
+**Attack Vectors**:
+- Path traversal: `../../../sensitive_data`
+- SQL injection via hash lookup
+- Hash collision attacks
+- Enumeration attacks
+
+**Proof of Concept**:
+```bash
+# Path traversal attempt
+curl -H "Accept: application/x-jinaga-feed-stream" \
+     "http://target/feeds/../../../etc/passwd"
+
+# Hash enumeration
+for i in {1..1000}; do
+    curl -s -o /dev/null -w "%{http_code}" \
+         "http://target/feeds/hash_$i" | grep -v 404
+done
+```
+
+**Remediation**:
+```typescript
+// Add hash validation
+const HASH_PATTERN = /^[a-fA-F0-9]{64}$/;
+if (!feedHash || !HASH_PATTERN.test(feedHash)) {
+    res.sendStatus(400);
+    return;
+}
+```
+
+### 3. Timing Attack Vulnerability
+**CVSS Score: 6.5 (Medium)**  
+**CWE-208: Observable Timing Discrepancy**
+
+**Location**: Lines 474-477  
+**Issue**: Feed lookup timing reveals hash existence.
+
+```typescript
+const feedDefinition = await this.feedCache.getFeed(feedHash);
+if (!feedDefinition) {
+    return null;  // Different timing for valid vs invalid hashes
+}
+```
+
+**Remediation**:
+```typescript
+// Constant-time hash validation
+const [feedDefinition, _] = await Promise.all([
+    this.feedCache.getFeed(feedHash),
+    this.addConstantTimeDelay()
+]);
+```
+
+### 4. Denial of Service Vulnerabilities
+**CVSS Score: 7.5 (High)**  
+**CWE-400: Uncontrolled Resource Consumption**
+
+#### 4.1 Memory Exhaustion
+**Location**: Lines 485-509, 511-568  
+**Issue**: Unbounded stream creation and memory usage.
+
+```typescript
+const stream = new Stream<FeedResponse>();  // No memory limits
+// Continuous streaming without backpressure management
+```
+
+#### 4.2 Connection Exhaustion
+**Location**: Lines 58-60, 61-63  
+**Issue**: Inadequate connection management.
+
+```typescript
+const timeout = setTimeout(() => {
+    response.close();
+}, 5 * 60 * 1000);  // 5-minute timeout may be too long
+```
+
+**Attack Scenario**:
+```bash
+# Connection exhaustion attack
+for i in {1..10000}; do
+    curl -H "Accept: application/x-jinaga-feed-stream" \
+         "http://target/feeds/valid_hash" &
+done
+```
+
+**Remediation**:
+```typescript
+// Add connection limits and backpressure
+const MAX_CONCURRENT_STREAMS = 100;
+const STREAM_TIMEOUT = 30000; // 30 seconds
+const MAX_MEMORY_PER_STREAM = 10 * 1024 * 1024; // 10MB
+```
+
+### 5. Race Condition Vulnerabilities
+**CVSS Score: 6.8 (Medium)**  
+**CWE-362: Concurrent Execution using Shared Resource**
+
+**Location**: Lines 492-502, 523-565  
+**Issue**: Concurrent access to shared resources without proper synchronization.
+
+```typescript
+// Multiple async operations without proper coordination
+const listeners = inverses.map(inverse => this.factManager.addSpecificationListener(
+    inverse.inverseSpecification,
+    async (results) => {
+        // Race condition: bookmark updates
+        bookmark = await this.streamFeedResponse(...);
+    }
+));
+```
+
+### 6. Information Disclosure Vulnerabilities
+**CVSS Score: 5.3 (Medium)**  
+**CWE-200: Information Exposure**
+
+#### 6.1 Error Information Leakage
+**Location**: Lines 722-738  
+**Issue**: Detailed error messages expose internal system information.
+
+```typescript
+res.status(500).send(error.message);  // Exposes internal errors
+```
+
+#### 6.2 Timing-Based Information Disclosure
+**Location**: Lines 523-565  
+**Issue**: Processing time reveals data characteristics.
+
+## Reliability and Performance Issues
+
+### 1. Infinite Loop Vulnerability
+**Severity: High**  
+**Location**: Lines 523-565
+
+```typescript
+while (hasMoreResults && pageCount < maxPages) {
+    // Potential infinite loop if bookmark doesn't change
+    if (newBookmark === bookmark) {
+        hasMoreResults = false;  // Mitigation exists but insufficient
+    }
+}
+```
+
+**Issue**: Insufficient loop termination conditions.
+
+### 2. Resource Leak Vulnerabilities
+**Severity: High**  
+**Location**: Lines 503-507
+
+```typescript
+stream.done(() => {
+    for (const listener of listeners) {
+        this.factManager.removeSpecificationListener(listener);
+    }
+});
+```
+
+**Issue**: Listeners may not be cleaned up if stream.done() is not called.
+
+### 3. Database Connection Pool Exhaustion
+**Severity: Medium**  
+**Location**: Lines 524, 571
+
+Multiple concurrent database calls without connection pooling limits.
+
+## Detailed Remediation Strategy
+
+### Priority 1: Critical Security Fixes
+
+1. **Implement Authentication Validation**
+   ```typescript
+   // Add to getOrStream function
+   if (requiresAuth && !user) {
+       res.sendStatus(401);
+       return;
+   }
+   ```
+
+2. **Add Hash Parameter Validation**
+   ```typescript
+   const validateHash = (hash: string): boolean => {
+       return /^[a-fA-F0-9]{64}$/.test(hash) && hash.length === 64;
+   };
+   ```
+
+3. **Implement Rate Limiting**
+   ```typescript
+   const rateLimiter = rateLimit({
+       windowMs: 15 * 60 * 1000, // 15 minutes
+       max: 100 // limit each IP to 100 requests per windowMs
+   });
+   ```
+
+### Priority 2: DoS Prevention
+
+1. **Connection Management**
+   ```typescript
+   const connectionManager = new ConnectionManager({
+       maxConcurrentStreams: 100,
+       maxStreamDuration: 30000,
+       maxMemoryPerStream: 10 * 1024 * 1024
+   });
+   ```
+
+2. **Backpressure Implementation**
+   ```typescript
+   class BackpressureStream<T> extends Stream<T> {
+       private buffer: T[] = [];
+       private maxBufferSize = 1000;
+       
+       feed(data: T) {
+           if (this.buffer.length >= this.maxBufferSize) {
+               this.pause();
+           }
+           super.feed(data);
+       }
+   }
+   ```
+
+### Priority 3: Monitoring and Logging
+
+1. **Security Event Logging**
+   ```typescript
+   const securityLogger = {
+       logSuspiciousActivity: (event: SecurityEvent) => {
+           // Log to SIEM system
+       },
+       logAccessAttempt: (user: string, resource: string, success: boolean) => {
+           // Audit trail
+       }
+   };
+   ```
+
+2. **Performance Monitoring**
+   ```typescript
+   const performanceMonitor = {
+       trackStreamDuration: (streamId: string, duration: number) => {},
+       trackMemoryUsage: (streamId: string, usage: number) => {},
+       trackConcurrentConnections: (count: number) => {}
+   };
+   ```
+
+## Supply Chain Security Analysis
+
+### Dependencies Review
+- **express**: Check for known CVEs in routing
+- **jinaga**: Internal dependency - requires separate audit
+- **Stream class**: Custom implementation needs security review
+
+### Recommended Actions
+1. Implement dependency scanning in CI/CD
+2. Regular security updates
+3. Vendor security assessment for jinaga library
+
+## Compliance Considerations
+
+### GDPR Compliance
+- Data minimization in streams
+- Right to erasure implementation
+- Consent management for streaming data
+
+### SOC 2 Compliance
+- Access logging and monitoring
+- Data encryption in transit
+- Incident response procedures
+
+## Testing Strategy
+
+### Security Testing
+1. **Penetration Testing**
+   - Authentication bypass attempts
+   - Input validation testing
+   - DoS attack simulation
+
+2. **Automated Security Scanning**
+   - SAST tools integration
+   - Dependency vulnerability scanning
+   - Runtime security monitoring
+
+### Performance Testing
+1. **Load Testing**
+   - Concurrent stream handling
+   - Memory usage under load
+   - Database connection pooling
+
+2. **Chaos Engineering**
+   - Network partition simulation
+   - Database failure scenarios
+   - Memory pressure testing
+
+## Implementation Timeline
+
+### Phase 1 (Immediate - 1 week)
+- Critical authentication fixes
+- Input validation implementation
+- Basic rate limiting
+
+### Phase 2 (Short-term - 2-4 weeks)
+- DoS prevention measures
+- Enhanced monitoring
+- Security logging
+
+### Phase 3 (Medium-term - 1-3 months)
+- Comprehensive testing
+- Performance optimization
+- Compliance implementation
+
+## Conclusion
+
+The `/feeds/:hash` handler contains multiple critical security vulnerabilities that require immediate attention. The streaming functionality, while providing real-time capabilities, introduces significant attack surface that must be properly secured. Implementation of the recommended remediation strategies will significantly improve the security posture and reliability of the system.
+
+**Risk Assessment**: **HIGH** - Immediate action required  
+**Business Impact**: **CRITICAL** - Potential for data breach and service disruption  
+**Remediation Complexity**: **MEDIUM** - Requires coordinated development effort
+
+## Appendix A: Security Checklist
+
+- [ ] Authentication validation implemented
+- [ ] Input sanitization and validation
+- [ ] Rate limiting configured
+- [ ] Connection management implemented
+- [ ] Error handling sanitized
+- [ ] Logging and monitoring deployed
+- [ ] Security testing completed
+- [ ] Performance testing completed
+- [ ] Documentation updated
+- [ ] Team training conducted
+
+## Appendix B: Monitoring Queries
+
+```sql
+-- Suspicious hash access patterns
+SELECT hash, COUNT(*), MIN(timestamp), MAX(timestamp)
+FROM feed_access_log
+WHERE timestamp > NOW() - INTERVAL '1 hour'
+GROUP BY hash
+HAVING COUNT(*) > 100;
+
+-- Failed authentication attempts
+SELECT ip_address, COUNT(*)
+FROM security_events
+WHERE event_type = 'auth_failure'
+  AND timestamp > NOW() - INTERVAL '1 hour'
+GROUP BY ip_address
+HAVING COUNT(*) > 10;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "jinaga-server",
-  "version": "3.5.1",
+  "version": "3.5.2",
   "description": "Data management for web and mobile applications.",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -48,7 +48,7 @@
   },
   "dependencies": {
     "express": "^4.21.2",
-    "jinaga": "^6.7.0",
+    "jinaga": "^6.7.9",
     "pg": "^8.13.1"
   }
 }