jinaga-server 3.5.1 → 3.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/authorization/authorization-keystore.js +2 -2
- package/dist/authorization/authorization-keystore.js.map +1 -1
- package/dist/http/csv-metadata.d.ts +24 -0
- package/dist/http/csv-metadata.js +3 -0
- package/dist/http/csv-metadata.js.map +1 -0
- package/dist/http/csv-validator.d.ts +23 -0
- package/dist/http/csv-validator.js +83 -0
- package/dist/http/csv-validator.js.map +1 -0
- package/dist/http/output-formatters.d.ts +28 -0
- package/dist/http/output-formatters.js +197 -0
- package/dist/http/output-formatters.js.map +1 -0
- package/dist/http/result-stream.d.ts +35 -0
- package/dist/http/result-stream.js +78 -0
- package/dist/http/result-stream.js.map +1 -0
- package/dist/http/router.d.ts +7 -1
- package/dist/http/router.js +276 -70
- package/dist/http/router.js.map +1 -1
- package/dist/http/stream.d.ts +3 -0
- package/dist/http/stream.js +79 -5
- package/dist/http/stream.js.map +1 -1
- package/dist/postgres/postgres-store.js +7 -5
- package/dist/postgres/postgres-store.js.map +1 -1
- package/dist/postgres/purge-sql.js +15 -2
- package/dist/postgres/purge-sql.js.map +1 -1
- package/dist/postgres/specification-result-sql.d.ts +2 -0
- package/dist/postgres/specification-result-sql.js +49 -5
- package/dist/postgres/specification-result-sql.js.map +1 -1
- package/dist/postgres/specification-sql.js +28 -6
- package/dist/postgres/specification-sql.js.map +1 -1
- package/docs/analysis/DIAGNOSTIC_LOGGING_README.md +201 -0
- package/docs/analysis/security-audit-feeds-hash-handler.md +407 -0
- package/docs/design/feeds-persistent-proxy-plan.md +294 -0
- package/integration-test/integration-test/jinaga-test/package-lock.json +6 -0
- package/integration-test/jinaga-test/Dockerfile +1 -6
- package/integration-test/jinaga-test/http-test-helpers.js +77 -0
- package/integration-test/jinaga-test/jinaga.test.js +9 -9
- package/integration-test/jinaga-test/package-lock.json +329 -32
- package/integration-test/jinaga-test/package.json +5 -1
- package/integration-test/jinaga-test/read-csv.test.js +330 -0
- package/integration-test/jinaga-test/read-endpoint.test.js +442 -0
- package/integration-test/jinaga-test/read-performance.test.js +221 -0
- package/integration-test/jinaga-test/read-streaming.test.js +199 -0
- package/integration-test/jinaga-test/specification-conditions.test.js +147 -0
- package/integration-test/jinaga-test/test-helpers.js +12 -0
- package/integration-test/jinaga-test/time-projection.test.js +235 -0
- package/package.json +3 -2
- package/src/authorization/authorization-keystore.ts +4 -4
- package/src/http/csv-metadata.ts +34 -0
- package/src/http/csv-validator.ts +90 -0
- package/src/http/output-formatters.ts +196 -0
- package/src/http/result-stream.ts +69 -0
- package/src/http/router.ts +344 -62
- package/src/http/stream.ts +97 -5
- package/src/postgres/postgres-store.ts +13 -6
- package/src/postgres/purge-sql.ts +18 -2
- package/src/postgres/specification-result-sql.ts +70 -8
- package/src/postgres/specification-sql.ts +36 -8
- package/test/http/csv-validatorSpec.ts +205 -0
- package/test/http/result-streamSpec.ts +151 -0
- package/test/postgres/postgresReadSpec.ts +192 -14
- package/test/postgres/postgresSpecificationSpec.ts +3 -3
- package/test/postgres/specificationConditionsSpec.ts +268 -0
- package/test-concurrent-streams.js +219 -0
- package/documentation/streaming-feeds-full-results-solution.md +0 -230
|
@@ -0,0 +1,407 @@
|
|
|
1
|
+
# Security and Reliability Audit: `/feeds/:hash` Handler - Streaming Path
|
|
2
|
+
|
|
3
|
+
## Executive Summary
|
|
4
|
+
|
|
5
|
+
This document presents a comprehensive security and reliability audit of the `/feeds/:hash` handler in `src/http/router.ts`, specifically analyzing the execution path when the Accept header contains `application/x-jinaga-feed-stream`. The audit identifies critical security vulnerabilities, reliability issues, and provides detailed remediation strategies.
|
|
6
|
+
|
|
7
|
+
## Scope of Analysis
|
|
8
|
+
|
|
9
|
+
**Target Handler**: `/feeds/:hash` GET endpoint
|
|
10
|
+
**Execution Path**: `application/x-jinaga-feed-stream` Accept header flow
|
|
11
|
+
**Key Methods Analyzed**:
|
|
12
|
+
- `getOrStream()` (lines 38-92)
|
|
13
|
+
- `streamFeed()` (lines 468-509)
|
|
14
|
+
- `streamAllInitialResults()` (lines 511-568)
|
|
15
|
+
- `streamFeedResponse()` (lines 570-583)
|
|
16
|
+
|
|
17
|
+
## Critical Security Findings
|
|
18
|
+
|
|
19
|
+
### 1. Authentication Bypass Vulnerability
|
|
20
|
+
**CVSS Score: 9.1 (Critical)**
|
|
21
|
+
**CWE-287: Improper Authentication**
|
|
22
|
+
|
|
23
|
+
**Location**: Lines 43, 468-472
|
|
24
|
+
**Issue**: The handler accepts null users without explicit authentication validation.
|
|
25
|
+
|
|
26
|
+
```typescript
|
|
27
|
+
const user = <RequestUser>(req as any).user; // Line 43
|
|
28
|
+
// No authentication check before proceeding
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Proof of Concept**:
|
|
32
|
+
```bash
|
|
33
|
+
curl -H "Accept: application/x-jinaga-feed-stream" \
|
|
34
|
+
http://target/feeds/arbitrary_hash
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
**Business Impact**:
|
|
38
|
+
- Unauthorized access to sensitive feed data
|
|
39
|
+
- Potential data exfiltration
|
|
40
|
+
- Compliance violations (GDPR, HIPAA)
|
|
41
|
+
|
|
42
|
+
**Remediation**:
|
|
43
|
+
```typescript
|
|
44
|
+
// Add explicit authentication check
|
|
45
|
+
if (!user && requiresAuthentication(feedHash)) {
|
|
46
|
+
res.sendStatus(401);
|
|
47
|
+
return;
|
|
48
|
+
}
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### 2. Hash Parameter Injection Vulnerability
|
|
52
|
+
**CVSS Score: 8.2 (High)**
|
|
53
|
+
**CWE-20: Improper Input Validation**
|
|
54
|
+
|
|
55
|
+
**Location**: Lines 469, 441
|
|
56
|
+
**Issue**: Hash parameter lacks validation and sanitization.
|
|
57
|
+
|
|
58
|
+
```typescript
|
|
59
|
+
const feedHash = params["hash"]; // No validation
|
|
60
|
+
if (!feedHash) {
|
|
61
|
+
return null; // Only checks for existence
|
|
62
|
+
}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
**Attack Vectors**:
|
|
66
|
+
- Path traversal: `../../../sensitive_data`
|
|
67
|
+
- SQL injection via hash lookup
|
|
68
|
+
- Hash collision attacks
|
|
69
|
+
- Enumeration attacks
|
|
70
|
+
|
|
71
|
+
**Proof of Concept**:
|
|
72
|
+
```bash
|
|
73
|
+
# Path traversal attempt
|
|
74
|
+
curl -H "Accept: application/x-jinaga-feed-stream" \
|
|
75
|
+
"http://target/feeds/../../../etc/passwd"
|
|
76
|
+
|
|
77
|
+
# Hash enumeration
|
|
78
|
+
for i in {1..1000}; do
|
|
79
|
+
curl -s -o /dev/null -w "%{http_code}" \
|
|
80
|
+
"http://target/feeds/hash_$i" | grep -v 404
|
|
81
|
+
done
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
**Remediation**:
|
|
85
|
+
```typescript
|
|
86
|
+
// Add hash validation
|
|
87
|
+
const HASH_PATTERN = /^[a-fA-F0-9]{64}$/;
|
|
88
|
+
if (!feedHash || !HASH_PATTERN.test(feedHash)) {
|
|
89
|
+
res.sendStatus(400);
|
|
90
|
+
return;
|
|
91
|
+
}
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
### 3. Timing Attack Vulnerability
|
|
95
|
+
**CVSS Score: 6.5 (Medium)**
|
|
96
|
+
**CWE-208: Observable Timing Discrepancy**
|
|
97
|
+
|
|
98
|
+
**Location**: Lines 474-477
|
|
99
|
+
**Issue**: Feed lookup timing reveals hash existence.
|
|
100
|
+
|
|
101
|
+
```typescript
|
|
102
|
+
const feedDefinition = await this.feedCache.getFeed(feedHash);
|
|
103
|
+
if (!feedDefinition) {
|
|
104
|
+
return null; // Different timing for valid vs invalid hashes
|
|
105
|
+
}
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
**Remediation**:
|
|
109
|
+
```typescript
|
|
110
|
+
// Constant-time hash validation
|
|
111
|
+
const [feedDefinition, _] = await Promise.all([
|
|
112
|
+
this.feedCache.getFeed(feedHash),
|
|
113
|
+
this.addConstantTimeDelay()
|
|
114
|
+
]);
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
### 4. Denial of Service Vulnerabilities
|
|
118
|
+
**CVSS Score: 7.5 (High)**
|
|
119
|
+
**CWE-400: Uncontrolled Resource Consumption**
|
|
120
|
+
|
|
121
|
+
#### 4.1 Memory Exhaustion
|
|
122
|
+
**Location**: Lines 485-509, 511-568
|
|
123
|
+
**Issue**: Unbounded stream creation and memory usage.
|
|
124
|
+
|
|
125
|
+
```typescript
|
|
126
|
+
const stream = new Stream<FeedResponse>(); // No memory limits
|
|
127
|
+
// Continuous streaming without backpressure management
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
#### 4.2 Connection Exhaustion
|
|
131
|
+
**Location**: Lines 58-60, 61-63
|
|
132
|
+
**Issue**: Inadequate connection management.
|
|
133
|
+
|
|
134
|
+
```typescript
|
|
135
|
+
const timeout = setTimeout(() => {
|
|
136
|
+
response.close();
|
|
137
|
+
}, 5 * 60 * 1000); // 5-minute timeout may be too long
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
**Attack Scenario**:
|
|
141
|
+
```bash
|
|
142
|
+
# Connection exhaustion attack
|
|
143
|
+
for i in {1..10000}; do
|
|
144
|
+
curl -H "Accept: application/x-jinaga-feed-stream" \
|
|
145
|
+
"http://target/feeds/valid_hash" &
|
|
146
|
+
done
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
**Remediation**:
|
|
150
|
+
```typescript
|
|
151
|
+
// Add connection limits and backpressure
|
|
152
|
+
const MAX_CONCURRENT_STREAMS = 100;
|
|
153
|
+
const STREAM_TIMEOUT = 30000; // 30 seconds
|
|
154
|
+
const MAX_MEMORY_PER_STREAM = 10 * 1024 * 1024; // 10MB
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
### 5. Race Condition Vulnerabilities
|
|
158
|
+
**CVSS Score: 6.8 (Medium)**
|
|
159
|
+
**CWE-362: Concurrent Execution using Shared Resource**
|
|
160
|
+
|
|
161
|
+
**Location**: Lines 492-502, 523-565
|
|
162
|
+
**Issue**: Concurrent access to shared resources without proper synchronization.
|
|
163
|
+
|
|
164
|
+
```typescript
|
|
165
|
+
// Multiple async operations without proper coordination
|
|
166
|
+
const listeners = inverses.map(inverse => this.factManager.addSpecificationListener(
|
|
167
|
+
inverse.inverseSpecification,
|
|
168
|
+
async (results) => {
|
|
169
|
+
// Race condition: bookmark updates
|
|
170
|
+
bookmark = await this.streamFeedResponse(...);
|
|
171
|
+
}
|
|
172
|
+
));
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
### 6. Information Disclosure Vulnerabilities
|
|
176
|
+
**CVSS Score: 5.3 (Medium)**
|
|
177
|
+
**CWE-200: Information Exposure**
|
|
178
|
+
|
|
179
|
+
#### 6.1 Error Information Leakage
|
|
180
|
+
**Location**: Lines 722-738
|
|
181
|
+
**Issue**: Detailed error messages expose internal system information.
|
|
182
|
+
|
|
183
|
+
```typescript
|
|
184
|
+
res.status(500).send(error.message); // Exposes internal errors
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
#### 6.2 Timing-Based Information Disclosure
|
|
188
|
+
**Location**: Lines 523-565
|
|
189
|
+
**Issue**: Processing time reveals data characteristics.
|
|
190
|
+
|
|
191
|
+
## Reliability and Performance Issues
|
|
192
|
+
|
|
193
|
+
### 1. Infinite Loop Vulnerability
|
|
194
|
+
**Severity: High**
|
|
195
|
+
**Location**: Lines 523-565
|
|
196
|
+
|
|
197
|
+
```typescript
|
|
198
|
+
while (hasMoreResults && pageCount < maxPages) {
|
|
199
|
+
// Potential infinite loop if bookmark doesn't change
|
|
200
|
+
if (newBookmark === bookmark) {
|
|
201
|
+
hasMoreResults = false; // Mitigation exists but insufficient
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
**Issue**: Insufficient loop termination conditions.
|
|
207
|
+
|
|
208
|
+
### 2. Resource Leak Vulnerabilities
|
|
209
|
+
**Severity: High**
|
|
210
|
+
**Location**: Lines 503-507
|
|
211
|
+
|
|
212
|
+
```typescript
|
|
213
|
+
stream.done(() => {
|
|
214
|
+
for (const listener of listeners) {
|
|
215
|
+
this.factManager.removeSpecificationListener(listener);
|
|
216
|
+
}
|
|
217
|
+
});
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
**Issue**: Listeners may not be cleaned up if stream.done() is not called.
|
|
221
|
+
|
|
222
|
+
### 3. Database Connection Pool Exhaustion
|
|
223
|
+
**Severity: Medium**
|
|
224
|
+
**Location**: Lines 524, 571
|
|
225
|
+
|
|
226
|
+
Multiple concurrent database calls without connection pooling limits.
|
|
227
|
+
|
|
228
|
+
## Detailed Remediation Strategy
|
|
229
|
+
|
|
230
|
+
### Priority 1: Critical Security Fixes
|
|
231
|
+
|
|
232
|
+
1. **Implement Authentication Validation**
|
|
233
|
+
```typescript
|
|
234
|
+
// Add to getOrStream function
|
|
235
|
+
if (requiresAuth && !user) {
|
|
236
|
+
res.sendStatus(401);
|
|
237
|
+
return;
|
|
238
|
+
}
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
2. **Add Hash Parameter Validation**
|
|
242
|
+
```typescript
|
|
243
|
+
const validateHash = (hash: string): boolean => {
|
|
244
|
+
return /^[a-fA-F0-9]{64}$/.test(hash) && hash.length === 64;
|
|
245
|
+
};
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
3. **Implement Rate Limiting**
|
|
249
|
+
```typescript
|
|
250
|
+
const rateLimiter = rateLimit({
|
|
251
|
+
windowMs: 15 * 60 * 1000, // 15 minutes
|
|
252
|
+
max: 100 // limit each IP to 100 requests per windowMs
|
|
253
|
+
});
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
### Priority 2: DoS Prevention
|
|
257
|
+
|
|
258
|
+
1. **Connection Management**
|
|
259
|
+
```typescript
|
|
260
|
+
const connectionManager = new ConnectionManager({
|
|
261
|
+
maxConcurrentStreams: 100,
|
|
262
|
+
maxStreamDuration: 30000,
|
|
263
|
+
maxMemoryPerStream: 10 * 1024 * 1024
|
|
264
|
+
});
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
2. **Backpressure Implementation**
|
|
268
|
+
```typescript
|
|
269
|
+
class BackpressureStream<T> extends Stream<T> {
|
|
270
|
+
private buffer: T[] = [];
|
|
271
|
+
private maxBufferSize = 1000;
|
|
272
|
+
|
|
273
|
+
feed(data: T) {
|
|
274
|
+
if (this.buffer.length >= this.maxBufferSize) {
|
|
275
|
+
this.pause();
|
|
276
|
+
}
|
|
277
|
+
super.feed(data);
|
|
278
|
+
}
|
|
279
|
+
}
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
### Priority 3: Monitoring and Logging
|
|
283
|
+
|
|
284
|
+
1. **Security Event Logging**
|
|
285
|
+
```typescript
|
|
286
|
+
const securityLogger = {
|
|
287
|
+
logSuspiciousActivity: (event: SecurityEvent) => {
|
|
288
|
+
// Log to SIEM system
|
|
289
|
+
},
|
|
290
|
+
logAccessAttempt: (user: string, resource: string, success: boolean) => {
|
|
291
|
+
// Audit trail
|
|
292
|
+
}
|
|
293
|
+
};
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
2. **Performance Monitoring**
|
|
297
|
+
```typescript
|
|
298
|
+
const performanceMonitor = {
|
|
299
|
+
trackStreamDuration: (streamId: string, duration: number) => {},
|
|
300
|
+
trackMemoryUsage: (streamId: string, usage: number) => {},
|
|
301
|
+
trackConcurrentConnections: (count: number) => {}
|
|
302
|
+
};
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
## Supply Chain Security Analysis
|
|
306
|
+
|
|
307
|
+
### Dependencies Review
|
|
308
|
+
- **express**: Check for known CVEs in routing
|
|
309
|
+
- **jinaga**: Internal dependency - requires separate audit
|
|
310
|
+
- **Stream class**: Custom implementation needs security review
|
|
311
|
+
|
|
312
|
+
### Recommended Actions
|
|
313
|
+
1. Implement dependency scanning in CI/CD
|
|
314
|
+
2. Regular security updates
|
|
315
|
+
3. Vendor security assessment for jinaga library
|
|
316
|
+
|
|
317
|
+
## Compliance Considerations
|
|
318
|
+
|
|
319
|
+
### GDPR Compliance
|
|
320
|
+
- Data minimization in streams
|
|
321
|
+
- Right to erasure implementation
|
|
322
|
+
- Consent management for streaming data
|
|
323
|
+
|
|
324
|
+
### SOC 2 Compliance
|
|
325
|
+
- Access logging and monitoring
|
|
326
|
+
- Data encryption in transit
|
|
327
|
+
- Incident response procedures
|
|
328
|
+
|
|
329
|
+
## Testing Strategy
|
|
330
|
+
|
|
331
|
+
### Security Testing
|
|
332
|
+
1. **Penetration Testing**
|
|
333
|
+
- Authentication bypass attempts
|
|
334
|
+
- Input validation testing
|
|
335
|
+
- DoS attack simulation
|
|
336
|
+
|
|
337
|
+
2. **Automated Security Scanning**
|
|
338
|
+
- SAST tools integration
|
|
339
|
+
- Dependency vulnerability scanning
|
|
340
|
+
- Runtime security monitoring
|
|
341
|
+
|
|
342
|
+
### Performance Testing
|
|
343
|
+
1. **Load Testing**
|
|
344
|
+
- Concurrent stream handling
|
|
345
|
+
- Memory usage under load
|
|
346
|
+
- Database connection pooling
|
|
347
|
+
|
|
348
|
+
2. **Chaos Engineering**
|
|
349
|
+
- Network partition simulation
|
|
350
|
+
- Database failure scenarios
|
|
351
|
+
- Memory pressure testing
|
|
352
|
+
|
|
353
|
+
## Implementation Timeline
|
|
354
|
+
|
|
355
|
+
### Phase 1 (Immediate - 1 week)
|
|
356
|
+
- Critical authentication fixes
|
|
357
|
+
- Input validation implementation
|
|
358
|
+
- Basic rate limiting
|
|
359
|
+
|
|
360
|
+
### Phase 2 (Short-term - 2-4 weeks)
|
|
361
|
+
- DoS prevention measures
|
|
362
|
+
- Enhanced monitoring
|
|
363
|
+
- Security logging
|
|
364
|
+
|
|
365
|
+
### Phase 3 (Medium-term - 1-3 months)
|
|
366
|
+
- Comprehensive testing
|
|
367
|
+
- Performance optimization
|
|
368
|
+
- Compliance implementation
|
|
369
|
+
|
|
370
|
+
## Conclusion
|
|
371
|
+
|
|
372
|
+
The `/feeds/:hash` handler contains multiple critical security vulnerabilities that require immediate attention. The streaming functionality, while providing real-time capabilities, introduces significant attack surface that must be properly secured. Implementation of the recommended remediation strategies will significantly improve the security posture and reliability of the system.
|
|
373
|
+
|
|
374
|
+
**Risk Assessment**: **HIGH** - Immediate action required
|
|
375
|
+
**Business Impact**: **CRITICAL** - Potential for data breach and service disruption
|
|
376
|
+
**Remediation Complexity**: **MEDIUM** - Requires coordinated development effort
|
|
377
|
+
|
|
378
|
+
## Appendix A: Security Checklist
|
|
379
|
+
|
|
380
|
+
- [ ] Authentication validation implemented
|
|
381
|
+
- [ ] Input sanitization and validation
|
|
382
|
+
- [ ] Rate limiting configured
|
|
383
|
+
- [ ] Connection management implemented
|
|
384
|
+
- [ ] Error handling sanitized
|
|
385
|
+
- [ ] Logging and monitoring deployed
|
|
386
|
+
- [ ] Security testing completed
|
|
387
|
+
- [ ] Performance testing completed
|
|
388
|
+
- [ ] Documentation updated
|
|
389
|
+
- [ ] Team training conducted
|
|
390
|
+
|
|
391
|
+
## Appendix B: Monitoring Queries
|
|
392
|
+
|
|
393
|
+
```sql
|
|
394
|
+
-- Suspicious hash access patterns
|
|
395
|
+
SELECT hash, COUNT(*), MIN(timestamp), MAX(timestamp)
|
|
396
|
+
FROM feed_access_log
|
|
397
|
+
WHERE timestamp > NOW() - INTERVAL '1 hour'
|
|
398
|
+
GROUP BY hash
|
|
399
|
+
HAVING COUNT(*) > 100;
|
|
400
|
+
|
|
401
|
+
-- Failed authentication attempts
|
|
402
|
+
SELECT ip_address, COUNT(*)
|
|
403
|
+
FROM security_events
|
|
404
|
+
WHERE event_type = 'auth_failure'
|
|
405
|
+
AND timestamp > NOW() - INTERVAL '1 hour'
|
|
406
|
+
GROUP BY ip_address
|
|
407
|
+
HAVING COUNT(*) > 10;
|
|
@@ -0,0 +1,294 @@
|
|
|
1
|
+
# Feeds: Persistent Client Proxy with Waitlist
|
|
2
|
+
Author: @michaellperry
|
|
3
|
+
Status: Draft
|
|
4
|
+
Scope: `/feeds/:hash` streaming path (`Accept: application/x-jinaga-feed-stream`)
|
|
5
|
+
|
|
6
|
+
## Summary
|
|
7
|
+
|
|
8
|
+
Introduce a per-connection "FeedStreamSession" (client proxy) that:
|
|
9
|
+
- Maintains a waitlist of fact references indicated by inverse observers.
|
|
10
|
+
- Serializes all fetching and streaming through a single "query cycle" loop driven by the current bookmark.
|
|
11
|
+
- Continues initial pagination until exhaustion, then transitions to real-time.
|
|
12
|
+
- Emits an explicit "caught up" signal (SYNC) without changing the response schema (empty `references` array + current `bookmark`).
|
|
13
|
+
|
|
14
|
+
This addresses race conditions between listener callbacks and bookmark mutation, ensures tuple completeness, and guarantees no missed updates when facts arrive between pages.
|
|
15
|
+
|
|
16
|
+
## Goals
|
|
17
|
+
|
|
18
|
+
- Deliver all historical feed results before push-only mode.
|
|
19
|
+
- Prevent lost updates during the transition to "caught up".
|
|
20
|
+
- Maintain tuple completeness ("projection is the tuple; send all facts in tuple").
|
|
21
|
+
- Preserve response shape and backwards compatibility.
|
|
22
|
+
- Provide predictable lifecycle and resource usage per stream.
|
|
23
|
+
|
|
24
|
+
## Non-goals
|
|
25
|
+
|
|
26
|
+
- Changing the `/feeds/:hash` response format.
|
|
27
|
+
- Introducing server-sent events or WebSockets (keep current HTTP streaming).
|
|
28
|
+
- Redesigning the authorization/feed query pipeline.
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Design
|
|
33
|
+
|
|
34
|
+
### High-level flow
|
|
35
|
+
|
|
36
|
+
1. On connection, create a `FeedStreamSession` with:
|
|
37
|
+
- `bookmark` (initial from query param `b` or empty),
|
|
38
|
+
- `waitlist` (Set of fact refs by hash/type),
|
|
39
|
+
- `activeCycle` (boolean),
|
|
40
|
+
- `shuttingDown` (boolean),
|
|
41
|
+
- `stream` (Stream<FeedResponse>).
|
|
42
|
+
|
|
43
|
+
2. Start inverse observers immediately and push matching fact refs into the waitlist.
|
|
44
|
+
- Observers do not fetch/stream themselves.
|
|
45
|
+
- If no cycle is active, they signal to start (or continue) the cycle.
|
|
46
|
+
|
|
47
|
+
3. Run the query cycle:
|
|
48
|
+
- While true:
|
|
49
|
+
- Call `authorization.feed(user, feed, start, bookmark)`.
|
|
50
|
+
- If tuples returned:
|
|
51
|
+
- Deduplicate facts across tuples in the page.
|
|
52
|
+
- Stream a frame `{ references, bookmark: results.bookmark }`.
|
|
53
|
+
- Remove any queued facts from the waitlist that are included in streamed tuples.
|
|
54
|
+
- Update `bookmark`.
|
|
55
|
+
- Continue.
|
|
56
|
+
- If no tuples returned:
|
|
57
|
+
- If `waitlist` is empty:
|
|
58
|
+
- Stream a SYNC frame with empty `references` and current `bookmark` (backwards compatible).
|
|
59
|
+
- End the cycle.
|
|
60
|
+
- Else:
|
|
61
|
+
- Facts arrived between last page and now. Continue the loop (run feed again) to collect their full tuples.
|
|
62
|
+
|
|
63
|
+
4. On stream close, remove all observers and mark `shuttingDown` to prevent new cycles.
|
|
64
|
+
|
|
65
|
+
### Class responsibilities
|
|
66
|
+
|
|
67
|
+
- FeedStreamSession
|
|
68
|
+
- Owns lifecycle and state for a connection.
|
|
69
|
+
- Public methods:
|
|
70
|
+
- `start()` => sets observers and kicks off initial paginated backfill to exhaustion.
|
|
71
|
+
- `onInverseMatch(factRefs[])` => add to waitlist; `startCycleIfIdle()`.
|
|
72
|
+
- `done()` => unregister observers; cancel cycle on stream close.
|
|
73
|
+
- Private:
|
|
74
|
+
- `runCycle()` => serializes fetch/stream; guarded by `activeCycle`.
|
|
75
|
+
- `streamPage(results)` => compute deduped references and stream with bookmark.
|
|
76
|
+
- `streamSync()` => stream `{ references: [], bookmark }`.
|
|
77
|
+
- `drainWaitlistIfNeeded()` => loop continues until `no tuples && waitlist empty`.
|
|
78
|
+
|
|
79
|
+
- Router integration (src/http/router.ts)
|
|
80
|
+
- Construct `FeedStreamSession` in the `/feeds/:hash` handler.
|
|
81
|
+
- Replace direct "listener => streamFeedResponse" pattern with session waitlist notifications.
|
|
82
|
+
|
|
83
|
+
### Waitlist semantics
|
|
84
|
+
|
|
85
|
+
- Waitlist keyed by `{hash,type}` ensures no duplicates.
|
|
86
|
+
- Removal happens only when the tuple containing the fact is streamed (preserves tuple completeness).
|
|
87
|
+
- Waitlist is a trigger to keep querying; we do not stream waitlist facts directly.
|
|
88
|
+
|
|
89
|
+
### SYNC signaling
|
|
90
|
+
|
|
91
|
+
- Preserve response schema:
|
|
92
|
+
- Emit SYNC as a frame with `references: []` and current `bookmark`.
|
|
93
|
+
- Clients that need an explicit signal can treat empty `references` as "caught up".
|
|
94
|
+
|
|
95
|
+
### Concurrency and races
|
|
96
|
+
|
|
97
|
+
- Single-cycle execution: `activeCycle` flag prevents concurrent runs.
|
|
98
|
+
- Listener callbacks never mutate `bookmark` and never fetch/stream; they only enqueue refs and wake the cycle.
|
|
99
|
+
- Bookmark progresses monotonically within the cycle; no interleaving updates.
|
|
100
|
+
|
|
101
|
+
### Backpressure and limits
|
|
102
|
+
|
|
103
|
+
- Add conservative limits (configurable):
|
|
104
|
+
- `maxInitialPages` (default 1000)
|
|
105
|
+
- `maxWaitlistSize` (e.g., 50k refs) -> if exceeded, drop connection with 503 and log.
|
|
106
|
+
- Optional small delay every N pages (e.g., 10ms every 10 pages).
|
|
107
|
+
- Consider extending Stream to a "BackpressureStream" in a follow-up to cap buffer length.
|
|
108
|
+
|
|
109
|
+
### Telemetry
|
|
110
|
+
|
|
111
|
+
- Per-stream diagnostic log context (feed hash, connection id).
|
|
112
|
+
- Metrics:
|
|
113
|
+
- initial pages fetched, total references streamed, cycle count per connection,
|
|
114
|
+
- time to SYNC,
|
|
115
|
+
- waitlist high-water mark,
|
|
116
|
+
- dropped due to limits.
|
|
117
|
+
|
|
118
|
+
---
|
|
119
|
+
|
|
120
|
+
## Integration Points
|
|
121
|
+
|
|
122
|
+
- src/http/router.ts
|
|
123
|
+
- `/feeds/:hash` handler (Accept: `application/x-jinaga-feed-stream`).
|
|
124
|
+
- Replace direct `streamFeed` flow by constructing `FeedStreamSession`.
|
|
125
|
+
|
|
126
|
+
- Authorization layer
|
|
127
|
+
- `authorization.feed(userIdentity, feed, start, bookmark)` pagination contract.
|
|
128
|
+
|
|
129
|
+
- Feed definition and inversion
|
|
130
|
+
- `feedCache.getFeed(feedHash)`
|
|
131
|
+
- `invertSpecification(feedDefinition.feed)`
|
|
132
|
+
- `computeTupleSubsetHash` and `computeObjectHash` for filtering listener matches.
|
|
133
|
+
|
|
134
|
+
- FactManager
|
|
135
|
+
- `addSpecificationListener` / `removeSpecificationListener` for inverse triggers.
|
|
136
|
+
|
|
137
|
+
- Stream
|
|
138
|
+
- `Stream<FeedResponse>` feeding and lifecycle (done/close).
|
|
139
|
+
|
|
140
|
+
- Configuration
|
|
141
|
+
- Add server config for page limits, waitlist cap, optional small delays.
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
## Dependencies and Assumptions
|
|
146
|
+
|
|
147
|
+
- Existing bookmark semantics remain stable and monotonic.
|
|
148
|
+
- `authorization.feed` returns `tuples` and `bookmark`, where `tuples.length < pageSize` implies near exhaustion.
|
|
149
|
+
- The test client and any existing consumers tolerate empty `references` frames.
|
|
150
|
+
|
|
151
|
+
---
|
|
152
|
+
|
|
153
|
+
## Acceptance Criteria
|
|
154
|
+
|
|
155
|
+
Functional correctness
|
|
156
|
+
- Initial backfill:
|
|
157
|
+
- The server streams all historical pages until the feed is exhausted (no tuples left) before transitioning to real-time.
|
|
158
|
+
- Bookmark advances monotonically across frames.
|
|
159
|
+
- Real-time updates:
|
|
160
|
+
- Facts arriving during initial pagination are not lost; the session continues querying until their tuples are streamed.
|
|
161
|
+
- Facts arriving after SYNC result in subsequent frames with their tuples, preserving order by bookmark.
|
|
162
|
+
- Tuple completeness and deduplication:
|
|
163
|
+
- Each frame's `references` contains the deduplicated set of facts from all tuples in that chunk.
|
|
164
|
+
- Waitlist items are removed only when their containing tuples are streamed.
|
|
165
|
+
- SYNC indication:
|
|
166
|
+
- On catching up (no tuples and empty waitlist), server emits a frame with `references: []` and current `bookmark`.
|
|
167
|
+
- Lifecycle:
|
|
168
|
+
- On connection close, inverse listeners are unregistered and the cycle stops.
|
|
169
|
+
- No concurrent cycles run per connection.
|
|
170
|
+
|
|
171
|
+
Performance and robustness
|
|
172
|
+
- Backfill eventually completes for large feeds within configured safety limits (e.g., 1000 pages).
|
|
173
|
+
- Waitlist does not grow unbounded; if exceeding configured cap, the server closes the stream gracefully with diagnostics.
|
|
174
|
+
- Under concurrent connections, memory and CPU remain within acceptable bounds; no deadlocks or starvation.
|
|
175
|
+
|
|
176
|
+
Security and reliability
|
|
177
|
+
- No race conditions update the bookmark concurrently.
|
|
178
|
+
- No unbounded buffers in the stream path.
|
|
179
|
+
- Listeners are always removed when streams end.
|
|
180
|
+
|
|
181
|
+
Observability
|
|
182
|
+
- Logs/metrics expose: total frames, initial pages, time to SYNC, cycle restarts due to waitlist, waitlist size peaks.
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
|
|
186
|
+
## Test Plan
|
|
187
|
+
|
|
188
|
+
### Unit tests (Jest)
|
|
189
|
+
|
|
190
|
+
FeedStreamSession
|
|
191
|
+
- "Backfill to exhaustion"
|
|
192
|
+
- Given multiple pages returned by `authorization.feed`, verify frames streamed for each page, bookmark progression, and final SYNC frame (empty `references`).
|
|
193
|
+
- "Waitlist triggers continuation"
|
|
194
|
+
- With a final page empty, but waitlist non-empty (push facts into waitlist between pages), verify the cycle continues and fetches at least one more page, then SYNCs when waitlist clears.
|
|
195
|
+
- "No concurrent cycles"
|
|
196
|
+
- Rapidly enqueue from listener multiple times; assert only one `authorization.feed` loop runs at a time (use mocks and counters).
|
|
197
|
+
- "Deduplication"
|
|
198
|
+
- Given overlapping facts across tuples in a page, confirm only unique `{hash,type}` appear in `references`.
|
|
199
|
+
- "Tuple completeness"
|
|
200
|
+
- Verify waitlist facts are only removed when their tuples appear in a streamed page.
|
|
201
|
+
- "SYNC emission"
|
|
202
|
+
- Assert SYNC frame emitted (empty `references`) exactly when both: last query returned no tuples and waitlist is empty.
|
|
203
|
+
- "Bookmark monotonicity"
|
|
204
|
+
- Enforce that sent bookmarks are strictly progressing or stable only for SYNC frames.
|
|
205
|
+
- "Listener filtering"
|
|
206
|
+
- Only enqueue facts from inverse results that match `givenSubset` (use `computeTupleSubsetHash` matching).
|
|
207
|
+
- "Lifecycle cleanup"
|
|
208
|
+
- On `stream.done()`, verify removal of all listeners and no further cycle runs.
|
|
209
|
+
- "Limits"
|
|
210
|
+
- If `waitlist` exceeds cap, session terminates gracefully (mock log) and removes listeners.
|
|
211
|
+
|
|
212
|
+
Router integration
|
|
213
|
+
- "Session wiring"
|
|
214
|
+
- `/feeds/:hash` with stream accept header constructs session, calls `start()`, and proxies `stream.feed` to response writer.
|
|
215
|
+
|
|
216
|
+
### Integration tests (Jest + supertest or Node http)
|
|
217
|
+
|
|
218
|
+
- "Complete historical stream"
|
|
219
|
+
- Seed DB with > pageSize tuples (e.g., 350) matching the feed. Connect and assert received multiple frames covering all facts, then SYNC.
|
|
220
|
+
- "Race during backfill"
|
|
221
|
+
- Start stream, insert facts while initial pagination runs. Assert no missed updates; frames include these facts by the time SYNC arrives.
|
|
222
|
+
- "Post-SYNC updates"
|
|
223
|
+
- After SYNC, insert new facts. Assert subsequent frames deliver them with updated bookmark.
|
|
224
|
+
- "Multiple concurrent clients"
|
|
225
|
+
- Spin up N clients (e.g., 20). Assert similar message counts and time-to-sync distribution; no client stalls. Reuse `test-concurrent-streams.js` patterns (or convert to automated test).
|
|
226
|
+
- "Disconnect handling"
|
|
227
|
+
- Client disconnect mid-backfill. Assert listeners removed and no residual cycle iterations.
|
|
228
|
+
- "Waitlist stress"
|
|
229
|
+
- Burst insert causing many inverse triggers quickly. Validate waitlist remains under cap or stream terminates with diagnostics.
|
|
230
|
+
- "Timeout/long feeds"
|
|
231
|
+
- Emulate large dataset to ensure the process respects configured page limits and still reaches SYNC or terminates cleanly.
|
|
232
|
+
|
|
233
|
+
Telemetry validation
|
|
234
|
+
- Expose counters to a test sink; assert metrics present (initial pages, cycles, SYNC count).
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## Rollout
|
|
239
|
+
|
|
240
|
+
- Feature flag: `feeds.fullInitialPagination` (default enabled if prior behavior already documented; if not, roll out gradually).
|
|
241
|
+
- Config options:
|
|
242
|
+
- `feeds.maxInitialPages` (default 1000)
|
|
243
|
+
- `feeds.waitlistCap` (default 50,000)
|
|
244
|
+
- `feeds.pageDelayEvery` (default 10 pages => 10ms delay)
|
|
245
|
+
- Logging: scoped per-connection id and feed hash, with sampling if needed.
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
## Implementation Steps
|
|
250
|
+
|
|
251
|
+
1. Create `FeedStreamSession` in `src/feeds/FeedStreamSession.ts`.
|
|
252
|
+
- Constructor: dependencies (`authorization`, `factManager`, `feedDefinition`, `userIdentity`, `start`, `stream`, `initialBookmark`, config).
|
|
253
|
+
- Methods: `start()`, `runCycle()`, `onInverseMatch()`, `streamPage()`, `streamSync()`, `dispose()`.
|
|
254
|
+
|
|
255
|
+
2. Update `/feeds/:hash` handler in `src/http/router.ts`.
|
|
256
|
+
- Where `streamFeed` currently instantiates `Stream<FeedResponse>` and registers listeners, replace with session creation and `start()`.
|
|
257
|
+
- On `stream.done()`, call `session.dispose()`.
|
|
258
|
+
|
|
259
|
+
3. Move/Reuse helper logic:
|
|
260
|
+
- Deduping facts per page.
|
|
261
|
+
- Inverse computation and given subset filtering.
|
|
262
|
+
- Bookmark management.
|
|
263
|
+
|
|
264
|
+
4. Add configuration and defaults.
|
|
265
|
+
|
|
266
|
+
5. Add tests:
|
|
267
|
+
- Unit tests for session (mocks for `authorization`, `factManager`, `stream`).
|
|
268
|
+
- Integration tests using real HTTP and seeded DB.
|
|
269
|
+
- Optionally adapt `test-concurrent-streams.js` into a CI test harness or keep as manual tool.
|
|
270
|
+
|
|
271
|
+
6. Add telemetry and structured logs.
|
|
272
|
+
|
|
273
|
+
7. Documentation:
|
|
274
|
+
- Update `documentation/streaming-feeds-full-results-solution.md` to reference the session model and SYNC semantics.
|
|
275
|
+
- Add this design document.
|
|
276
|
+
|
|
277
|
+
---
|
|
278
|
+
|
|
279
|
+
## Risks and Mitigations
|
|
280
|
+
|
|
281
|
+
- Risk: Unbounded memory due to large waitlists or slow clients.
|
|
282
|
+
- Mitigate: Waitlist caps, future backpressure stream implementation, connection duration limits.
|
|
283
|
+
- Risk: Deadlock or missed wake-ups.
|
|
284
|
+
- Mitigate: Simple state machine (`activeCycle`, `needsCycle`) and thorough unit tests for concurrency.
|
|
285
|
+
- Risk: Client incompatibility with empty `references` frames.
|
|
286
|
+
- Mitigate: Already valid JSON per existing schema; integration tests against existing client.
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## Open Questions
|
|
291
|
+
|
|
292
|
+
- Should SYNC frames be optional behind a config gate?
|
|
293
|
+
- Do we need a small debounce before emitting SYNC to reduce "thrash" under constant writes?
|
|
294
|
+
- Do we need connection-level backpressure limits immediately, or as a follow-up (as per security audit notes)?
|