jinaga-server 3.5.1 → 3.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (64) hide show
  1. package/dist/authorization/authorization-keystore.js +2 -2
  2. package/dist/authorization/authorization-keystore.js.map +1 -1
  3. package/dist/http/csv-metadata.d.ts +24 -0
  4. package/dist/http/csv-metadata.js +3 -0
  5. package/dist/http/csv-metadata.js.map +1 -0
  6. package/dist/http/csv-validator.d.ts +23 -0
  7. package/dist/http/csv-validator.js +83 -0
  8. package/dist/http/csv-validator.js.map +1 -0
  9. package/dist/http/output-formatters.d.ts +28 -0
  10. package/dist/http/output-formatters.js +197 -0
  11. package/dist/http/output-formatters.js.map +1 -0
  12. package/dist/http/result-stream.d.ts +35 -0
  13. package/dist/http/result-stream.js +78 -0
  14. package/dist/http/result-stream.js.map +1 -0
  15. package/dist/http/router.d.ts +7 -1
  16. package/dist/http/router.js +276 -70
  17. package/dist/http/router.js.map +1 -1
  18. package/dist/http/stream.d.ts +3 -0
  19. package/dist/http/stream.js +79 -5
  20. package/dist/http/stream.js.map +1 -1
  21. package/dist/postgres/postgres-store.js +7 -5
  22. package/dist/postgres/postgres-store.js.map +1 -1
  23. package/dist/postgres/purge-sql.js +15 -2
  24. package/dist/postgres/purge-sql.js.map +1 -1
  25. package/dist/postgres/specification-result-sql.d.ts +2 -0
  26. package/dist/postgres/specification-result-sql.js +49 -5
  27. package/dist/postgres/specification-result-sql.js.map +1 -1
  28. package/dist/postgres/specification-sql.js +28 -6
  29. package/dist/postgres/specification-sql.js.map +1 -1
  30. package/docs/analysis/DIAGNOSTIC_LOGGING_README.md +201 -0
  31. package/docs/analysis/security-audit-feeds-hash-handler.md +407 -0
  32. package/docs/design/feeds-persistent-proxy-plan.md +294 -0
  33. package/integration-test/integration-test/jinaga-test/package-lock.json +6 -0
  34. package/integration-test/jinaga-test/Dockerfile +1 -6
  35. package/integration-test/jinaga-test/http-test-helpers.js +77 -0
  36. package/integration-test/jinaga-test/jinaga.test.js +9 -9
  37. package/integration-test/jinaga-test/package-lock.json +329 -32
  38. package/integration-test/jinaga-test/package.json +5 -1
  39. package/integration-test/jinaga-test/read-csv.test.js +330 -0
  40. package/integration-test/jinaga-test/read-endpoint.test.js +442 -0
  41. package/integration-test/jinaga-test/read-performance.test.js +221 -0
  42. package/integration-test/jinaga-test/read-streaming.test.js +199 -0
  43. package/integration-test/jinaga-test/specification-conditions.test.js +147 -0
  44. package/integration-test/jinaga-test/test-helpers.js +12 -0
  45. package/integration-test/jinaga-test/time-projection.test.js +235 -0
  46. package/package.json +3 -2
  47. package/src/authorization/authorization-keystore.ts +4 -4
  48. package/src/http/csv-metadata.ts +34 -0
  49. package/src/http/csv-validator.ts +90 -0
  50. package/src/http/output-formatters.ts +196 -0
  51. package/src/http/result-stream.ts +69 -0
  52. package/src/http/router.ts +344 -62
  53. package/src/http/stream.ts +97 -5
  54. package/src/postgres/postgres-store.ts +13 -6
  55. package/src/postgres/purge-sql.ts +18 -2
  56. package/src/postgres/specification-result-sql.ts +70 -8
  57. package/src/postgres/specification-sql.ts +36 -8
  58. package/test/http/csv-validatorSpec.ts +205 -0
  59. package/test/http/result-streamSpec.ts +151 -0
  60. package/test/postgres/postgresReadSpec.ts +192 -14
  61. package/test/postgres/postgresSpecificationSpec.ts +3 -3
  62. package/test/postgres/specificationConditionsSpec.ts +268 -0
  63. package/test-concurrent-streams.js +219 -0
  64. package/documentation/streaming-feeds-full-results-solution.md +0 -230
@@ -0,0 +1,407 @@
1
+ # Security and Reliability Audit: `/feeds/:hash` Handler - Streaming Path
2
+
3
+ ## Executive Summary
4
+
5
+ This document presents a comprehensive security and reliability audit of the `/feeds/:hash` handler in `src/http/router.ts`, specifically analyzing the execution path when the Accept header contains `application/x-jinaga-feed-stream`. The audit identifies critical security vulnerabilities, reliability issues, and provides detailed remediation strategies.
6
+
7
+ ## Scope of Analysis
8
+
9
+ **Target Handler**: `/feeds/:hash` GET endpoint
10
+ **Execution Path**: `application/x-jinaga-feed-stream` Accept header flow
11
+ **Key Methods Analyzed**:
12
+ - `getOrStream()` (lines 38-92)
13
+ - `streamFeed()` (lines 468-509)
14
+ - `streamAllInitialResults()` (lines 511-568)
15
+ - `streamFeedResponse()` (lines 570-583)
16
+
17
+ ## Critical Security Findings
18
+
19
+ ### 1. Authentication Bypass Vulnerability
20
+ **CVSS Score: 9.1 (Critical)**
21
+ **CWE-287: Improper Authentication**
22
+
23
+ **Location**: Lines 43, 468-472
24
+ **Issue**: The handler accepts null users without explicit authentication validation.
25
+
26
+ ```typescript
27
+ const user = <RequestUser>(req as any).user; // Line 43
28
+ // No authentication check before proceeding
29
+ ```
30
+
31
+ **Proof of Concept**:
32
+ ```bash
33
+ curl -H "Accept: application/x-jinaga-feed-stream" \
34
+ http://target/feeds/arbitrary_hash
35
+ ```
36
+
37
+ **Business Impact**:
38
+ - Unauthorized access to sensitive feed data
39
+ - Potential data exfiltration
40
+ - Compliance violations (GDPR, HIPAA)
41
+
42
+ **Remediation**:
43
+ ```typescript
44
+ // Add explicit authentication check
45
+ if (!user && requiresAuthentication(feedHash)) {
46
+ res.sendStatus(401);
47
+ return;
48
+ }
49
+ ```
50
+
51
+ ### 2. Hash Parameter Injection Vulnerability
52
+ **CVSS Score: 8.2 (High)**
53
+ **CWE-20: Improper Input Validation**
54
+
55
+ **Location**: Lines 469, 441
56
+ **Issue**: Hash parameter lacks validation and sanitization.
57
+
58
+ ```typescript
59
+ const feedHash = params["hash"]; // No validation
60
+ if (!feedHash) {
61
+ return null; // Only checks for existence
62
+ }
63
+ ```
64
+
65
+ **Attack Vectors**:
66
+ - Path traversal: `../../../sensitive_data`
67
+ - SQL injection via hash lookup
68
+ - Hash collision attacks
69
+ - Enumeration attacks
70
+
71
+ **Proof of Concept**:
72
+ ```bash
73
+ # Path traversal attempt
74
+ curl -H "Accept: application/x-jinaga-feed-stream" \
75
+ "http://target/feeds/../../../etc/passwd"
76
+
77
+ # Hash enumeration
78
+ for i in {1..1000}; do
79
+ curl -s -o /dev/null -w "%{http_code}" \
80
+ "http://target/feeds/hash_$i" | grep -v 404
81
+ done
82
+ ```
83
+
84
+ **Remediation**:
85
+ ```typescript
86
+ // Add hash validation
87
+ const HASH_PATTERN = /^[a-fA-F0-9]{64}$/;
88
+ if (!feedHash || !HASH_PATTERN.test(feedHash)) {
89
+ res.sendStatus(400);
90
+ return;
91
+ }
92
+ ```
93
+
94
+ ### 3. Timing Attack Vulnerability
95
+ **CVSS Score: 6.5 (Medium)**
96
+ **CWE-208: Observable Timing Discrepancy**
97
+
98
+ **Location**: Lines 474-477
99
+ **Issue**: Feed lookup timing reveals hash existence.
100
+
101
+ ```typescript
102
+ const feedDefinition = await this.feedCache.getFeed(feedHash);
103
+ if (!feedDefinition) {
104
+ return null; // Different timing for valid vs invalid hashes
105
+ }
106
+ ```
107
+
108
+ **Remediation**:
109
+ ```typescript
110
+ // Constant-time hash validation
111
+ const [feedDefinition, _] = await Promise.all([
112
+ this.feedCache.getFeed(feedHash),
113
+ this.addConstantTimeDelay()
114
+ ]);
115
+ ```
116
+
117
+ ### 4. Denial of Service Vulnerabilities
118
+ **CVSS Score: 7.5 (High)**
119
+ **CWE-400: Uncontrolled Resource Consumption**
120
+
121
+ #### 4.1 Memory Exhaustion
122
+ **Location**: Lines 485-509, 511-568
123
+ **Issue**: Unbounded stream creation and memory usage.
124
+
125
+ ```typescript
126
+ const stream = new Stream<FeedResponse>(); // No memory limits
127
+ // Continuous streaming without backpressure management
128
+ ```
129
+
130
+ #### 4.2 Connection Exhaustion
131
+ **Location**: Lines 58-60, 61-63
132
+ **Issue**: Inadequate connection management.
133
+
134
+ ```typescript
135
+ const timeout = setTimeout(() => {
136
+ response.close();
137
+ }, 5 * 60 * 1000); // 5-minute timeout may be too long
138
+ ```
139
+
140
+ **Attack Scenario**:
141
+ ```bash
142
+ # Connection exhaustion attack
143
+ for i in {1..10000}; do
144
+ curl -H "Accept: application/x-jinaga-feed-stream" \
145
+ "http://target/feeds/valid_hash" &
146
+ done
147
+ ```
148
+
149
+ **Remediation**:
150
+ ```typescript
151
+ // Add connection limits and backpressure
152
+ const MAX_CONCURRENT_STREAMS = 100;
153
+ const STREAM_TIMEOUT = 30000; // 30 seconds
154
+ const MAX_MEMORY_PER_STREAM = 10 * 1024 * 1024; // 10MB
155
+ ```
156
+
157
+ ### 5. Race Condition Vulnerabilities
158
+ **CVSS Score: 6.8 (Medium)**
159
+ **CWE-362: Concurrent Execution using Shared Resource**
160
+
161
+ **Location**: Lines 492-502, 523-565
162
+ **Issue**: Concurrent access to shared resources without proper synchronization.
163
+
164
+ ```typescript
165
+ // Multiple async operations without proper coordination
166
+ const listeners = inverses.map(inverse => this.factManager.addSpecificationListener(
167
+ inverse.inverseSpecification,
168
+ async (results) => {
169
+ // Race condition: bookmark updates
170
+ bookmark = await this.streamFeedResponse(...);
171
+ }
172
+ ));
173
+ ```
174
+
175
+ ### 6. Information Disclosure Vulnerabilities
176
+ **CVSS Score: 5.3 (Medium)**
177
+ **CWE-200: Information Exposure**
178
+
179
+ #### 6.1 Error Information Leakage
180
+ **Location**: Lines 722-738
181
+ **Issue**: Detailed error messages expose internal system information.
182
+
183
+ ```typescript
184
+ res.status(500).send(error.message); // Exposes internal errors
185
+ ```
186
+
187
+ #### 6.2 Timing-Based Information Disclosure
188
+ **Location**: Lines 523-565
189
+ **Issue**: Processing time reveals data characteristics.
190
+
191
+ ## Reliability and Performance Issues
192
+
193
+ ### 1. Infinite Loop Vulnerability
194
+ **Severity: High**
195
+ **Location**: Lines 523-565
196
+
197
+ ```typescript
198
+ while (hasMoreResults && pageCount < maxPages) {
199
+ // Potential infinite loop if bookmark doesn't change
200
+ if (newBookmark === bookmark) {
201
+ hasMoreResults = false; // Mitigation exists but insufficient
202
+ }
203
+ }
204
+ ```
205
+
206
+ **Issue**: Insufficient loop termination conditions.
207
+
208
+ ### 2. Resource Leak Vulnerabilities
209
+ **Severity: High**
210
+ **Location**: Lines 503-507
211
+
212
+ ```typescript
213
+ stream.done(() => {
214
+ for (const listener of listeners) {
215
+ this.factManager.removeSpecificationListener(listener);
216
+ }
217
+ });
218
+ ```
219
+
220
+ **Issue**: Listeners may not be cleaned up if stream.done() is not called.
221
+
222
+ ### 3. Database Connection Pool Exhaustion
223
+ **Severity: Medium**
224
+ **Location**: Lines 524, 571
225
+
226
+ Multiple concurrent database calls without connection pooling limits.
227
+
228
+ ## Detailed Remediation Strategy
229
+
230
+ ### Priority 1: Critical Security Fixes
231
+
232
+ 1. **Implement Authentication Validation**
233
+ ```typescript
234
+ // Add to getOrStream function
235
+ if (requiresAuth && !user) {
236
+ res.sendStatus(401);
237
+ return;
238
+ }
239
+ ```
240
+
241
+ 2. **Add Hash Parameter Validation**
242
+ ```typescript
243
+ const validateHash = (hash: string): boolean => {
244
+ return /^[a-fA-F0-9]{64}$/.test(hash) && hash.length === 64;
245
+ };
246
+ ```
247
+
248
+ 3. **Implement Rate Limiting**
249
+ ```typescript
250
+ const rateLimiter = rateLimit({
251
+ windowMs: 15 * 60 * 1000, // 15 minutes
252
+ max: 100 // limit each IP to 100 requests per windowMs
253
+ });
254
+ ```
255
+
256
+ ### Priority 2: DoS Prevention
257
+
258
+ 1. **Connection Management**
259
+ ```typescript
260
+ const connectionManager = new ConnectionManager({
261
+ maxConcurrentStreams: 100,
262
+ maxStreamDuration: 30000,
263
+ maxMemoryPerStream: 10 * 1024 * 1024
264
+ });
265
+ ```
266
+
267
+ 2. **Backpressure Implementation**
268
+ ```typescript
269
+ class BackpressureStream<T> extends Stream<T> {
270
+ private buffer: T[] = [];
271
+ private maxBufferSize = 1000;
272
+
273
+ feed(data: T) {
274
+ if (this.buffer.length >= this.maxBufferSize) {
275
+ this.pause();
276
+ }
277
+ super.feed(data);
278
+ }
279
+ }
280
+ ```
281
+
282
+ ### Priority 3: Monitoring and Logging
283
+
284
+ 1. **Security Event Logging**
285
+ ```typescript
286
+ const securityLogger = {
287
+ logSuspiciousActivity: (event: SecurityEvent) => {
288
+ // Log to SIEM system
289
+ },
290
+ logAccessAttempt: (user: string, resource: string, success: boolean) => {
291
+ // Audit trail
292
+ }
293
+ };
294
+ ```
295
+
296
+ 2. **Performance Monitoring**
297
+ ```typescript
298
+ const performanceMonitor = {
299
+ trackStreamDuration: (streamId: string, duration: number) => {},
300
+ trackMemoryUsage: (streamId: string, usage: number) => {},
301
+ trackConcurrentConnections: (count: number) => {}
302
+ };
303
+ ```
304
+
305
+ ## Supply Chain Security Analysis
306
+
307
+ ### Dependencies Review
308
+ - **express**: Check for known CVEs in routing
309
+ - **jinaga**: Internal dependency - requires separate audit
310
+ - **Stream class**: Custom implementation needs security review
311
+
312
+ ### Recommended Actions
313
+ 1. Implement dependency scanning in CI/CD
314
+ 2. Regular security updates
315
+ 3. Vendor security assessment for jinaga library
316
+
317
+ ## Compliance Considerations
318
+
319
+ ### GDPR Compliance
320
+ - Data minimization in streams
321
+ - Right to erasure implementation
322
+ - Consent management for streaming data
323
+
324
+ ### SOC 2 Compliance
325
+ - Access logging and monitoring
326
+ - Data encryption in transit
327
+ - Incident response procedures
328
+
329
+ ## Testing Strategy
330
+
331
+ ### Security Testing
332
+ 1. **Penetration Testing**
333
+ - Authentication bypass attempts
334
+ - Input validation testing
335
+ - DoS attack simulation
336
+
337
+ 2. **Automated Security Scanning**
338
+ - SAST tools integration
339
+ - Dependency vulnerability scanning
340
+ - Runtime security monitoring
341
+
342
+ ### Performance Testing
343
+ 1. **Load Testing**
344
+ - Concurrent stream handling
345
+ - Memory usage under load
346
+ - Database connection pooling
347
+
348
+ 2. **Chaos Engineering**
349
+ - Network partition simulation
350
+ - Database failure scenarios
351
+ - Memory pressure testing
352
+
353
+ ## Implementation Timeline
354
+
355
+ ### Phase 1 (Immediate - 1 week)
356
+ - Critical authentication fixes
357
+ - Input validation implementation
358
+ - Basic rate limiting
359
+
360
+ ### Phase 2 (Short-term - 2-4 weeks)
361
+ - DoS prevention measures
362
+ - Enhanced monitoring
363
+ - Security logging
364
+
365
+ ### Phase 3 (Medium-term - 1-3 months)
366
+ - Comprehensive testing
367
+ - Performance optimization
368
+ - Compliance implementation
369
+
370
+ ## Conclusion
371
+
372
+ The `/feeds/:hash` handler contains multiple critical security vulnerabilities that require immediate attention. The streaming functionality, while providing real-time capabilities, introduces significant attack surface that must be properly secured. Implementation of the recommended remediation strategies will significantly improve the security posture and reliability of the system.
373
+
374
+ **Risk Assessment**: **HIGH** - Immediate action required
375
+ **Business Impact**: **CRITICAL** - Potential for data breach and service disruption
376
+ **Remediation Complexity**: **MEDIUM** - Requires coordinated development effort
377
+
378
+ ## Appendix A: Security Checklist
379
+
380
+ - [ ] Authentication validation implemented
381
+ - [ ] Input sanitization and validation
382
+ - [ ] Rate limiting configured
383
+ - [ ] Connection management implemented
384
+ - [ ] Error handling sanitized
385
+ - [ ] Logging and monitoring deployed
386
+ - [ ] Security testing completed
387
+ - [ ] Performance testing completed
388
+ - [ ] Documentation updated
389
+ - [ ] Team training conducted
390
+
391
+ ## Appendix B: Monitoring Queries
392
+
393
+ ```sql
394
+ -- Suspicious hash access patterns
395
+ SELECT hash, COUNT(*), MIN(timestamp), MAX(timestamp)
396
+ FROM feed_access_log
397
+ WHERE timestamp > NOW() - INTERVAL '1 hour'
398
+ GROUP BY hash
399
+ HAVING COUNT(*) > 100;
400
+
401
+ -- Failed authentication attempts
402
+ SELECT ip_address, COUNT(*)
403
+ FROM security_events
404
+ WHERE event_type = 'auth_failure'
405
+ AND timestamp > NOW() - INTERVAL '1 hour'
406
+ GROUP BY ip_address
407
+ HAVING COUNT(*) > 10;
@@ -0,0 +1,294 @@
1
+ # Feeds: Persistent Client Proxy with Waitlist
2
+ Author: @michaellperry
3
+ Status: Draft
4
+ Scope: `/feeds/:hash` streaming path (`Accept: application/x-jinaga-feed-stream`)
5
+
6
+ ## Summary
7
+
8
+ Introduce a per-connection "FeedStreamSession" (client proxy) that:
9
+ - Maintains a waitlist of fact references indicated by inverse observers.
10
+ - Serializes all fetching and streaming through a single "query cycle" loop driven by the current bookmark.
11
+ - Continues initial pagination until exhaustion, then transitions to real-time.
12
+ - Emits an explicit "caught up" signal (SYNC) without changing the response schema (empty `references` array + current `bookmark`).
13
+
14
+ This addresses race conditions between listener callbacks and bookmark mutation, ensures tuple completeness, and guarantees no missed updates when facts arrive between pages.
15
+
16
+ ## Goals
17
+
18
+ - Deliver all historical feed results before push-only mode.
19
+ - Prevent lost updates during the transition to "caught up".
20
+ - Maintain tuple completeness ("projection is the tuple; send all facts in tuple").
21
+ - Preserve response shape and backwards compatibility.
22
+ - Provide predictable lifecycle and resource usage per stream.
23
+
24
+ ## Non-goals
25
+
26
+ - Changing the `/feeds/:hash` response format.
27
+ - Introducing server-sent events or WebSockets (keep current HTTP streaming).
28
+ - Redesigning the authorization/feed query pipeline.
29
+
30
+ ---
31
+
32
+ ## Design
33
+
34
+ ### High-level flow
35
+
36
+ 1. On connection, create a `FeedStreamSession` with:
37
+ - `bookmark` (initial from query param `b` or empty),
38
+ - `waitlist` (Set of fact refs by hash/type),
39
+ - `activeCycle` (boolean),
40
+ - `shuttingDown` (boolean),
41
+ - `stream` (Stream<FeedResponse>).
42
+
43
+ 2. Start inverse observers immediately and push matching fact refs into the waitlist.
44
+ - Observers do not fetch/stream themselves.
45
+ - If no cycle is active, they signal to start (or continue) the cycle.
46
+
47
+ 3. Run the query cycle:
48
+ - While true:
49
+ - Call `authorization.feed(user, feed, start, bookmark)`.
50
+ - If tuples returned:
51
+ - Deduplicate facts across tuples in the page.
52
+ - Stream a frame `{ references, bookmark: results.bookmark }`.
53
+ - Remove any queued facts from the waitlist that are included in streamed tuples.
54
+ - Update `bookmark`.
55
+ - Continue.
56
+ - If no tuples returned:
57
+ - If `waitlist` is empty:
58
+ - Stream a SYNC frame with empty `references` and current `bookmark` (backwards compatible).
59
+ - End the cycle.
60
+ - Else:
61
+ - Facts arrived between last page and now. Continue the loop (run feed again) to collect their full tuples.
62
+
63
+ 4. On stream close, remove all observers and mark `shuttingDown` to prevent new cycles.
64
+
65
+ ### Class responsibilities
66
+
67
+ - FeedStreamSession
68
+ - Owns lifecycle and state for a connection.
69
+ - Public methods:
70
+ - `start()` => sets observers and kicks off initial paginated backfill to exhaustion.
71
+ - `onInverseMatch(factRefs[])` => add to waitlist; `startCycleIfIdle()`.
72
+ - `done()` => unregister observers; cancel cycle on stream close.
73
+ - Private:
74
+ - `runCycle()` => serializes fetch/stream; guarded by `activeCycle`.
75
+ - `streamPage(results)` => compute deduped references and stream with bookmark.
76
+ - `streamSync()` => stream `{ references: [], bookmark }`.
77
+ - `drainWaitlistIfNeeded()` => loop continues until `no tuples && waitlist empty`.
78
+
79
+ - Router integration (src/http/router.ts)
80
+ - Construct `FeedStreamSession` in the `/feeds/:hash` handler.
81
+ - Replace direct "listener => streamFeedResponse" pattern with session waitlist notifications.
82
+
83
+ ### Waitlist semantics
84
+
85
+ - Waitlist keyed by `{hash,type}` ensures no duplicates.
86
+ - Removal happens only when the tuple containing the fact is streamed (preserves tuple completeness).
87
+ - Waitlist is a trigger to keep querying; we do not stream waitlist facts directly.
88
+
89
+ ### SYNC signaling
90
+
91
+ - Preserve response schema:
92
+ - Emit SYNC as a frame with `references: []` and current `bookmark`.
93
+ - Clients that need an explicit signal can treat empty `references` as "caught up".
94
+
95
+ ### Concurrency and races
96
+
97
+ - Single-cycle execution: `activeCycle` flag prevents concurrent runs.
98
+ - Listener callbacks never mutate `bookmark` and never fetch/stream; they only enqueue refs and wake the cycle.
99
+ - Bookmark progresses monotonically within the cycle; no interleaving updates.
100
+
101
+ ### Backpressure and limits
102
+
103
+ - Add conservative limits (configurable):
104
+ - `maxInitialPages` (default 1000)
105
+ - `maxWaitlistSize` (e.g., 50k refs) -> if exceeded, drop connection with 503 and log.
106
+ - Optional small delay every N pages (e.g., 10ms every 10 pages).
107
+ - Consider extending Stream to a "BackpressureStream" in a follow-up to cap buffer length.
108
+
109
+ ### Telemetry
110
+
111
+ - Per-stream diagnostic log context (feed hash, connection id).
112
+ - Metrics:
113
+ - initial pages fetched, total references streamed, cycle count per connection,
114
+ - time to SYNC,
115
+ - waitlist high-water mark,
116
+ - dropped due to limits.
117
+
118
+ ---
119
+
120
+ ## Integration Points
121
+
122
+ - src/http/router.ts
123
+ - `/feeds/:hash` handler (Accept: `application/x-jinaga-feed-stream`).
124
+ - Replace direct `streamFeed` flow by constructing `FeedStreamSession`.
125
+
126
+ - Authorization layer
127
+ - `authorization.feed(userIdentity, feed, start, bookmark)` pagination contract.
128
+
129
+ - Feed definition and inversion
130
+ - `feedCache.getFeed(feedHash)`
131
+ - `invertSpecification(feedDefinition.feed)`
132
+ - `computeTupleSubsetHash` and `computeObjectHash` for filtering listener matches.
133
+
134
+ - FactManager
135
+ - `addSpecificationListener` / `removeSpecificationListener` for inverse triggers.
136
+
137
+ - Stream
138
+ - `Stream<FeedResponse>` feeding and lifecycle (done/close).
139
+
140
+ - Configuration
141
+ - Add server config for page limits, waitlist cap, optional small delays.
142
+
143
+ ---
144
+
145
+ ## Dependencies and Assumptions
146
+
147
+ - Existing bookmark semantics remain stable and monotonic.
148
+ - `authorization.feed` returns `tuples` and `bookmark`, where `tuples.length < pageSize` implies near exhaustion.
149
+ - The test client and any existing consumers tolerate empty `references` frames.
150
+
151
+ ---
152
+
153
+ ## Acceptance Criteria
154
+
155
+ Functional correctness
156
+ - Initial backfill:
157
+ - The server streams all historical pages until the feed is exhausted (no tuples left) before transitioning to real-time.
158
+ - Bookmark advances monotonically across frames.
159
+ - Real-time updates:
160
+ - Facts arriving during initial pagination are not lost; the session continues querying until their tuples are streamed.
161
+ - Facts arriving after SYNC result in subsequent frames with their tuples, preserving order by bookmark.
162
+ - Tuple completeness and deduplication:
163
+ - Each frame's `references` contains the deduplicated set of facts from all tuples in that chunk.
164
+ - Waitlist items are removed only when their containing tuples are streamed.
165
+ - SYNC indication:
166
+ - On catching up (no tuples and empty waitlist), server emits a frame with `references: []` and current `bookmark`.
167
+ - Lifecycle:
168
+ - On connection close, inverse listeners are unregistered and the cycle stops.
169
+ - No concurrent cycles run per connection.
170
+
171
+ Performance and robustness
172
+ - Backfill eventually completes for large feeds within configured safety limits (e.g., 1000 pages).
173
+ - Waitlist does not grow unbounded; if exceeding configured cap, the server closes the stream gracefully with diagnostics.
174
+ - Under concurrent connections, memory and CPU remain within acceptable bounds; no deadlocks or starvation.
175
+
176
+ Security and reliability
177
+ - No race conditions update the bookmark concurrently.
178
+ - No unbounded buffers in the stream path.
179
+ - Listeners are always removed when streams end.
180
+
181
+ Observability
182
+ - Logs/metrics expose: total frames, initial pages, time to SYNC, cycle restarts due to waitlist, waitlist size peaks.
183
+
184
+ ---
185
+
186
+ ## Test Plan
187
+
188
+ ### Unit tests (Jest)
189
+
190
+ FeedStreamSession
191
+ - "Backfill to exhaustion"
192
+ - Given multiple pages returned by `authorization.feed`, verify frames streamed for each page, bookmark progression, and final SYNC frame (empty `references`).
193
+ - "Waitlist triggers continuation"
194
+ - With a final page empty, but waitlist non-empty (push facts into waitlist between pages), verify the cycle continues and fetches at least one more page, then SYNCs when waitlist clears.
195
+ - "No concurrent cycles"
196
+ - Rapidly enqueue from listener multiple times; assert only one `authorization.feed` loop runs at a time (use mocks and counters).
197
+ - "Deduplication"
198
+ - Given overlapping facts across tuples in a page, confirm only unique `{hash,type}` appear in `references`.
199
+ - "Tuple completeness"
200
+ - Verify waitlist facts are only removed when their tuples appear in a streamed page.
201
+ - "SYNC emission"
202
+ - Assert SYNC frame emitted (empty `references`) exactly when both: last query returned no tuples and waitlist is empty.
203
+ - "Bookmark monotonicity"
204
+ - Enforce that sent bookmarks are strictly progressing or stable only for SYNC frames.
205
+ - "Listener filtering"
206
+ - Only enqueue facts from inverse results that match `givenSubset` (use `computeTupleSubsetHash` matching).
207
+ - "Lifecycle cleanup"
208
+ - On `stream.done()`, verify removal of all listeners and no further cycle runs.
209
+ - "Limits"
210
+ - If `waitlist` exceeds cap, session terminates gracefully (mock log) and removes listeners.
211
+
212
+ Router integration
213
+ - "Session wiring"
214
+ - `/feeds/:hash` with stream accept header constructs session, calls `start()`, and proxies `stream.feed` to response writer.
215
+
216
+ ### Integration tests (Jest + supertest or Node http)
217
+
218
+ - "Complete historical stream"
219
+ - Seed DB with > pageSize tuples (e.g., 350) matching the feed. Connect and assert received multiple frames covering all facts, then SYNC.
220
+ - "Race during backfill"
221
+ - Start stream, insert facts while initial pagination runs. Assert no missed updates; frames include these facts by the time SYNC arrives.
222
+ - "Post-SYNC updates"
223
+ - After SYNC, insert new facts. Assert subsequent frames deliver them with updated bookmark.
224
+ - "Multiple concurrent clients"
225
+ - Spin up N clients (e.g., 20). Assert similar message counts and time-to-sync distribution; no client stalls. Reuse `test-concurrent-streams.js` patterns (or convert to automated test).
226
+ - "Disconnect handling"
227
+ - Client disconnect mid-backfill. Assert listeners removed and no residual cycle iterations.
228
+ - "Waitlist stress"
229
+ - Burst insert causing many inverse triggers quickly. Validate waitlist remains under cap or stream terminates with diagnostics.
230
+ - "Timeout/long feeds"
231
+ - Emulate large dataset to ensure the process respects configured page limits and still reaches SYNC or terminates cleanly.
232
+
233
+ Telemetry validation
234
+ - Expose counters to a test sink; assert metrics present (initial pages, cycles, SYNC count).
235
+
236
+ ---
237
+
238
+ ## Rollout
239
+
240
+ - Feature flag: `feeds.fullInitialPagination` (default enabled if prior behavior already documented; if not, roll out gradually).
241
+ - Config options:
242
+ - `feeds.maxInitialPages` (default 1000)
243
+ - `feeds.waitlistCap` (default 50,000)
244
+ - `feeds.pageDelayEvery` (default 10 pages => 10ms delay)
245
+ - Logging: scoped per-connection id and feed hash, with sampling if needed.
246
+
247
+ ---
248
+
249
+ ## Implementation Steps
250
+
251
+ 1. Create `FeedStreamSession` in `src/feeds/FeedStreamSession.ts`.
252
+ - Constructor: dependencies (`authorization`, `factManager`, `feedDefinition`, `userIdentity`, `start`, `stream`, `initialBookmark`, config).
253
+ - Methods: `start()`, `runCycle()`, `onInverseMatch()`, `streamPage()`, `streamSync()`, `dispose()`.
254
+
255
+ 2. Update `/feeds/:hash` handler in `src/http/router.ts`.
256
+ - Where `streamFeed` currently instantiates `Stream<FeedResponse>` and registers listeners, replace with session creation and `start()`.
257
+ - On `stream.done()`, call `session.dispose()`.
258
+
259
+ 3. Move/Reuse helper logic:
260
+ - Deduping facts per page.
261
+ - Inverse computation and given subset filtering.
262
+ - Bookmark management.
263
+
264
+ 4. Add configuration and defaults.
265
+
266
+ 5. Add tests:
267
+ - Unit tests for session (mocks for `authorization`, `factManager`, `stream`).
268
+ - Integration tests using real HTTP and seeded DB.
269
+ - Optionally adapt `test-concurrent-streams.js` into a CI test harness or keep as manual tool.
270
+
271
+ 6. Add telemetry and structured logs.
272
+
273
+ 7. Documentation:
274
+ - Update `documentation/streaming-feeds-full-results-solution.md` to reference the session model and SYNC semantics.
275
+ - Add this design document.
276
+
277
+ ---
278
+
279
+ ## Risks and Mitigations
280
+
281
+ - Risk: Unbounded memory due to large waitlists or slow clients.
282
+ - Mitigate: Waitlist caps, future backpressure stream implementation, connection duration limits.
283
+ - Risk: Deadlock or missed wake-ups.
284
+ - Mitigate: Simple state machine (`activeCycle`, `needsCycle`) and thorough unit tests for concurrency.
285
+ - Risk: Client incompatibility with empty `references` frames.
286
+ - Mitigate: Already valid JSON per existing schema; integration tests against existing client.
287
+
288
+ ---
289
+
290
+ ## Open Questions
291
+
292
+ - Should SYNC frames be optional behind a config gate?
293
+ - Do we need a small debounce before emitting SYNC to reduce "thrash" under constant writes?
294
+ - Do we need connection-level backpressure limits immediately, or as a follow-up (as per security audit notes)?
@@ -0,0 +1,6 @@
1
+ {
2
+ "name": "jinaga-test",
3
+ "lockfileVersion": 3,
4
+ "requires": true,
5
+ "packages": {}
6
+ }